Data transmission method, data storage server and data storage system
By establishing a data download channel between the online data storage server and the offline data storage server, converting offline data into database file format, and directly transmitting database files, the problem of poor offline data timeliness in Hadoop is solved, and data import efficiency and system are improved stability.
Patent Information
- Application Number
- CN201910335421.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-04-24
- Publication Date
- 2024-11-12
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing distributed storage system Hadoop has poor timeliness in offline data storage and cannot meet the requirements of online services to read data at low latency, resulting in an increase in query delay of data storage nodes and affecting service stability.
By establishing a data download channel between the online data storage server and the offline data storage server, dividing offline data into multiple data blocks, and converting them into database file format, directly transmitting database files, thereby reducing the number of data forwarding and writing To improve data import efficiency and system stability.
It effectively reduces the data import time, improves the data import efficiency, and maintains the stability of the data storage system providing query services to the outside world.
Smart Images

Figure CN111857534B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of big data processing, and in particular to a data transmission method, including a data storage server of an online data storage server and an offline data storage server, and a data storage system. Background Art
[0002] With the continuous development of science and technology, the amount of network data is increasing rapidly and growing exponentially. At present, in order to store a large amount of data reasonably, the distributed storage system Hadoop is generally used to store big data. Hadoop is mainly used in offline data storage scenarios, and the timeliness of offline data in Hadoop is poor, which cannot meet the requirements of low-latency data reading of online business. Therefore, it is necessary to write offline data to the online storage system first, and then the online storage system provides low-latency query for online business.
[0003] Writing offline data in the offline data storage system Hadoop to the online data storage system is mostly done by constructing a virtual data storage engine in Hadoop to establish a connection with the data cluster in the online data storage system. After reading the offline data entry, a data request for writing data is constructed and sent to the online data storage system. The online data storage system will read the data request and data entry in each internal node in turn. Figure 1 As shown in Figure 1 This is a structural diagram of an online data storage system. An online data storage system generally includes an application client, a VIP node, a proxy node, and a data storage node. Taking the data storage node in the online data storage system as a Redis node as an example, that is, the data cluster in the online data storage system is a Redis cluster. In the process of writing offline data from Hadoop to the online data storage system, it is necessary to construct a Redis client on Hadoop to establish a connection with the Redis cluster in the online data storage system; the Redis client will construct a Redis request for data writing and send the Redis request to the online data storage system. After the application client of the online data storage system receives the Redis request, the Redis request will be forwarded to the VIP node, the Proxy node, and the Redis node in sequence. Each node reads and writes the offline data entry according to the Redis request.
[0004] It can be seen that data is sent to the Redis cluster through read and write requests one by one, and each request goes through several layers of forwarding before finally reaching the storage node. The link is long, and a large number of write requests pour into the data storage nodes in the online data storage system at the same time, which will cause the query delay of the data storage nodes to increase and affect service stability. Summary of the invention
[0005] In view of this, the purpose of this application is to provide a data transmission method, an online data storage server, an offline data storage server and a data storage system, which can convert data into database files and directly transmit database files, thereby effectively reducing data import time, improving data import efficiency and the stability of the data storage system in providing external query services.
[0006] According to one aspect of the present application, an electronic device is provided, which may include a storage medium and a processor communicating with the storage medium. The storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the storage medium via a bus, and when the processor executes the machine-readable instructions, one or more of the following operations may be performed:
[0007] Establishing a data download channel between the online data storage server and the offline data storage server;
[0008] Obtaining, from the offline data storage server through the data download channel, a database file matching the data import request sent by the online data storage server, wherein the database file includes data content of a plurality of data blocks, and the plurality of data blocks are obtained by dividing the data matching the data import request by the offline data storage server according to the topological structure information of each storage node in the online data storage server;
[0009] The database file is loaded into a preset storage location.
[0010] In some embodiments, when the processor executes the machine-readable instructions, one or more of the following operations may be performed:
[0011] Generating a blank data storage space in the online data storage server that matches the file format of the database file;
[0012] The database file is loaded into the generated data storage space.
[0013] In some embodiments, when the processor executes the machine-readable instructions, one or more of the following operations may be performed:
[0014] Generating a blank data storage space in the online data storage server;
[0015] Converting the file format of the database file into a file format that matches the data storage format of the generated data storage space;
[0016] The database file after the file format conversion is loaded into the generated data storage space.
[0017] In some embodiments, when the processor executes the machine-readable instructions, one or more of the following operations may be performed:
[0018] The existing data storage space corresponding to the database file in the online data storage server and the data in the existing data storage space are deleted.
[0019] In some embodiments, when the processor executes the machine-readable instructions, one or more of the following operations may be performed:
[0020] Determine a generated data storage space in the online data storage server that matches the file format of the data in the database file;
[0021] The database file is loaded into a determined data storage space.
[0022] In some embodiments, when the processor executes the machine-readable instructions, one or more of the following operations may be performed:
[0023] Determining a generated data storage space that matches the database file;
[0024] Converting the data in the database file into data in a file format that matches the determined generated data storage space;
[0025] The data in the database file after format conversion is loaded into the determined generated data storage space.
[0026] According to another aspect of the present application, an electronic device is provided, which may include a storage medium and a processor communicating with the storage medium. The storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the storage medium via a bus, and when the processor executes the machine-readable instructions, one or more of the following operations may be performed:
[0027] Establishing a data download channel between the offline data storage server and the online data storage server;
[0028] Acquire topological structure information of each storage node in the online data storage server;
[0029] Based on the topological structure information, dividing the data matching the data import request of the online data storage server into a plurality of data blocks, and adding the data contents of the plurality of data blocks into a database file;
[0030] The database file is sent to the online data storage server through the data download channel.
[0031] In some embodiments, when the processor executes the machine-readable instructions, one or more of the following operations may be performed:
[0032] Determine the data carrying attribute and data carrying capacity of each storage node based on the attribute information of each storage node represented by the topological structure information;
[0033] Based on the data carrying attribute and data carrying capacity of each storage node, the data set matching the data import request is divided into a plurality of sub-data blocks matching each storage node.
[0034] According to one aspect of the present application, a data transmission method is provided, the data transmission method being applied to an online data storage server, the data transmission method comprising:
[0035] Establishing a data download channel between the online data storage server and the offline data storage server;
[0036] Obtaining, from the offline data storage server through the data download channel, a database file matching the data import request sent by the online data storage server, wherein the database file includes data content of a plurality of data blocks, and the plurality of data blocks are obtained by dividing the data matching the data import request by the offline data storage server according to the topological structure information of each storage node in the online data storage server;
[0037] The database file is loaded into a preset storage location.
[0038] In some embodiments, loading the database file into a preset storage location includes:
[0039] Generating a blank data storage space in the online data storage server that matches the file format of the database file;
[0040] The database file is loaded into the generated data storage space.
[0041] In some embodiments, loading the database file into a preset storage location includes:
[0042] Generating a blank data storage space in the online data storage server;
[0043] Converting the file format of the database file into a file format that matches the data storage format of the generated data storage space;
[0044] The database file after the file format conversion is loaded into the generated data storage space.
[0045] In some embodiments, after loading the database file into a preset storage location, the method includes:
[0046] The existing data storage space corresponding to the database file in the online data storage server and the data in the existing data storage space are deleted.
[0047] In some embodiments, loading the database file into a preset storage location includes:
[0048] Determine a generated data storage space in the online data storage server that matches the file format of the data in the database file;
[0049] The database file is loaded into a determined data storage space.
[0050] In some embodiments, loading the database file into a preset storage location includes:
[0051] Determining a generated data storage space that matches the database file;
[0052] Converting the data in the database file into data in a file format that matches the determined generated data storage space;
[0053] The data in the database file after format conversion is loaded into the determined generated data storage space.
[0054] According to another aspect of the present application, a data transmission method is provided, the data transmission method being applied to an offline data storage server, the data transmission method comprising:
[0055] Establishing a data download channel between the offline data storage server and the online data storage server;
[0056] Acquire topological structure information of each storage node in the online data storage server;
[0057] Based on the topological structure information, dividing the data matching the data import request of the online data storage server into a plurality of data blocks, and adding the data contents of the plurality of data blocks into a database file;
[0058] The database file is sent to the online data storage server through the data download channel.
[0059] In some embodiments, dividing the data set matching the data import request into a plurality of sub-data blocks based on the topological structure information includes:
[0060] Determine the data carrying attribute and data carrying capacity of each storage node based on the attribute information of each storage node represented by the topological structure information;
[0061] Based on the data carrying attribute and data carrying capacity of each storage node, the data set matching the data import request is divided into a plurality of sub-data blocks matching each storage node.
[0062] According to one aspect of the present application, an online data storage server is provided, the online data storage server comprising:
[0063] A channel establishment module, used to establish a data download channel between the online data storage server and the offline data storage server;
[0064] A data download module, configured to obtain, from the offline data storage server, through the data download channel established by the channel establishment module, a database file matching the data import request sent by the online data storage server, wherein the database file includes data content of a plurality of data blocks, and the plurality of data blocks are obtained by dividing the data matching the data import request by the offline data storage server according to the topological structure information of each storage node in the online data storage server;
[0065] The file loading module is used to load the database file downloaded by the data downloading module to a preset storage location.
[0066] In some embodiments, the file loading module is specifically used to:
[0067] Generating a blank data storage space in the online data storage server that matches the file format of the database file;
[0068] The database file is loaded into the generated data storage space.
[0069] In some embodiments, the file loading module is specifically used to:
[0070] Generating a blank data storage space in the online data storage server;
[0071] Converting the file format of the database file into a file format that matches the data storage format of the generated data storage space;
[0072] The database file after the file format conversion is loaded into the generated data storage space.
[0073] In some embodiments, the online data storage server further includes:
[0074] The deleting module is used to delete the existing data storage space corresponding to the database file in the online data storage server, and the data in the existing data storage space.
[0075] In some embodiments, the file loading module is further used to:
[0076] Determine a generated data storage space in the online data storage server that matches the file format of the data in the database file;
[0077] The database file is loaded into a determined data storage space.
[0078] In some embodiments, the file loading module is further used to:
[0079] Determining a generated data storage space that matches the database file;
[0080] Converting the data in the database file into data in a file format that matches the determined generated data storage space;
[0081] The data in the database file after format conversion is loaded into the determined generated data storage space.
[0082] According to one aspect of the present application, an offline data storage server is provided, the offline data storage server comprising:
[0083] A channel building module, used to establish a data download channel between the offline data storage server and the online data storage server;
[0084] An acquisition module, used to acquire topological structure information of each storage node in the online data storage server;
[0085] A data conversion module, configured to divide the data matching the data import request of the online data storage server into a plurality of data blocks based on the topological structure information acquired by the acquisition module, and add the data contents of the plurality of data blocks to a database file;
[0086] A file sending module is used to send the database file to the online data storage server through the data download channel established by the channel building module.
[0087] In some embodiments, the data conversion module is specifically used to:
[0088] Determine the data carrying attribute and data carrying capacity of each storage node based on the attribute information of each storage node represented by the topological structure information;
[0089] Based on the data carrying attribute and data carrying capacity of each storage node, the data set matching the data import request is divided into a plurality of sub-data blocks matching each storage node.
[0090] According to one aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data transmission methods in the above two aspects are executed.
[0091] The data transmission method, online data storage server, offline data storage server and data storage system provided by the embodiments of the present application, the offline data storage server can obtain the topological structure information of each storage node in the online data storage server; based on the topological structure information, the data matching the data import request of the online data storage server is divided into multiple data blocks, and the data contents of the multiple data blocks are added to the database file; the database file is sent to the online data storage server through a data download channel; thereby, the online data storage server can obtain the database file matching the data import request sent by the online data storage server from the offline data storage server through the data download channel, and load the database file to a preset storage location.
[0092] In this way, by converting data into database files and directly transmitting database files, the number of data forwarding between nodes in the data storage system and the number of writes to storage nodes can be effectively reduced, the data import time can be reduced, the efficiency of data import can be improved, and the stability of the data storage system in providing external query services can be maintained. BRIEF DESCRIPTION OF THE DRAWINGS
[0093] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0094] Figure 1 It is a structural diagram of the online data storage system;
[0095] Figure 2 A schematic diagram of a data storage system of the present application;
[0096] Figure 3 A flowchart of a data transmission method provided in an embodiment of the present application;
[0097] Figure 4 A flowchart of a data transmission method provided in an embodiment of the present application;
[0098] Figure 5 A schematic diagram of an online data storage server provided in an embodiment of the present application;
[0099] Figure 6 A schematic diagram of an offline data storage server provided in an embodiment of the present application;
[0100] Figure 7 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION
[0101] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.
[0102] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.
[0103] In order to enable those skilled in the art to use the contents of this application, the following implementation is provided in conjunction with a specific application scenario of "importing offline data in an offline data storage server into an online data storage server". For those skilled in the art, the general principles defined herein may be applied to other embodiments and application scenarios without departing from the spirit and scope of this application. Although this application is mainly described around the importation of offline data in an offline data storage server into an online data storage server, it should be understood that this is only an exemplary embodiment.
[0104] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.
[0105] One aspect of the present application relates to a data storage system, which can establish a data download channel between multiple data storage servers by controlling, especially establishing a data download channel between an offline data storage server and an online data storage server, and controlling the offline data storage server to add offline data to a database file. The online data storage server can directly obtain the database file, so that the offline data in the offline data storage server can be efficiently and quickly imported into the online data storage server.
[0106] It is worth noting that currently big data is generally stored in the distributed storage system Hadoop. However, the timeliness of offline data in Hadoop is poor and cannot meet the requirements of low-latency data reading for online business. Therefore, offline data needs to be written to the online storage system first, and then the online storage system provides low-latency queries for online business.
[0107] Before the present application was filed, offline data in the offline data storage system Hadoop was written to the online data storage system mostly by constructing a data request for writing data and sending it to the online data storage system after Hadoop reads the offline data entry. The online data storage system reads the data request and data entry in turn at each internal node. Since the data is sent to each data storage cluster in the online data storage system through read and write requests one by one, and each request goes through several layers of forwarding before finally reaching the storage node, the link is long. At the same time, a large number of write requests pour into the data storage nodes in the online data storage system, which will cause the query delay of the data storage node to increase, affecting the service stability.
[0108] However, the data storage system provided by the present application can establish a data download channel between the online data storage server and the offline data storage server, divide the offline data into multiple data blocks, convert the data in the multiple data blocks into a database file format, and add the data content of the multiple data blocks to the database file. Therefore, the database file can be directly sent from the offline data storage server to the online data storage server through the established data download channel, and the online data storage server loads its own data in the database file to the corresponding storage location to complete the data import. The data storage system of the present application can import offline data from the offline data storage server to the online data server, thereby effectively reducing the read and write requests for the corresponding data of each node, reducing the data import time and the workload of the data storage system, and improving the efficiency of data import and the stability of the data storage system.
[0109] Based on this, one aspect of the present application relates to a data storage system, the data storage system includes an online data storage server and an offline data storage server, the online data storage server is connected to the offline data storage server, and the offline data storage server can import the stored offline data into the online data storage server, so that the online data storage server provides delayed data query and service query for online business personnel. Specifically, in the data storage system, a data download channel can be established between the online data storage server and the offline data storage server, the offline data storage server divides the stored offline data into a plurality of data blocks by data structure, and can convert the data in the plurality of data blocks into a database file format, thereby adding the data content of the plurality of data blocks after format conversion into a database file, the online data storage server can obtain the database file from the offline data storage server by downloading through the established data download channel, thereby obtaining the data in the plurality of data blocks, and then can load the data in the data blocks to the corresponding positions, thereby completing the data import.
[0110] Wherein, both the online data storage server and the offline data storage server may include a plurality of databases for data storage, so as to realize distributed storage of data.
[0111] The terms "online data storage server" and "online data storage system" in this application can be used interchangeably to refer to a database or data system that can be used to store real-time data or call or access real-time data. "Offline data storage server" and "offline data storage system" can also be used interchangeably to refer to a database or data system that can be used to store offline data, such as the distributed storage system Hadoop.
[0112] The data storage system may be a data platform for industries such as finance, the Internet, medical or education. For example, it may be applied to online car-hailing services in the Internet industry.
[0113] See also Figure 2 , Figure 2 This is a schematic diagram of the data storage system 200 of the present application, which includes an online data storage server 210 and an offline data storage server 220. The online data storage server 210 may include an application client, a VIP node, a proxy node, and a data storage node. The offline data storage server 220 may adopt the distributed storage system Hadoop, and may specifically refer to a Hadoop server cluster or a server in the cluster.
[0114] In some embodiments, the online data storage server 210 and the offline data storage server 220 may include a processor. The processor may process information and / or data related to the service request to perform one or more functions described in the present application. For example, the processor may import and export data based on the service request obtained from the client. In some embodiments, the processor may include one or more processing cores (e.g., a single-core processor (S) or a multi-core processor (S)). By way of example only, the processor may include a central processing unit (CPU), an application specific integrated circuit (ASIC), an application specific instruction set processor (ASIP), a graphics processing unit (GPU), a physical processing unit (PPU), a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic device (PLD), a controller, a microcontroller unit, a reduced instruction set computer (RISC), or a microprocessor, or any combination thereof.
[0115] The following is combined with the content described in the above data storage system and Figure 2 The description of the online data storage server and the offline data storage server shown is used to describe in detail the data transmission method provided in the embodiment of the present application.
[0116] Reference Figure 3 As shown, Figure 3 A flow chart of a data transmission method provided in an embodiment of the present application is provided. The data transmission method is applied to the online data storage server in the data storage system and can be executed by one or more processors in the online data storage server. The specific execution process is as follows:
[0117] S301: Establishing a data download channel between the online data storage server and the offline data storage server.
[0118] In this step, before the offline data in the offline data storage server needs to be imported into the online data storage server, the online data storage server can, as needed, such as regularly importing data for updating, or a user needs to perform data query or business query through the client in the online data storage server. The online data storage server can send a data import request to the offline data storage server through the client to request that the offline data in the offline data storage server be imported into the online data storage server, and after receiving the feedback instruction from the offline data storage server agreeing to import the data, the online data storage server can first control the establishment of a data download channel between the online data storage server and the offline data storage server.
[0119] Among them, establishing a data download channel between the online data storage server and the offline data storage server can be to completely build a new data channel between the online data storage server and the offline data storage server, or to build a data download channel with the help of the inherent download channel for data transmission in the online data storage server and / or the offline data storage server.
[0120] S302: Obtaining a database file that matches a data import request sent by an online data storage server from an offline data storage server through the data download channel, wherein the database file includes data content of a plurality of data blocks, and the plurality of data blocks are obtained by dividing the data that matches the data import request by the offline data storage server according to topological structure information of each storage node in the online data storage server.
[0121] In this step, after the online data storage server establishes the data download channel with the offline data storage server, the online data storage server can obtain a database file matching the data import request sent by the online data storage server from the offline data storage server through the data download channel by file download.
[0122] The database file includes data contents of a plurality of data blocks, and the plurality of data blocks are obtained by dividing the data matching the data import request by the offline data storage server according to the topological structure information of each storage node in the online data storage server.
[0123] Exemplarily, in an embodiment of the present application, the online data storage server may be illustrated by taking the RocksDB storage engine as an example. The online data storage server may download a database file in a format corresponding to RocksDB from an offline data storage server through an established data download process, wherein the format of the data in the multiple data blocks carried in the database file may be a file in the SST format corresponding to the RocksDB storage engine.
[0124] Accordingly, considering that both the online data storage server and the offline data storage server can currently use distributed storage for the storage of big data, the data can be stored on multiple different data storage nodes, such as in multiple storage spaces, multiple storage media or multiple databases, and the multiple storage nodes are connected to each other in a network, such as through routing, thereby forming a complex topological structure, and the connection formed between the online data storage server and the offline data storage server, such as the connection between the storage node of the data storage in the online data storage server and the storage node of the data storage in the offline data storage server, will also form a complex topological structure.
[0125] The multiple data blocks are obtained by dividing the data matching the data import request by the offline data storage server according to the topological structure information of each storage node in the online data storage server. The topological structure information of each storage node in the online data storage server may include, in addition to the topological structure information between each storage node in the online data storage server, the topological structure information of the connection between each storage node in the online data storage server and each storage node in the offline data storage server.
[0126] S303: Load the database file to a preset storage location.
[0127] In this step, after obtaining the database file, the online data storage server can load the database file into a preset storage location. The preset storage location can be a same data storage location that is set, and all the obtained database files can be loaded into the set data storage location, or it can be a plurality of distributed data storage locations, and the obtained database files can be stored in each data storage location respectively.
[0128] Exemplarily, after the online data storage server obtains the database file, it can be cached in the storage node and then directly downloaded to the local by the storage node, that is, multiple RocksDB-style data storage spaces in the online data storage server, such as downloaded to the database.
[0129] In some embodiments of the present application, S303 includes:
[0130] Generate a blank data storage space in the online data storage server that matches the file format of the database file; and load the database file into the generated data storage space.
[0131] In this step, after the online data storage server obtains the database file, the online data storage server can control the generation of a blank data storage space in the online data storage server that matches the file format of the database file, that is, a new data storage space is constructed in the online data storage server, and the format of the data storage in the newly constructed data storage space needs to match the file format of the database file, that is, match the format of the data in the database file, and then the obtained database file can be loaded into the generated data storage space for data storage.
[0132] The generated blank data storage space has data import and export functions.
[0133] In this way, full data import can be achieved. During the data import process, there is no need to send application write requests one by one, but to directly import the database file. The database file can be compressed and directly downloaded to the local computer and then loaded into the RocksDB database. Therefore, the import speed is several times faster than conventional means.
[0134] In some embodiments of the present application, S303 includes:
[0135] Generate a blank data storage space in the online data storage server; convert the file format of the database file into a file format that matches the data storage format of the generated data storage space; and load the database file after the file format conversion into the generated data storage space.
[0136] In this step, after the online data storage server obtains the database file, the online data storage server can control the generation of a blank data storage space in the online data storage server, that is, a new data storage space is constructed in the online data storage server, and the format of the data stored in the newly constructed data storage space is different from the format of the database file. At this time, the file format of the database file, that is, the format of the data in the database file, can be converted into a file format that matches the data storage format of the generated data storage space, and then, the database file with the converted file format can be loaded into the generated data storage space.
[0137] In this way, full data import can be achieved, and during the data import process, there is no need to send application write requests one by one, but to directly import the database file. The database file can be compressed, decompressed and format converted after caching, and directly downloaded to the local computer, and then loaded into the database. Therefore, the import speed is several times faster than conventional means.
[0138] In the above embodiment, after the database file is loaded into a preset storage location, the method includes:
[0139] The existing data storage space corresponding to the database file in the online data storage server and the data in the existing data storage space are deleted.
[0140] In this step, after the online data storage server generates a new data storage space and loads the database file into the generated data storage space, the online data storage server can control the deletion of the existing data storage space corresponding to the database file in the online data storage server and store the data in the existing data storage space.
[0141] Among them, the existing data storage space corresponding to the database file can be the data storage space for imported data generated in the online data storage server before the current data import or the last data import or data update, that is, the existing data storage space and the data storage space generated this time are the same storage space, but the data in the existing data storage space is an old version and has not been updated.
[0142] In some embodiments of the present application, S303 includes:
[0143] Determine a data storage space that has been generated in the online data storage server and that matches the file format of the data in the database file; and load the database file into the determined data storage space.
[0144] In this step, after the online data storage server obtains the database file, the online data storage server can detect the data storage space it has generated to determine the generated data storage space that matches the file format of the data in the database file from multiple data storage spaces, and then load the database file into the determined data storage space for storage.
[0145] Among them, the generated data storage space that matches the file format of the data in the database file is a plurality of data storage spaces already existing in the online data storage server itself. In addition to the data storage space that matches the stored data format, it is also required to be a data storage space that stores part of the data in the database file. That is to say, the data in the database file requires a complete data update of the data in the determined data storage space.
[0146] The generated data storage space may be a data storage space generated during a data import process before the current data import.
[0147] Exemplarily, the online data storage server stores data storage space for different categories of data such as order data and user information, and the purpose of this data import is to update the order data. Then the existing data storage space that matches the imported database file is the data storage space for storing the order data, and the data format of the data stored in the two is the same.
[0148] In this way, incremental data import can be achieved. The incremental import process imports data into the existing data storage space, that is, the database. Although the import process may cause hard disk reading and writing during the data decompression part, experiments have shown that compared with the existing incremental data import, it can effectively reduce the number and process of application write requests, and can be combined with the speed of controlling file import to further reduce the impact of read and write requests. The speed of incremental data import is faster than the conventional incremental data import by sending requests.
[0149] In some embodiments of the present application, S303 includes:
[0150] Determine a generated data storage space that matches the database file; convert the data in the database file into data in a file format that matches the determined generated data storage space; and load the data in the database file after format conversion into the determined generated data storage space.
[0151] In this step, after the online data storage server obtains the database file, the online data storage server can detect the generated data storage space it already has, so as to determine the generated data storage space that matches the database file from multiple data storage spaces, that is, the determined generated data storage space is the storage space that needs to use the data in the database file for data update, but because the data format of the data in the determined generated data storage space is different from the data format of the data in the database file, the data in the database file can be format converted to convert it into the data format of the data in the determined generated data storage space, and then the data in the database file after the format conversion can be loaded into the determined generated data storage space for storage.
[0152] The generated data storage space may be a data storage space generated during a data import process before the current data import.
[0153] The data transmission method provided in the embodiment of the present application establishes a data download channel between an offline data storage server and an online data storage server, and downloads a database file of imported data through the data download channel to realize direct import of database files, thereby effectively reducing the read and write requests for data corresponding to each node, reducing data import time and the workload of the data storage system, and improving the efficiency of data import and the stability of the data storage system.
[0154] See also Figure 4 As shown in Figure 4 A flow chart of a data transmission method provided in an embodiment of the present application is provided. Based on the same inventive concept, an embodiment of the present application further provides a data transmission method, which is applied to the offline data storage server in the data storage system and can be executed by one or more processors in the offline data storage server. The specific execution process is as follows:
[0155] S401: Establishing a data download channel between the offline data storage server and the online data storage server.
[0156] In this step, before the offline data in the offline data storage server needs to be imported into the online data storage server, the offline data storage server can send a data import request to the online data storage server, and then send a feedback instruction to the online data storage server agreeing to import the data, and then control to establish a data download channel between the online data storage server and the offline data storage server.
[0157] Among them, establishing a data download channel between the online data storage server and the offline data storage server can be to completely build a new data channel between the online data storage server and the offline data storage server, or to build a data download channel with the help of the inherent download channel for data transmission in the online data storage server and / or the offline data storage server.
[0158] S402: Acquire topological structure information of each storage node in the online data storage server.
[0159] In this step, after the data download channel is established, the offline data storage server may obtain the topological structure information of each storage node in the online data storage server from the online data storage server.
[0160] Among them, the topological structure information of each storage node in the online data storage server, in addition to the topological structure information between each storage node in the online data storage server, can also include the topological structure information of the connection between each storage node in the online data storage server and each storage node in the offline data storage server.
[0161] S403: Based on the topology information, the data matching the data import request of the online data storage server is divided into a plurality of data blocks, and the data contents of the plurality of data blocks are added to a database file.
[0162] In this step, after obtaining the topology information, the offline data storage server can divide the data matching the data import request of the online data storage server into multiple data blocks according to the topology conditions indicated in the topology information, and then add the data contents of the multiple data blocks to the database file to form a database file that can be transmitted.
[0163] Specifically, based on the topology information, the data matching the data import request of the online data storage server is divided into multiple data blocks. The data carrying attributes and data carrying capacity of each storage node can be determined based on the attribute information of each storage node represented in the topology information, such as the connection status between each storage node and the connected storage nodes. Then, based on the data carrying attributes and data carrying capacity of each storage node, the data set matching the data import request can be divided into multiple sub-data blocks matching each storage node.
[0164] Exemplarily, taking the storage of offline data in an offline data storage server through Hadoop as an example, the data to be imported can be divided on Hadoop through MapReduce calculation method, combined with the attribute information of each storage node, such as the routing information of each storage node, and then the divided imported data is converted into RocksDB SST file and added to the database file.
[0165] S404: Sending the database file to the online data storage server through the data download channel.
[0166] In this step, after the data processing is completed, the offline data storage server can send the database file to the online data storage server through the data download channel.
[0167] The data transmission method provided in the embodiment of the present application establishes a data download channel between an offline data storage server and an online data storage server, and downloads a database file of imported data through the data download channel to realize direct import of database files, thereby effectively reducing the read and write requests for data corresponding to each node, reducing data import time and the workload of the data storage system, and improving the efficiency of data import and the stability of the data storage system.
[0168] Based on the same inventive concept, the embodiments of the present application also provide an online data storage server and an offline data storage server corresponding to the data transmission method. Since the principles of solving the problems by the online data storage server and the offline data storage server in the embodiments of the present application are similar to the data transmission method mentioned above in the embodiments of the present application, the implementation of the online data storage server and the offline data storage server can refer to the implementation of the method, and the repeated parts will not be repeated.
[0169] Please also see Figure 5 , Figure 5 A schematic diagram of an online data storage server provided in an embodiment of the present application, wherein the online data storage server 500 includes:
[0170] A channel establishing module 510, used to establish a data download channel between the online data storage server and the offline data storage server;
[0171] A data download module 520 is configured to obtain, from the offline data storage server, through the data download channel established by the channel establishment module 510, a database file that matches the data import request sent by the online data storage server, wherein the database file includes data content of a plurality of data blocks, and the plurality of data blocks are obtained by dividing the data that matches the data import request by the offline data storage server according to the topological structure information of each storage node in the online data storage server;
[0172] The file loading module 530 is used to load the database file downloaded by the data downloading module 520 to a preset storage location.
[0173] In some embodiments of the present application, the file loading module 530 is specifically used to:
[0174] Generating a blank data storage space in the online data storage server that matches the file format of the database file;
[0175] The database file is loaded into the generated data storage space.
[0176] In some embodiments of the present application, the file loading module 530 is specifically used to:
[0177] Generating a blank data storage space in the online data storage server;
[0178] Converting the file format of the database file into a file format that matches the data storage format of the generated data storage space;
[0179] The database file after the file format conversion is loaded into the generated data storage space.
[0180] In the above embodiment, the online data storage server 500 further includes:
[0181] The deletion module 540 is used to delete the existing data storage space corresponding to the database file in the online data storage server 500 and the data in the existing data storage space.
[0182] In some embodiments of the present application, the file loading module 530 is further used to:
[0183] Determine a generated data storage space in the online data storage server that matches the file format of the data in the database file;
[0184] The database file is loaded into a determined data storage space.
[0185] In some embodiments of the present application, the file loading module 530 is further used to:
[0186] Determining a generated data storage space that matches the database file;
[0187] Converting the data in the database file into data in a file format that matches the determined generated data storage space;
[0188] The data in the database file after format conversion is loaded into the determined generated data storage space.
[0189] For descriptions of the processing flow of the above modules and the interaction flow between the modules, reference can be made to the relevant descriptions in the above method embodiments, which will not be described in detail here.
[0190] The online data storage server provided in the embodiment of the present application establishes a data download channel between the offline data storage server and the online data storage server, and downloads the database file of the imported data through the data download channel to realize the direct import of the database file, thereby effectively reducing the read and write requests of the corresponding data of each node, reducing the data import time and the workload of the data storage system, and improving the efficiency of data import and the stability of the data storage system.
[0191] Please also see Figure 6 , Figure 6 A schematic diagram of an offline data storage server provided in an embodiment of the present application, wherein the offline data storage server 600 includes:
[0192] A channel building module 610 is used to establish a data download channel between the offline data storage server and the online data storage server;
[0193] An acquisition module 620 is used to acquire topological structure information of each storage node in the online data storage server;
[0194] A data conversion module 630 is used to divide the data matching the data import request of the online data storage server into a plurality of data blocks based on the topological structure information obtained by the obtaining module 620, and add the data contents of the plurality of data blocks to a database file;
[0195] The file sending module 640 is used to send the database file to the online data storage server through the data download channel established by the channel building module 610.
[0196] In some embodiments of the present application, the data conversion module 630 is specifically used to:
[0197] Determine the data carrying attribute and data carrying capacity of each storage node based on the attribute information of each storage node represented by the topological structure information;
[0198] Based on the data carrying attribute and data carrying capacity of each storage node, the data set matching the data import request is divided into a plurality of sub-data blocks matching each storage node.
[0199] The offline data storage server provided in the embodiment of the present application establishes a data download channel between the offline data storage server and the online data storage server, and downloads the database file of the imported data through the data download channel to realize the direct import of the database file, thereby effectively reducing the read and write requests of the corresponding data of each node, reducing the data import time and the workload of the data storage system, and improving the efficiency of data import and the stability of the data storage system.
[0200] For descriptions of the processing flow of the above modules and the interaction flow between the modules, reference can be made to the relevant descriptions in the above method embodiments, which will not be described in detail here.
[0201] The present application embodiment also provides a computer device 700, such as Figure 7 As shown, it is a schematic diagram of the structure of a computer device 700 provided in an embodiment of the present application, including: a processor 710, a memory 720, and a bus 730. The memory 720 stores machine-readable instructions executable by the processor 710. When the computer device 700 is running, the processor 710 communicates with the memory 720 through the bus 730. When the machine-readable instructions are executed by the processor 710, one or more of the following operations may also be performed:
[0202] A data download channel is established between the online data storage server and the offline data storage server; a database file matching the data import request sent by the online data storage server is obtained from the offline data storage server through the data download channel, wherein the database file includes data contents of a plurality of data blocks, and the plurality of data blocks are obtained by dividing the data matching the data import request by the offline data storage server according to the topological structure information of each storage node in the online data storage server; and the database file is loaded into a preset storage location.
[0203] When the machine-readable instructions are executed by the processor 710, the following processing is performed, and one or more of the following operations may also be performed:
[0204] A data download channel is established between the offline data storage server and the online data storage server; the topological structure information of each storage node in the online data storage server is obtained; based on the topological structure information, the data matching the data import request of the online data storage server is divided into a plurality of data blocks, and the data contents of the plurality of data blocks are added to a database file; and the database file is sent to the online data storage server through the data download channel.
[0205] Correspondingly, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by the processor, the steps of the above-mentioned data transmission method are executed.
[0206] Specifically, the storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the storage medium is run, the above-mentioned data transmission method can be executed, thereby solving the problems of high data import delay and poor server stability, thereby achieving the effect of reducing the data import delay, reducing the data import time, and improving the stability of the server in providing query services.
[0207] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.
[0208] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0209] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0210] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.
[0211] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A data transmission method, characterized in that: The data transmission method is applied to an online data storage server, and the data transmission method comprises: Establishing a data download channel between the online data storage server and the offline data storage server; Obtaining, from the offline data storage server through the data download channel, a database file matching the data import request sent by the online data storage server, wherein the database file includes data content of a plurality of data blocks, and the plurality of data blocks are obtained by dividing the data matching the data import request by the offline data storage server according to the topological structure information of each storage node in the online data storage server; Loading the database file to a preset storage location; Among them, the offline data storage server determines the data carrying attributes and data carrying capacity of each storage node according to the attribute information of each storage node represented in the topology structure information, and based on the data carrying attributes and data carrying capacity of each storage node, divides the data set matching the data import request into multiple data blocks matching each storage node.
2. The data transmission method according to claim 1, characterized in that: Loading the database file to a preset storage location includes: Generating a blank data storage space in the online data storage server that matches the file format of the database file; The database file is loaded into the generated data storage space.
3. The data transmission method according to claim 1, characterized in that: Loading the database file to a preset storage location includes: Generating a blank data storage space in the online data storage server; Converting the file format of the database file into a file format that matches the data storage format of the generated data storage space; The database file after the file format conversion is loaded into the generated data storage space.
4. The data transmission method according to any one of claims 2 and 3, characterized in that: After loading the database file into a preset storage location, the method includes: Deleting the existing data storage space corresponding to the database file in the online data storage server, and the data in the existing data storage space; The existing data storage space corresponding to the database file is specifically the data storage space for the imported data generated in the online data storage server before the current data import or during the last data import or data update.
5. The data transmission method according to claim 1, characterized in that: Loading the database file to a preset storage location includes: Determine a generated data storage space in the online data storage server that matches the file format of the data in the database file; The database file is loaded into a determined data storage space.
6. The data transmission method according to claim 1, characterized in that: Loading the database file to a preset storage location includes: Determining a generated data storage space that matches the database file; Converting the data in the database file into data in a file format that matches the determined generated data storage space; The data in the database file after format conversion is loaded into the determined generated data storage space.
7. A data transmission method, characterized in that: The data transmission method is applied to an offline data storage server, and the data transmission method includes: Establishing a data download channel between the offline data storage server and the online data storage server; Acquire topological structure information of each storage node in the online data storage server; Based on the topological structure information, dividing the data matching the data import request of the online data storage server into a plurality of data blocks, and adding the data contents of the plurality of data blocks into a database file; Sending the database file to the online data storage server through the data download channel; The step of dividing the data set matching the data import request into a plurality of data blocks based on the topological structure information includes: Determine the data carrying attribute and data carrying capacity of each storage node based on the attribute information of each storage node represented by the topological structure information; Based on the data carrying attributes and data carrying capacity of each storage node, the data set matching the data import request is divided into a plurality of data blocks matching each storage node.
8. An online data storage server, characterized in that: The online data storage server comprises: A channel establishment module, used to establish a data download channel between the online data storage server and the offline data storage server; A data download module, configured to obtain, from the offline data storage server, through the data download channel established by the channel establishment module, a database file matching the data import request sent by the online data storage server, wherein the database file includes data content of a plurality of data blocks, and the plurality of data blocks are obtained by dividing the data matching the data import request by the offline data storage server according to the topological structure information of each storage node in the online data storage server; A file loading module, used for loading the database file downloaded by the data downloading module into a preset storage location; Among them, the offline data storage server determines the data carrying attributes and data carrying capacity of each storage node according to the attribute information of each storage node represented in the topology structure information, and based on the data carrying attributes and data carrying capacity of each storage node, divides the data set matching the data import request into multiple data blocks matching each storage node.
9. The online data storage server according to claim 8, characterized in that: The file loading module is specifically used for: Generating a blank data storage space in the online data storage server that matches the file format of the database file; The database file is loaded into the generated data storage space.
10. The online data storage server according to claim 8, characterized in that: The file loading module is specifically used for: Generating a blank data storage space in the online data storage server; Converting the file format of the database file into a file format that matches the data storage format of the generated data storage space; The database file after the file format conversion is loaded into the generated data storage space.
11. The online data storage server according to any one of claims 9 and 10, characterized in that: The online data storage server also includes: A deletion module, used to delete the existing data storage space corresponding to the database file in the online data storage server, and the data in the existing data storage space; The existing data storage space corresponding to the database file is specifically the data storage space for the imported data generated in the online data storage server before the current data import or during the last data import or data update.
12. The online data storage server according to claim 8, characterized in that: The file loading module is also specifically used for: Determine a generated data storage space in the online data storage server that matches the file format of the data in the database file; The database file is loaded into a determined data storage space.
13. The online data storage server according to claim 8, characterized in that: The file loading module is also specifically used for: Determining a generated data storage space that matches the database file; Converting the data in the database file into data in a file format that matches the determined generated data storage space; The data in the database file after format conversion is loaded into the determined generated data storage space.
14. An offline data storage server, characterized in that: The offline data storage server comprises: A channel building module, used to establish a data download channel between the offline data storage server and the online data storage server; An acquisition module, used to acquire topological structure information of each storage node in the online data storage server; A data conversion module, configured to divide the data matching the data import request of the online data storage server into a plurality of data blocks based on the topological structure information acquired by the acquisition module, and add the data contents of the plurality of data blocks to a database file; A file sending module, used for sending the database file to the online data storage server through the data download channel established by the channel building module; Wherein, the data conversion module is specifically used for: Determine the data carrying attribute and data carrying capacity of each storage node based on the attribute information of each storage node represented by the topological structure information; Based on the data carrying attributes and data carrying capacity of each storage node, the data set matching the data import request is divided into a plurality of data blocks matching each storage node.
15. A data storage system, characterized in that: The data storage system comprises the online data storage server according to any one of claims 8 to 13, and the offline data storage server according to claim 14.
16. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the storage medium communicate via the bus, and the processor executes the machine-readable instructions to execute the steps of the data transmission method described in any one of claims 1 to 6, or executes the steps of the data transmission method described in claim 7.
17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program executes the steps of the data transmission method according to any one of claims 1 to 6, or executes the steps of the data transmission method according to claim 7.
