Data importing method and device
By directly parsing the imported source file into the underlying coded data that can be recognized by the Store module in a distributed database, the problems of long data links and large overhead in traditional import methods are solved, and the rapid import and efficient processing of distributed database data are realized. .
Patent Information
- Application Number
- CN202310841790.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-07
- Publication Date
- 2025-01-07
AI Technical Summary
The traditional distributed database data import method has problems such as long data links, large DB module resolution overhead, large CPU overhead for Store modules and large consistency synchronization overhead.
Through the DM module, the imported source file is directly parsed into the underlying coded data that can be recognized by the Store module, eliminating the intermediate steps of the DB module, and directly sending the underlying coded data files corresponding to the shard to the memory set, optimizing the data link, and reducing the Store module CPU overhead and consistency protocol synchronization overhead.
It realizes the rapid import of distributed database data, shortens the import link, saves CPU overhead, reduces the burden of consistent protocol synchronization, and improves the import efficiency.
Smart Images
Figure CN119271183A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of big data technology, and specifically to the field of distributed database data import technology. Background Art
[0002] With the popularization of big data technology, more and more companies are introducing big data technology stacks and using big data technology to process and mine massive data, and then transmitting the calculation results to downstream systems for use.
[0003] At present, the traditional distributed database data import is to compile the imported data into SQL (Structured Query Language) and import the data by executing SQL. Summary of the invention
[0004] The embodiments of the present disclosure provide a data import method, apparatus, device, storage medium, and program product.
[0005] In a first aspect, an embodiment of the present disclosure proposes a data import method, comprising: reading an import source file; parsing the import source file to obtain underlying coding data recognizable by a memory; scanning the underlying coding data to obtain underlying coding data files corresponding to the slices; sending the underlying coding data files corresponding to the slices to a corresponding memory set, so that the memories in the memory set write the underlying coding data files corresponding to the slices into the slices.
[0006] In the second aspect, the embodiment of the present disclosure proposes a data import device, comprising: a reading module, configured to read an import source file; a parsing module, configured to parse the import source file to obtain underlying coding data recognizable by a memory; a scanning module, configured to scan the underlying coding data to obtain underlying coding data files corresponding to the slices; and a sending module, configured to send the underlying coding data files corresponding to the slices to a corresponding memory set, so that the memories in the memory set write the underlying coding data files corresponding to the slices into the slices.
[0007] In a third aspect, an embodiment of the present disclosure proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any implementation manner in the first aspect.
[0008] In a fourth aspect, an embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, where the computer instructions are used to cause a computer to execute a method as described in any implementation manner in the first aspect.
[0009] In a fifth aspect, an embodiment of the present disclosure proposes a computer program product, including a computer program, which implements the method described in any implementation manner in the first aspect when executed by a processor.
[0010] In the data import method provided by the embodiment of the present disclosure, the DM module reads the import source file and directly parses each line of data into the underlying coded data that can be directly recognized by the Store module. When all the data are processed, the underlying coded data file corresponding to each shard is scanned. These underlying coded data files are directly sent to all copies of the corresponding shards for absorption processing to complete the data import. The data link of the data import scheme disclosed in the present disclosure is: import source file->DM->Store, and the import link is shorter. In addition, the data link is optimized, and the overhead of the DB module is completely eliminated. The overhead of calculating and generating the underlying coded data recognized by the Store module is externalized to the DM module, and the Store module only needs to simply receive the file for digestion, saving the CPU overhead of the Store module. In addition, the DM module can collect millions or hundreds of millions of lines of data at one time and send them to the Store module for absorption processing, reducing the overhead of the Store module consistency protocol synchronization. Thereby realizing a distributed database data fast import technology based on the database underlying format file.
[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] Other features, purposes and advantages of the present disclosure will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings. The drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. Among them:
[0013] Figure 1 It is a traditional distributed database architecture;
[0014] Figure 2 is the distributed database architecture of the present disclosure;
[0015] Figure 3 is a flow chart of an embodiment of a data import method according to the present disclosure;
[0016] Figure 4 is a flow chart of another embodiment of the data import method according to the present disclosure;
[0017] Figure 5 is a flow chart of another embodiment of the data import method according to the present disclosure;
[0018] Figure 6 is a structural schematic diagram of an embodiment of a data import device according to the present disclosure;
[0019] Figure 7 It is a block diagram of an electronic device used to implement the data import method of the embodiment of the present disclosure. DETAILED DESCRIPTION
[0020] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0021] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0022] Figure 1 A traditional distributed database architecture is shown.
[0023] The traditional distributed database architecture may include: Meta (meta) module, DB (Data Base) module, Store (storage) module and DM (Data Migrate) module. Among them, the Meta module can be responsible for metadata management and shard scheduling. The DB module can be responsible for executing SQL, parsing SQL into execution plans, and sending them to the Store module for specific execution. The Store module can be responsible for storing data, and the data is sharded according to the range. Multiple copies of the same shard can synchronize logs through distributed consistency algorithms (such as Raft, Paxos) to ensure data consistency. The DM module can be responsible for data import, parsing the imported source files into SQLs one by one and sending them to the DB module for execution.
[0024] The SQL import solution is: the DM module reads the import source file and translates each line into the import SQL. The DM module sends the import SQL to the DB module for parsing and generates the execution plan corresponding to the SQL. The DB module sends the execution plan corresponding to the SQL to the corresponding Store module for execution. The Store module executes the execution plan sent by the DB module, generates the corresponding underlying encoding data and writes it locally. Multiple copies of the shard synchronize logs through a distributed consistency algorithm to ensure data consistency. That is, the data link of the SQL import solution is: import source file->DM->DB->Store.
[0025] However, the SQL import solution has the following technical problems:
[0026] 1. The data link is long.
[0027] 2. The DB module needs to parse each imported SQL statement and generate the corresponding logical plan and physical execution plan, which is very costly.
[0028] 3. When the Store module executes each SQL execution plan sent by the DB module, it needs to calculate and generate the corresponding underlying encoding format data, and then write it to the local computer, which results in high CPU (Central Processing Unit) overhead.
[0029] 4. The Store module uses a distributed consistency algorithm to keep multiple copies of each SQL data consistent, which results in high synchronization overhead.
[0030] In order to solve the above technical problems, the present disclosure proposes a faster distributed database data import solution. Figure 2 The distributed database architecture of the present disclosure is shown.
[0031] The distributed database architecture of the present disclosure may include: a Meta module, a Store module and a DM module.
[0032] The data import scheme disclosed in the present invention is: the DM module reads the import source file and directly parses each row of data into the underlying coded data that can be directly recognized by the Store module. When all data are processed, the corresponding underlying coded data files are scanned according to the data range of each shard in the import table. These underlying coded data files are directly sent to all copies of the corresponding shards for absorption processing to complete the data import. That is, the data link of the data import scheme disclosed in the present invention is: import source file->DM->Store.
[0033] Compared with the SQL import solution, the data import solution disclosed in this disclosure has the following advantages:
[0034] 1. The import link is shorter.
[0035] 2. Optimize the data link and completely eliminate the overhead of the DB module.
[0036] 3. The overhead of calculating and generating the underlying encoded data recognized by the Store module is externalized to the DM module. The Store module only needs to simply receive and digest the file, saving the CPU overhead of the Store module.
[0037] 4. The DM module can collect millions or hundreds of millions of rows of data at one time and send them to the Store module for absorption and processing, reducing the overhead of Store module consistency protocol synchronization.
[0038] Continue to refer Figure 3 , which shows a process 100 of an embodiment of a data import method according to the present disclosure. The data import method comprises the following steps:
[0039] Step 301, read the import source file.
[0040] In this embodiment, the DM module can read and import source files.
[0041] Usually, the import source file can be the data that needs to be imported that the DM module has accumulated. Usually, the DM module can accumulate millions or hundreds of millions of rows of data and then import the data once, which reduces the overhead of the Store module consistency protocol synchronization.
[0042] Step 302: parse the imported source file to obtain underlying coded data recognizable by the memory.
[0043] In this embodiment, the DM module can parse the imported source file to obtain the underlying coded data recognizable by the memory (Store module).
[0044] Usually, the DM module can directly parse each line of data in the imported source file into the underlying coded data that the Store module can directly recognize. The overhead of calculating and generating the underlying coded data recognized by the Store module is externalized to the DM module, and the Store module only needs to simply receive the file and digest it, saving the CPU overhead of the Store module.
[0045] Step 303, scan the underlying coding data to obtain the underlying coding data file corresponding to the slice.
[0046] In this embodiment, the DM module can scan the underlying coded data to obtain the underlying coded data file corresponding to the slice.
[0047] Among them, different slices can correspond to different data ranges, and according to the data range corresponding to a slice, the underlying coded data file corresponding to the slice can be scanned. For example, the data range corresponding to slice 1 is 1-80, and the data with ID 1-80 can be assigned to slice 1 to obtain the underlying coded data file corresponding to slice 1; the data range corresponding to slice 2 is 81-160, and the data with ID 81-160 can be assigned to slice 2 to obtain the underlying coded data file corresponding to slice 2; and so on, until all the underlying coded data are assigned.
[0048] Step 304: Send the underlying coded data file corresponding to the slice to the corresponding storage set, so that the storage in the storage set writes the underlying coded data file corresponding to the slice into the slice.
[0049] In this embodiment, the DM module may send the underlying coded data file corresponding to the slice to the corresponding memory set. The memory in the memory set may write the underlying coded data file corresponding to the slice into the slice.
[0050] Usually, there are multiple copies of the same shard, so a shard needs to be stored in a memory set consisting of multiple memories. The DM module needs to send the underlying coded data file corresponding to the shard to all memories in the memory set corresponding to the shard at the same time. All memories in the memory set can write the underlying coded data file corresponding to the shard into the shard.
[0051] In some embodiments, the data import scheme of the present disclosure can adaptively import speed limits. Specifically, the DM module can send the underlying encoded data files corresponding to the shard to all memories in the memory set corresponding to the shard at the same time. If the load of a memory is too high, a high load message will be sent to the DM module. When the DM module receives the high load message sent by the memory, it can stop sending the underlying encoded data files to the memory. When the memory load is normal, a normal load message will be sent to the DM module. When the DM module receives the normal load message sent by the memory, it can continue to send the underlying encoded data files to the memory. In this way, the DM module can be infinitely expanded, while also avoiding the DM module importing too fast to affect the normal read and write services of the memory. Among them, the load of the memory can be represented by the CPU usage and / or the amount of undigested imported data. For example, if the CPU usage rate exceeds 70%, and the undigested imported data exceeds 100G, the load can be artificially too high.
[0052] In some embodiments, there is a leader memory in the memory set corresponding to a shard. Other memories in the memory set will send their respective shard storage conditions to the leader memory. The leader memory can send the collected shard storage conditions of each memory to the DM module so that the DM module can obtain the shard storage conditions of each memory in a timely manner. Among them, the leader memory corresponding to the shard can be used to collect and feedback the storage conditions of the underlying coded data files corresponding to the shard by the memory in the memory set corresponding to the shard.
[0053] In the data import method provided by the embodiment of the present disclosure, the DM module reads the import source file and directly parses each line of data into the underlying coded data that can be directly recognized by the Store module. When all the data are processed, the underlying coded data file corresponding to each shard is scanned. These underlying coded data files are directly sent to all copies of the corresponding shards for absorption processing to complete the data import. The data link of the data import scheme disclosed in the present disclosure is: import source file->DM->Store, and the import link is shorter. In addition, the data link is optimized, and the overhead of the DB module is completely eliminated. The overhead of calculating and generating the underlying coded data recognized by the Store module is externalized to the DM module, and the Store module only needs to simply receive the file for digestion, saving the CPU overhead of the Store module. In addition, the DM module can collect millions or hundreds of millions of rows of data and send them to the Store module for absorption processing at one time, reducing the overhead of the Store module consistency protocol synchronization. Thereby realizing a distributed database data fast import technology based on the database underlying format file.
[0054] Continue to refer Figure 4 , which shows a process 400 of another embodiment of a data import method according to the present disclosure. The data import method comprises the following steps:
[0055] Step 401, read the import source file.
[0056] In this embodiment, the specific operation of step 401 is already Figure 3 Step 301 is described in detail in the illustrated embodiment and will not be described again here.
[0057] Step 402 , adaptively adjusting the underlying encoding format according to the database type corresponding to the memory, and parsing the imported source file into underlying encoding data corresponding to the underlying encoding format.
[0058] In this embodiment, the DM module can adaptively adjust the underlying encoding format according to the database type corresponding to the memory, and parse the imported source file into underlying encoding data corresponding to the underlying encoding format.
[0059] Generally, the database import solution disclosed in the present invention is a universal method, which can be applied to, including but not limited to, graph databases, relational databases, vector databases, etc. It only needs to adaptively adjust the underlying encoding format according to different database types, and the solution has a wider scope of application.
[0060] Step 403: Obtain an import table from the metadata management module.
[0061] In this embodiment, the DM may obtain an import table from a metadata management (Meta) module, wherein the import table records the data range of each shard.
[0062] Step 404: Aggregate the underlying coded data according to the fragment data range in the import table to obtain the underlying coded data file corresponding to the fragment.
[0063] In this embodiment, the DM module may aggregate the underlying coded data according to the fragment data range in the import table to obtain the underlying coded data file corresponding to the fragment.
[0064] Among them, different slices can correspond to different data ranges, and according to the data range corresponding to a slice, the underlying coded data file corresponding to the slice can be scanned. For example, the data range corresponding to slice 1 is 1-80, and the data with ID 1-80 can be assigned to slice 1 to obtain the underlying coded data file corresponding to slice 1; the data range corresponding to slice 2 is 81-160, and the data with ID 81-160 can be assigned to slice 2 to obtain the underlying coded data file corresponding to slice 2; and so on, until all the underlying coded data are assigned.
[0065] Step 405: divide the underlying encoded data file corresponding to the slice into multiple subtasks.
[0066] In this embodiment, the DM module may divide the underlying encoded data file corresponding to the slice into multiple subtasks.
[0067] Generally, a DM module may include a main DM module and multiple sub-DM modules. The main DM module may divide the underlying coded data file into multiple sub-tasks according to the file size. One sub-DM module receives one sub-task.
[0068] Step 406: Send the corresponding underlying coded data files to the corresponding storage set through multiple subtasks at the same time, so that the storage in the storage set writes the underlying coded data files corresponding to the slices into the slices.
[0069] In this embodiment, the DM module can send the corresponding underlying coded data file to the corresponding memory set simultaneously through multiple subtasks, so that the memory in the memory set writes the underlying coded data file corresponding to the slice into the slice.
[0070] Usually, multiple subtasks received by multiple sub-DM modules can import data simultaneously, thus solving the problem of insufficient import performance of a single DM module.
[0071] Step 407: If there is a subtask exception, the abnormal subtask is removed, and the corresponding underlying encoded data file of the abnormal subtask is allocated to other subtasks.
[0072] In this embodiment, if there is an abnormal subtask, the DM module may remove the abnormal subtask and allocate the corresponding underlying encoded data file of the abnormal subtask to other subtasks.
[0073] Usually, if a sub-DM module is abnormal, the sub-tasks received by the sub-DM module cannot be u-imported. In this case, the main DM module can remove the sub-DM module and select other sub-DM modules for the sub-tasks received by it to re-import, thereby ensuring the integrity of data import.
[0074] from Figure 4 It can be seen that Figure 3 Compared with the corresponding embodiment, the data import method in this embodiment highlights the parsing step and the import step. Therefore, the scheme described in this embodiment adaptively adjusts the underlying encoding format according to the database type, and the scheme has a wider scope of application. The main DM module divides the underlying encoded data file into multiple subtasks according to the file size. Multiple subtasks received by multiple sub-DM modules can import data at the same time, thereby solving the problem of insufficient import performance of a single DM module. In addition, for abnormal sub-DM modules, other sub-DM modules are selected to re-import the subtasks they have received, thereby ensuring the integrity of the data import.
[0075] Further references Figure 5 , which shows a process 500 of another embodiment of the data import method according to the present disclosure. The database type in the data import method is a graph database, and the underlying encoding format is a key-value pair Key-Value format. The data import method includes the following steps:
[0076] Step 501, read the import source file.
[0077] In this embodiment, the specific operation of step 501 is already Figure 3 Step 301 is described in detail in the illustrated embodiment and will not be described again here.
[0078] Step 502, encode the point identification length, data type and number of graph space fragments defined in the graph space of the graph database to obtain the Key value of the point encoding data.
[0079] In this embodiment, the DM module may perform encoding according to the point identification length, data type, and number of graph space fragments defined in the graph space of the graph database to obtain the Key value of the point encoding data.
[0080] Step 503, encoding is performed according to the attribute information and attribute value of the point type defined in the graph space to obtain the Value value of the point encoding data.
[0081] In this embodiment, the DM module may perform encoding according to the attribute information and attribute value of the point type defined in the graph space to obtain the Value value of the point encoding data.
[0082] Specifically, for the points in the graph database, the point identification length, data type and number of graph space fragments defined in the graph database are encoded to obtain the Key value of the point coded data; the attribute information and attribute value of the point type defined in the graph space are encoded to obtain the Value value of the point coded data, that is, (Key, Value) is a point coded data after the encoding is completed.
[0083] Step 504, perform positive and negative encoding twice according to the point identification length, data type and number of graph space fragments defined in the graph space of the graph database, and the preset edge rank value to obtain two key values of the edge encoding data.
[0084] In this embodiment, the DM module can perform positive and negative encoding twice according to the point identification length, data type and number of graph space fragments defined in the graph space of the graph database, as well as the preset edge rank value to obtain two key values of the edge encoding data.
[0085] Step 505 , encoding is performed according to the attribute information of the edge type defined in the graph space and the preset attribute value to obtain the Value value of the edge encoding data.
[0086] In this embodiment, the DM module may perform encoding according to the attribute information of the edge type defined in the graph space and the preset attribute value to obtain the Value value of the edge encoding data.
[0087] Specifically, for the edges in the graph database, since the graph data stores forward and reverse edges at the bottom layer, a data source record will be encoded into a pair of (key, value) pairs that conform to the preset file format. According to the point identification length defined in the graph space of the graph database (i.e., the length of the source point identification and the target point identification of the edge), the data type and the number of graph space shards, and the preset edge rank value (user configuration, the default is 0), two positive and negative encodings are performed to obtain two Key values of the edge encoding data; according to the attribute information of the edge type defined in the graph space and the preset attribute value, the encoding is performed to obtain the Value value of the edge encoding data. The positive and negative keys and the value form two (Key, Value) data respectively, that is, a pair of edge encoding data after the encoding is completed.
[0088] Step 506, scan the underlying coding data to obtain the underlying coding data file corresponding to the slice.
[0089] Step 507: Send the underlying coded data file corresponding to the slice to the corresponding storage set, so that the storage in the storage set writes the underlying coded data file corresponding to the slice into the slice.
[0090] In this embodiment, the specific operations of steps 506-507 are already Figure 3 In the illustrated embodiment, steps 303 - 304 are described in detail and will not be repeated here.
[0091] from Figure 5 It can be seen that Figure 3 Compared with the corresponding embodiment, the data import method in this embodiment highlights the underlying coding steps of the graph database. Therefore, the solution described in this embodiment supports directly constructing multi-source data into the underlying data storage format supported by the graph database, isolating the data import process from the read and write services of the graph database, and avoiding resource preemption between data import and data services.
[0092] Further references Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a data import device, which is similar to Figure 3 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0093] like Figure 3 As shown, the data import device 600 of this embodiment may include: an acquisition module 601, a parsing module 602, a scanning module 603 and a sending module 604. Among them, the reading module 601 is configured to read the import source file; the parsing module 602 is configured to parse the import source file to obtain the underlying coding data recognizable by the memory; the scanning module 603 is configured to scan the underlying coding data to obtain the underlying coding data file corresponding to the slice; the sending module 604 is configured to send the underlying coding data file corresponding to the slice to the corresponding memory set, so that the memory in the memory set writes the underlying coding data file corresponding to the slice into the slice.
[0094] In this embodiment, in the data import device 600, the specific processing of the acquisition module 601, the parsing module 602, the scanning module 603 and the sending module 604 and the technical effects thereof can be referred to in Figure 3 The relevant descriptions of steps 301 - 304 in the corresponding embodiment are not repeated here.
[0095] In some optional implementations of this embodiment, the scanning module 603 is further configured to: obtain an import table from the metadata management module, wherein the import table records the shard data range; aggregate the underlying encoded data according to the shard data range in the import table to obtain the underlying encoded data file corresponding to the shard.
[0096] In some optional implementations of this embodiment, the sending module 604 is further configured to: divide the underlying encoded data file corresponding to the slice into multiple subtasks; and simultaneously send the corresponding underlying encoded data file to the corresponding storage set through multiple subtasks.
[0097] In some optional implementations of this embodiment, the sending module 604 is further configured to: if there is a subtask exception, remove the abnormal subtask, and assign the corresponding underlying encoded data file of the abnormal subtask to other subtasks.
[0098] In some optional implementations of this embodiment, the data import device 600 also includes: a stop module, configured to stop sending the underlying encoded data file to the memory in response to receiving a message that the load is too high sent by the memory; and a continue sending module, configured to continue sending the underlying encoded data file to the memory in response to receiving a message that the load is normal sent by the memory.
[0099] In some optional implementations of this embodiment, the data import device 600 also includes: a receiving module, configured to receive the shard storage status sent by the leader storage device corresponding to the shard, wherein the leader storage device corresponding to the shard is used to collect and feedback the storage status of the underlying encoded data file corresponding to the shard in the storage device in the storage device set corresponding to the shard.
[0100] In some optional implementations of this embodiment, the parsing module 602 is further configured to: adaptively adjust the underlying encoding format according to the database type corresponding to the memory, and parse the imported source file into underlying encoding data corresponding to the underlying encoding format.
[0101] In some optional implementations of the present embodiment, the database type is a graph database, the underlying encoding format is a key-value pair format; and the parsing module 602 is further configured to: encode according to the point identification length, data type and number of graph space fragments defined in the graph space of the graph database to obtain the Key value of the point encoding data; encode according to the attribute information and attribute value of the point type defined in the graph space to obtain the Value value of the point encoding data; encode twice in positive and negative directions according to the point identification length, data type and number of graph space fragments defined in the graph space of the graph database, and the preset edge rank value to obtain two Key values of the edge encoding data; encode according to the attribute information and preset attribute value of the edge type defined in the graph space to obtain the Value value of the edge encoding data.
[0102] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0103] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
[0104] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0105] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.
[0106] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0107] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as the data import method. For example, in some embodiments, the data import method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the data import method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the data import method in any other appropriate manner (e.g., by means of firmware).
[0108] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0109] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0110] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0111] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0112] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0113] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.
[0114] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions provided by this disclosure can be achieved, and this document does not limit this.
[0115] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A data import method, comprising: Read the import source file; Parsing the imported source file to obtain underlying coded data recognizable by the memory; Scan the underlying coded data to obtain underlying coded data files corresponding to the slices; The underlying coded data file corresponding to the slice is sent to a corresponding storage set, so that the storage in the storage set writes the underlying coded data file corresponding to the slice into the slice.
2. The method according to claim 1, wherein: The scanning of the underlying coded data to obtain the underlying coded data file corresponding to the slice includes: Obtain an import table from a metadata management module, wherein the import table records a shard data range; The underlying coded data is aggregated according to the fragment data range in the import table to obtain the underlying coded data file corresponding to the fragment.
3. The method according to claim 1, wherein: The sending of the underlying coded data file corresponding to the slice to the corresponding storage set includes: Splitting the underlying encoded data file corresponding to the slice into multiple subtasks; The corresponding underlying encoded data files are sent to the corresponding memory set simultaneously through the multiple subtasks.
4. The method according to claim 3, wherein: The sending of the underlying coded data file corresponding to the slice to the corresponding storage set also includes: If there is a subtask exception, the abnormal subtask is removed, and the corresponding underlying encoded data file of the abnormal subtask is allocated to other subtasks.
5. The method according to claim 1, wherein: The method further comprises: In response to receiving a load-too-high message sent by the memory, stopping sending the underlying encoded data file to the memory; In response to receiving a load normal message sent by the memory, continue to send the underlying encoded data file to the memory.
6. The method according to claim 1, wherein: The method further comprises: Receive the storage status of the slice sent by the leader storage corresponding to the slice, wherein the leader storage corresponding to the slice is used to collect and feedback the storage status of the underlying encoded data file corresponding to the slice by the storage in the storage set corresponding to the slice.
7. The method according to claim 1, wherein: The step of parsing the imported source file to obtain underlying coded data recognizable by the memory includes: The underlying encoding format is adaptively adjusted according to the database type corresponding to the memory, and the imported source file is parsed into underlying encoding data corresponding to the underlying encoding format.
8. The method according to claim 7, wherein: The database type is a graph database, and the underlying encoding format is a key-value pair format; as well as The step of adaptively adjusting the underlying encoding format according to the database type of the memory and parsing the imported source file into underlying encoding data corresponding to the underlying encoding format includes: Encode the point identification length, data type and number of graph space fragments defined in the graph space of the graph database to obtain a Key value of the point encoding data; Encode the attribute information and attribute value of the point type defined in the graph space to obtain the Value value of the point encoding data; According to the point identification length, data type and number of graph space shards defined in the graph space of the graph database, and the preset edge rank value, two positive and negative encodings are performed to obtain two key values of the edge encoding data; Encoding is performed according to the attribute information of the edge type defined in the graph space and the preset attribute value to obtain the Value value of the edge encoding data.
9. A data import device, comprising: A reading module, configured to read an import source file; A parsing module, configured to parse the imported source file to obtain underlying coded data recognizable by the memory; A scanning module is configured to scan the underlying coded data to obtain underlying coded data files corresponding to the slices; The sending module is configured to send the underlying coded data file corresponding to the slice to the corresponding storage set, so that the storage in the storage set writes the underlying coded data file corresponding to the slice into the slice.
10. The device according to claim 9, wherein: The scanning module is further configured to: Obtain an import table from a metadata management module, wherein the import table records a shard data range; The underlying coded data is aggregated according to the fragment data range in the import table to obtain the underlying coded data file corresponding to the fragment.
11. The device according to claim 9, wherein: The sending module is further configured to: Splitting the underlying encoded data file corresponding to the slice into multiple subtasks; The corresponding underlying encoded data files are sent to the corresponding memory set simultaneously through the multiple subtasks.
12. The device according to claim 11, wherein The sending module is further configured to: If there is a subtask exception, the abnormal subtask is removed, and the corresponding underlying encoded data file of the abnormal subtask is allocated to other subtasks.
13. The device according to claim 9, wherein: The device also includes: A stop module, configured to stop sending the underlying encoded data file to the memory in response to receiving a message of excessive load sent by the memory; The continuing sending module is configured to continue sending the bottom layer encoded data file to the memory in response to receiving a load normal message sent by the memory.
14. The device according to claim 9, wherein: The device also includes: The receiving module is configured to receive the shard storage status sent by the leader storage corresponding to the shard, wherein the leader storage corresponding to the shard is used to collect and feedback the storage status of the underlying encoded data file corresponding to the shard in the storage set corresponding to the shard.
15. The device according to claim 9, wherein: The parsing module is further configured to: The underlying encoding format is adaptively adjusted according to the database type corresponding to the memory, and the imported source file is parsed into underlying encoding data corresponding to the underlying encoding format.
16. The device according to claim 15, wherein: The database type is a graph database, and the underlying encoding format is a key-value pair format; and The parsing module is further configured to: Encode the point identification length, data type and number of graph space fragments defined in the graph space of the graph database to obtain a Key value of the point encoding data; Encode the attribute information and attribute value of the point type defined in the graph space to obtain the Value value of the point encoding data; According to the point identification length, data type and number of graph space shards defined in the graph space of the graph database, and the preset edge rank value, two positive and negative encodings are performed to obtain two key values of the edge encoding data; Encoding is performed according to the attribute information of the edge type defined in the graph space and the preset attribute value to obtain the Value value of the edge encoding data.
17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.
18. A non-transitory computer-readable storage medium storing computer instructions, the computer instructions being used to cause the computer to execute the method of any one of claims 1-8.
19. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 8.