Data export and import method, device and system, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a data exporting and importing method, device and system, electronic equipment and a storage medium, and the data exporting method comprises the steps: responding to a data exporting instruction of a first data range, carrying out the hierarchical splitting of the first data range, and obtaining a second data range with a hierarchical relationship; determining a data list according to the second data range with the hierarchical relationship; exporting the data in the third data range to obtain a data file; wherein the third data range comprises the second data range located at the bottom layer of the hierarchical relationship; and determining an export file according to the data list and the data file. And high efficiency of data export can be realized.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular, to a method, apparatus, system, electronic device, and storage medium for data export and import. Background Art

[0002] In the current era of rapid digital development, data has become one of the important assets of enterprises. With the continuous expansion of enterprise business and the accelerating promotion of digital transformation, the needs of enterprises for data management and application are becoming increasingly complex and diverse. To improve data risk prevention capabilities, to meet the flexible use of data in different business scenarios, or to support the switching of data-owned business service deployment methods, etc., the importance of data export / import operations such as data backup / migration has become increasingly prominent. However, there are many problems with current data export and import methods: the export and import efficiency of large amounts of data is relatively low. Summary of the Invention

[0003] Embodiments of the present disclosure provide a method, apparatus, system, electronic device, and storage medium for data export and import, which can achieve high efficiency in data export and import.

[0004] In a first aspect, embodiments of the present disclosure provide a data export method, including:

[0005] In response to a data export instruction for a first data range, hierarchically split the first data range to obtain a second data range with a hierarchical relationship;

[0006] Determine a data list according to the second data range with the hierarchical relationship;

[0007] Export the data within a third data range to obtain a data file; wherein, the third data range includes the second data range at the bottom layer of the hierarchical relationship;

[0008] Determine an export file according to the data list and the data file.

[0009] In a second aspect, embodiments of the present disclosure provide a data import method, including:

[0010] In response to a data import instruction for a first data range, obtain an import file corresponding to the first data range; wherein, the import file includes a data list and a data file;

[0011] Reconstruct a second data range with a hierarchical relationship according to the data list; wherein, the second data range includes the data range obtained by hierarchically splitting the first data range in advance;

[0012] Import the data in the data file corresponding to the third data range; wherein, the third data range includes the second data range at the bottom layer of the hierarchical relationship.

[0013] Thirdly, an embodiment of the present disclosure further provides a data export device, including:

[0014] A splitting module, configured to perform hierarchical splitting on the first data range in response to a data export instruction for the first data range, so as to obtain a second data range with a hierarchical relationship;

[0015] A list determination module, configured to determine a data list according to the second data range with the hierarchical relationship;

[0016] A data export module, configured to export the data within the third data range to obtain a data file; wherein, the third data range includes the second data range at the bottom layer of the hierarchical relationship;

[0017] An export file determination module, configured to determine an export file according to the data list and the data file.

[0018] Fourthly, an embodiment of the present disclosure further provides a data import device, including:

[0019] An acquisition module, configured to acquire an import file corresponding to the first data range in response to a data import instruction for the first data range; wherein, the import file includes a data list and a data file;

[0020] A reconstruction module, configured to reconstruct a second data range with a hierarchical relationship according to the data list; wherein, the second data range includes the data range obtained by hierarchically splitting the first data range in advance;

[0021] A data import module, configured to import the data in the data file corresponding to the third data range; wherein, the third data range includes the second data range at the bottom layer of the hierarchical relationship.

[0022] Fifthly, an embodiment of the present disclosure further provides a data export and import system, including any one of the data export devices and data import devices in the embodiments of the present disclosure.

[0023] Sixthly, an embodiment of the present disclosure further provides an electronic device, the electronic device includes:

[0024] One or more processors;

[0025] A storage device, configured to store one or more programs,

[0026] When the one or more programs are executed by the one or more processors, the one or more processors implement the data export method or the data import method according to any one of the embodiments of the present disclosure.

[0027] In a seventh aspect, an embodiment of the present disclosure further provides a storage medium containing computer-executable instructions, which are used to execute the data export method or the data import method according to any one of the embodiments of the present disclosure when executed by a computer processor

[0028] In an eighth aspect, an embodiment of the present disclosure further provides a computer program product, characterized in that the computer program product includes a computer program, and the computer program implements the data export method or the data import method according to any one of the embodiments of the present disclosure when executed by a processor.

[0029] In the technical solution of the embodiment of the present disclosure, the data export method may include: in response to a data export instruction for a first data range, hierarchically splitting the first data range to obtain a second data range with a hierarchical relationship; determining a data list according to the second data range with a hierarchical relationship; exporting the data within a third data range to obtain a data file; wherein the third data range includes the second data range at the bottom layer of the hierarchical relationship; determining an export file according to the data list and the data file. During the data export process, by splitting the data range, it is beneficial to parallelly export the data within the split range, thereby ensuring the efficiency of data export. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In combination with the accompanying drawings and with reference to the following specific embodiments, the above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the original elements and elements are not necessarily drawn to scale.

[0031] Figure 1 It is a flowchart of a data export method provided by an embodiment of the present disclosure;

[0032] Figure 2 It is a schematic diagram of a data export and import service architecture provided by an embodiment of the present disclosure;

[0033] Figure 3 It is a signaling diagram of a data migration service provided by an embodiment of the present disclosure;

[0034] Figure 4 It is a schematic diagram of an export file in a data export method provided by an embodiment of the present disclosure;

[0035] Figure 5Schematic diagram of range splitting and reconstruction logic in a data export method provided by an embodiment of the present disclosure;

[0036] Figure 6 Example diagram of data range splitting in a data export method provided by an embodiment of the present disclosure;

[0037] Figure 7 Schematic diagram of splitting logic in a data export method provided by an embodiment of the present disclosure;

[0038] Figure 8 Schematic diagram of encryption and decryption of a data export and import service provided by an embodiment of the present disclosure;

[0039] Figure 9 Flow schematic diagram of a data import method provided by an embodiment of the present disclosure;

[0040] Figure 10 Schematic diagram of the creation logic of a data import task in a data import method provided by an embodiment of the present disclosure;

[0041] Figure 11 Schematic diagram of the structure of a data export device provided by an embodiment of the present disclosure;

[0042] Figure 12 Schematic diagram of the structure of a data import device provided by an embodiment of the present disclosure;

[0043] Figure 13 Schematic diagram of the hierarchical structure of a data export and import system provided by an embodiment of the present disclosure;

[0044] Figure 14 Schematic diagram of a migration pipeline in a data export and import system provided by an embodiment of the present disclosure;

[0045] Figure 15 Schematic block diagram of the data reading and writing process in a data export and import system provided by an embodiment of the present disclosure;

[0046] Figure 16 Schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Detailed implementation manners

[0047] Embodiments of the present disclosure will be described in more detail with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not used to limit the protection scope of the present disclosure.

[0048] It should be understood that the various steps described in the method embodiments of the present disclosure may be executed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this regard.

[0049] As used herein, the term "comprising" and its variations are open-ended, that is, "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0050] It should be noted that the concepts such as "first", "second", etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0051] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise clearly specified in the context, it should be understood as "one or more".

[0052] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only for illustrative purposes and are not used to limit the scope of these messages or information.

[0053] To clearly explain the technical solutions provided by the embodiments of the present disclosure, the relevant terms are explained as follows:

[0054] Data export is the process of copying and storing business data from service components in various online environments to a static data storage.

[0055] Data import is the process of writing business data from a static data storage to various service components in a target environment.

[0056] Static data storage is a data storage service used to reliably store and support efficient transmission across physical devices, mainly for data transfer between physically isolated deployment environments.

[0057] Source environment is the environment where the data is located before data export.

[0058] Target environment is the environment where the data is located after data import.

[0059] Service deployment environment is the actual physical deployment environment of applications and related data storage services, such as including private environments, cloud-native environments, and software as a service (SaaS) environments, etc.

[0060] Data migration, a complete process of data export and data import;

[0061] Data entity, the business layer has relatively independent data objects; data entities can be divided and isolated as independent objects during the data migration process;

[0062] Migration pipeline, including the processing logic for chunking, exporting, and importing data entities.

[0063] Figure 1 The flowchart of a data export method provided by an embodiment of the present disclosure. The embodiments of the present disclosure are applicable to the situation of data export, for example, applicable to the situation of exporting various heterogeneous data. This method can be executed by a data export device, which can be implemented in the form of software and / or hardware, and the device can be configured in an electronic device, such as configured in a server.

[0064] As Figure 1 shown, the data export method provided by this embodiment may include:

[0065] S110. In response to a data export instruction for a first data range, hierarchically split the first data range to obtain a second data range with a hierarchical relationship.

[0066] Exemplarily, Figure 2 The schematic diagram of a data export and import service architecture provided by an embodiment of the present disclosure. Refer to Figure 2 , in the source environment, data can be stored through various heterogeneous data storage services ( Figure 2 represented by storage service 1 in), and various storage services may include application services, relational database services (RDS), remote dictionary services (Redis), and search servers (Elasticsearch, ES), etc. Among them, the data stored by the storage service may include the business data of the enterprise, such as structured data, static files, semi-persistent cache records, and index files, etc. In the embodiments of the present disclosure, the data in various storage services can be exported, and it is supported to finely select the data range (i.e., the first data range) that needs to be exported. Among them, the dimension of the supported fine selection can be pre-configured according to the actual business scenario.

[0067] In the embodiments of the present disclosure, splitting logic and export logic may be defined in advance according to the associations and dependencies among the data in the first data range. Among them, the splitting logic may define splitting levels, splitting dimensions, splitting operations, etc. for each splitting level; the export logic may define export order, export dependencies, etc. Correspondingly, in response to receiving an instruction to export data within the first data range (i.e., a data export instruction), the splitting logic and export logic related to the data within the first data range may be obtained, laying a foundation for subsequent execution of data range splitting and data export operations.

[0068] In the embodiments of the present disclosure, the first data range may be split at least one level in series according to the splitting logic. Exemplarily, Figure 2 the first data range may be input into a three-level series splitter (which may be denoted as splitter 1 - splitter 3), where each splitter may divide the input data range into sub-ranges according to the splitting logic.

[0069] Among them, the data range obtained by splitting each level may be referred to as a second data range, that is, there may be a hierarchical relationship among the second data ranges. Referring again to Figure 2 , splitter 1 may split the input first data range to obtain the second data ranges of the first splitting level; the second data ranges of the first splitting level may be input (i.e., upper-level input) into splitter 2 for splitting to obtain the second data ranges of the second splitting level; the second data ranges of the second splitting level may be input (i.e., upper-level input) into splitter 3 for splitting to obtain the second data ranges of the third splitting level. Among them, the more levels the first data range is split, the lower the corresponding second data range is considered to be and the finer the corresponding data granularity is.

[0070] S120. Determine a data list according to the second data ranges with a hierarchical relationship.

[0071] In the embodiments of the present disclosure, the second data ranges obtained by splitting each level, as well as the hierarchical relationship among the data ranges, may be recorded in the data list. As Figure 2 shown, a data list may be generated according to the second data ranges of each level output by splitter 1 - splitter 3. Among them, this data list may support the reconstruction of hierarchical data ranges during subsequent data import processes, thus facilitating the parallel import of data within the data range.

[0072] In some alternative implementation manners, determining a data list according to the second data ranges with a hierarchical relationship may include: determining a data list in a tree structure according to the second data ranges and the hierarchical relationship to which the second data ranges belong; where the nodes in the tree structure are used to represent the second data ranges.

[0073] Since the data splitting logic can be understood as the logic of splitting the parent data range into child data ranges, in these alternative implementation manners, a tree node structure can be adopted to determine the data list. Among them, the nodes at the same layer in the tree structure can represent the second data ranges corresponding to the same splitting level. And each node can include, but is not limited to, information such as list identifier, parent data range identifier, data range identifier, data entity type and data entity name within the data range, etc. In addition, in some other implementation manners, files (such as excel files, relational database files, and embedded database files, etc.) can also be used to record each second data range and its hierarchical relationships to obtain the data list. Or, other methods can be used to determine the data list, which will not be enumerated here.

[0074] S130. Export the data within the third data range to obtain a data file; wherein, the third data range includes the second data range located at the bottom layer of the hierarchical relationship.

[0075] Among them, the second data range located at the bottom layer of the hierarchical relationship can be considered as the data range corresponding to the last splitting level in the second data range. Referring again to Figure 2 , the second data range at the third splitting level output by the splitter 3 can be called the third data range. Since each third data range is obtained by hierarchical splitting of the first data range, all the data within each third data range is exported, which means all the data within the first data range is exported.

[0076] In the embodiments of the present disclosure, the data within each third data range can be exported at least partially in parallel according to the export logic, and the exported data can be stored as a data file. In some implementation manners, exporting the data within the third data range may include: obtaining the export order of the third data range; wherein, the export order includes a serial export order and / or a parallel export order; exporting the data within the third data range according to the export order. Among them, if there is no restriction on the export order among the data within the third data range, the export logic can indicate that the export order of the data within these third ranges is a parallel export order; if there is a restriction on the export order among the data within the third data range, the export logic can indicate that the data within these third ranges is in a serial export order. By exporting the data within the third data range according to the export order, at least partial parallel export of the data within the third data range can be achieved. By exporting the data within the third data range at least partially in parallel according to the export logic, not only can the efficiency of data export be ensured, but also the consistency and integrity of the association relationship during data export can be ensured.

[0077] S140. Determine the export file according to the data list and the data file.

[0078] In the embodiments of the present disclosure, the data list can provide the list information of the exported data, and the data file can provide the actual data of the exported data. Among them, the list data and the data file can be merged and stored to obtain an export file. Among them, the export file can be wholly stored in the static data storage (such as Figure 2 External Storage 1 in the source environment), so that the data import process can be performed based on the export file.

[0079] Exemplarily, Figure 3 is a signaling schematic diagram of a data migration service provided by an embodiment of the present disclosure. Figure 3 The source environment shown in Figure 2 can correspond to the source environment shown in

[0080] Both can be deployed with Storage Service 1, a data export service, and External Storage 1, and the data export service can be used to execute the data export method provided by the embodiments of the present disclosure. Figure 3 As shown in

[0081] the signaling transmission in the data export process can include: First, the data export service can split the first data range composed of the data in Storage Service 1 according to the splitting logic to construct a data list; then the data export service can pull at least part of the data in the first data range in parallel according to the export logic; after that, the data export service can perform operations such as converting and locally storing the pulled data to obtain a data file; finally, the data export service can persistently store the data file and the data list to External Storage 1 to obtain an export file, so as to complete the data export process. In addition, after exporting data from the source environment and before importing the data into the target environment, the export file can be completely transferred from External Storage 1 to External Storage 2 in the target environment to complete the data transmission process, laying a foundation for performing the subsequent data import process.

[0082] Among them, the data in each third data range can be stored as multiple data files. For example, structured data can be stored as database files, and static files (such as images, etc.) can be directly stored, etc. Among them, the data files corresponding to the same third data range can be constructed under the same first directory. It can be considered that the data files containing actual data can be obtained by querying the first directory.

[0083] Among them, the directory name of the first directory and the file name of the data file can be stored in the data list. For example, the directory names of each first target and the file names of each data file can be associated and stored with each third data range in the data list. Thus, the quantity and detailed storage conditions of the exported data can be recorded in the data list. Among them, the data list storing the directory name and the file name can be constructed under the second directory. It can be considered that by querying the second directory, the directory name and the data name of the first directory corresponding to each third data range can be obtained. Furthermore, the data file containing the actual data can be obtained by querying the first directory.

[0084] Among them, the first directory and the second directory can be constructed under the root directory, so that an exported file storing the data list and the data file in the form of a hierarchical directory can be obtained. During the data transmission process, all the files under the root directory can be completely copied to the external storage of the target environment according to the directory hierarchy, which can ensure the integrity of the data transmission.

[0085] In these alternative implementation manners, by storing the data list and the data file in the form of a hierarchical directory, a quick query function for the list file and the data file can be provided, which is beneficial to improving the subsequent data import efficiency.

[0086] In some further implementation manners, since different data within the first data range can be exported according to different splitting logics and export logics, at least one data list can be generated. After generating the first directory, it may further include generating a third directory according to the first directory. For example, the first directories corresponding to the same data list can be constructed under the same third directory. Correspondingly, a second directory can be generated according to the data list, the first directory, the third directory, and the file name of the data file. For example, the directory name of the third directory can also be stored in the corresponding data list, and then the second directory can be constructed according to the data list. Correspondingly, the exported file can be determined according to the first directory, the second directory, the third directory, the data list, and the data file.

[0087] Exemplarily, Figure 4 is a schematic diagram of an exported file in a data export method provided by an embodiment of the present disclosure. Refer to Figure 4 , the exported file may include multi-level directories, where the first-level directory can be called the root directory, and the directory name of the root directory can be named according to the data export task of the first data range. Under the root directory, there may be an information directory (i.e., the second directory) and at least one data directory (i.e., the third directory). Among them, the second directory may contain at least one data list, and each data list may have a corresponding third directory.

[0088] Figure 4 In, in addition to the data list, the second directory may also contain transfer files (such asFigure 4 The transfer files (transfer file 1 - transfer file N) therein. Among them, the transfer files can be used to separately store the content in the data list when the size of the content stored in the data list is greater than or equal to a preset threshold, and the names of the transfer files can be stored in the data list, so as to achieve the transfer storage of the content. Among them, the preset threshold can be set according to experiments or empirical values. By setting the transfer files, the light weight of the data list can be ensured, which is convenient for quickly reconstructing the hierarchical data range based on the data list during the subsequent data import process to improve the data import efficiency. Referring again to Figure 4 , at least two first directories can be included in the third directory (such as Figure 4 the directory 1 - directory N) in Figure 4 the embedded database files and business files) in

[0089] In these further implementation manners, export files with a richer directory hierarchy can be generated to support the construction of export files in complex data export scenarios.

[0090] Referring again to Figure 2 , in some optional implementation manners, the data export method may further include: generating a data export log in response to the export of the data in the third data range. In these optional implementation manners, by intercepting the input and output of the access to the storage service 1 interface during the data export process, the actual data export log can be dynamically generated. Among them, the data export log may include multi-dimensional data export information, for example, it may include: data entity name, data export details and quantity (such as data fields, data query methods and ranges, etc.) and data access duration, etc. By generating the export log, not only can the data export situation be provided for users to refer to, but also it is helpful for locating exceptions during the data export process, and the data export experience can be improved.

[0091] In the technical solution of the embodiment of the present disclosure, in response to the data export instruction of the first data range, the first data range is hierarchically split to obtain a second data range with a hierarchical relationship; a data list is determined according to the second data range with a hierarchical relationship; the data in the third data range is exported to obtain data files; among them, the third data range includes the second data range located at the bottom layer of the hierarchical relationship; an export file is determined according to the data list and the data files. During the data export process, by splitting the data range, it is beneficial to export the data in the split range in parallel, thereby ensuring the high efficiency of data export.

[0092] The optional solutions in the data export method provided in the embodiments of the present disclosure can be combined with those in the above embodiments. The data export method provided in this embodiment describes in detail the process of splitting the data range. By classifying the data according to the business into different business data ranges and splitting the data for different business ranges according to the corresponding splitting logic, it is possible to support the flexible export of data in scenarios with complex business logics and data association relationships.

[0093] In the data export method provided in this embodiment, the first data range includes at least one business data range; the hierarchical splitting of the first data range may include: hierarchically splitting the business data range according to the splitting logic corresponding to the business data range.

[0094] Exemplarily, Figure 5 is a schematic diagram of the range splitting and reconstruction logic in a data export method provided in an embodiment of the present disclosure. Refer to Figure 5 , the first data range may include the business data ranges of three businesses, namely Business A - Business C. Among them, the data between different business data ranges may not overlap; among them, the data within different business data ranges may correspond to different migration pipelines, that is, they may correspond to different splitting logics, export logics, and import logics. Thus, the corresponding business data ranges can be hierarchically split according to the splitting logic corresponding to each business data range. Figure 5 In, the data export process only exemplarily shows the splitting process of the business data range corresponding to Business A, and it can be considered that the splitting processes of the business data ranges corresponding to Business B and Business C may have different splitting logics.

[0095] Refer to again Figure 5 , for the second data ranges with hierarchical relationships obtained by splitting each business data range, data lists can be determined respectively; and the data lists can be used to reconstruct the second data ranges with hierarchical relationships during the data import process, which is beneficial to the parallel import of data within each data range.

[0096] Exemplarily, Figure 6 is an example diagram of data range splitting in a data export method provided in an embodiment of the present disclosure. Refer to Figure 6, the process of splitting the data range may include: First, the business data range can be split according to the first splitting dimension to obtain the second data range 1 and the second data range 2 at the first splitting level; then, the second data range 1 and the second data range 2 can be split according to the second splitting dimension to obtain the second data range 3-5 and the second data range 6-7 at the second splitting level; where the second data range 3-5 belongs to the second data range 1 and the second data range 6-7 belongs to the second data range 2; finally, the second data range 3-7 can be split according to the third splitting dimension to obtain the second data range 8-N at the third splitting level. Among them, the second data range at the third splitting level can be used as the third data range, and the data within each third data range can be exported at least partially in parallel to obtain each data file. It can be understood that the splitting process of the first data range can refer to Figure 6 the splitting process of the business data range shown.

[0097] Exemplarily, Figure 7 is a schematic diagram of the splitting logic in a data export method provided by an embodiment of the present disclosure. Figure 7 In the splitting logic shown, N splitting levels can be defined, and the splitting operations performed in each splitting level can all include decoding, paging, and encoding. Among them, the splitting operation of each splitting level can be executed by a splitter, and the splitting result (i.e., the second data range) corresponding to each splitting level can be recorded in the data list. Refer to Figure 7 , the splitting result corresponding to the N-level splitting (i.e., the second data range at the bottom layer of the hierarchical relationship) can be input into the data export component to execute the at least partially parallel export of the data within the third data range.

[0098] The technical solution of the embodiment of the present disclosure describes in detail the process of splitting the data range. By classifying the data according to business categories into different business data ranges and splitting the data in different business ranges according to the corresponding splitting logic, it can support the flexible export of data in scenarios with complex business logics and data association relationships. The data export method provided by the embodiment of the present disclosure and the data export method provided by the above embodiment belong to the same general concept. For the technical details not described in detail in this embodiment, reference can be made to the above embodiment, and the same technical features have the same beneficial effects in this embodiment and the above embodiment.

[0099] The various optional schemes in the data export method provided in the embodiment of the present disclosure and the above-mentioned embodiment can be combined. The data export method provided in this embodiment describes the process of data export in detail. By using distributed task scheduling to export data in parallel, it is possible to tolerate errors caused by the instability of basic components during the data export process, which can significantly reduce the time consumption of data export, and has high stability and easy maintenance. In addition, during the data export process, business files can be exported directly, and structured business data can be stored in an embedded database for export, which can greatly improve the export efficiency. Moreover, during the export process, the data storage capacity of the preset storage tool can be called to export data, which can reduce the difficulty of developing data export services.

[0100] In the data export method provided in this embodiment, exporting data within the third data range may include: determining a corresponding data export task based on the third data range; wherein the data export task is stored in the first task queue; and pulling the data export task from the first task queue through a distributed thread and executing it to export the data within the third data range.

[0101] For example, Figure 2 The splitting result output by the third splitter (i.e., the second data range at the bottom of the hierarchical relationship) can be input into the distributed task scheduling component, so that the distributed task scheduling component can create a third data range to determine the corresponding data export task. Among them, the data export task can be persisted in the first task queue according to the corresponding export logic. Among them, a distributed thread can be created through the existing thread creation method, and the distributed thread can be called through the distributed task scheduling component to actively pull the data export task from the shared first task queue and execute it. Thereby, the data within the third data range can be exported at least partially in parallel. Among them, after the data export task is executed, the data export task status can also be updated through the distributed task scheduling component.

[0102] In addition, if Figure 2 The distributed task scheduling component can also be used to connect the splitters in series. That is, the split results output by each splitter can be input into the distributed task scheduling component, and output to the splitter of the next split level through the task scheduling component. Thus, the distributed task scheduling component can construct the corresponding data export task according to the second data range of each split level to obtain the data export task with a hierarchical relationship. Correspondingly, in the data export process, the distributed task scheduling component can support the dependent triggering of the data export task with a hierarchical relationship. For example, when all child data export tasks are completed, the parent data export task is marked as completed; for another example, when the child data export task fails to execute, the parent data export task is recursively marked as failed; for another example, the execution of the parent data export task can trigger the execution of the child data export task, etc.

[0103] In addition, the distributed task scheduling component can also provide the following task scheduling capabilities: support for querying the status of data export tasks; support for timeout monitoring during the execution of data export tasks; support for limited error retries during the execution of data export tasks; support for distributed execution of data export tasks. Among them, the support for distributed execution of data export tasks can include: support for dynamically adjusting the number of data export tasks executed in parallel; in the case of failure of data export tasks, re-executing the data export tasks on other threads through an error retry mechanism, etc.

[0104] By adopting the method of distributed task scheduling for parallel data export, it is possible to tolerate errors caused by unstable basic components during data export, significantly reduce the data export time, and have high stability and easy maintenance.

[0105] In the data export method provided in this embodiment, the data within the third data range includes business files and / or structured business data; the process of exporting the data within the third data range to obtain a data file may include: storing the business file to obtain a data file; and / or writing the business data into an embedded database file to obtain a data file.

[0106] Among them, the business file may include the static files mentioned above. Among them, the business file can be directly stored in the form of a file during the export process, for example, it can be a data file that can be described by binary byte streams such as attachments, large text files, compressed packages, and files that can be self-parsed by the business. Among them, the business data may include data stored in a structured database (such as Redis, etc.). Among them, the business data can be centrally and regularly stored in a local database during the export process to obtain a data file. The local database may include an embedded database, and the data file corresponding to the corresponding business data may include, for example, Figure 2 the embedded database file shown. The embedded database may include a lightweight database (SQLite), etc. Since SQLite is relatively lightweight, data migration through it can improve the migration efficiency to a certain extent.

[0107] In this embodiment, different types of data can be exported in different ways to obtain corresponding data files. In addition, after obtaining the data file, the data file can also be compressed through existing data compression algorithms, thereby improving the efficiency of data export, transmission, and import.

[0108] In the data export method provided in this embodiment, exporting the data within the third data range may include: exporting the data within the third data range through a preset storage tool. See again Figure 2, the data export component can directly call the storage interfaces provided by the preset storage tools (such as data storage devices, file writing components, and data writing components) to perform data export operations (such as writing, transfer, and upload, etc.) within the third data range, which can reduce the development difficulty of the data export service.

[0109] The technical solution of the embodiment of the present disclosure, by adopting the method of distributed task scheduling for parallel data export, can tolerate errors caused by unstable basic components during the data export process, can significantly reduce the data export time, has high stability and is easy to maintain. In addition, during the data export process, business files can be directly exported, and structured business data can be stored in an embedded database for export, which can greatly improve the export efficiency. Moreover, during the export process, the data storage capabilities of the preset storage tools can be called to export data, which can reduce the development difficulty of the data export service. The data export method provided by the embodiment of the present disclosure and the data export method provided by the above embodiment belong to the same general concept. Technical details not described in detail in this embodiment can be referred to the above embodiment, and the same technical features have the same beneficial effects in this embodiment and the above embodiment.

[0110] The various alternative solutions in the data export methods provided in the embodiment of the present disclosure and the above embodiment can be combined. The data export method provided in this embodiment supplements the data export process. By encrypting the data export list and data files during the process, the security of the data migration process can be ensured, and risks such as data leakage and tampering can be reduced.

[0111] In the data export method provided in this embodiment, determining the export file according to the data export list and data files may include: encrypting the data export list and data files; determining the export file according to the encrypted data export list and the encrypted data files. See Figure 2 , the encryption component in the data export service can be used to encrypt the data export list and data files by using existing encryption algorithms. Then, the encrypted data export list and data files can be merged and stored to obtain the export file.

[0112] The data export method provided in this embodiment may further include: obtaining a preset public key; generating a root key in response to a data export instruction; generating a sub-key corresponding to the third data range in response to splitting to obtain the third data range; wherein the sub-key is stored in the data export list.

[0113] Among them, the preset public key can be considered as the public key in the public-private key pair generated by relevant components in the target environment, and the preset public key can be pre-sent to the encryption component of the data export service in the source environment to obtain the preset public key. Among them, in response to the data export instruction for the first data range, the first key can also be generated by an existing key generation method, and the first key can be called the root key. Among them, in response to splitting to obtain each third data range, the second key corresponding to each third data range can be generated by an existing key generation method, and the second key can be called the sub-key. Among them, each sub-key can be stored in the data list and can be associated and stored with each third data range in the data list.

[0114] Exemplarily, Figure 8 is a schematic diagram of encryption and decryption for a data export and import service provided by an embodiment of the present disclosure. It can be considered that Figure 8 is Figure 2 an expansion schematic of the encryption and decryption process based on this, and for parts not described in detail, reference can be made to Figure 2 the relevant description. As Figure 8 shown, in the data export method provided in this embodiment, encrypting the data list and the data file can include: encrypting the data file corresponding to the third data range with the sub-key to obtain the encrypted data file; encrypting the data list with the root key to obtain the encrypted data list; encrypting the root key with the preset public key to obtain the encrypted root key; among them, the encrypted root key is transmitted to the target environment along with the export file.

[0115] Among them, an existing symmetric encryption algorithm can be used to encrypt the data file corresponding to the third data range with the sub-key and encrypt the data list with the root key. Among them, an existing asymmetric encryption algorithm can be used to encrypt the root key with the preset public key. Among them, the encrypted root key is transmitted to the target environment for the data import service in the target environment to perform decryption and data import operations.

[0116] The technical solution of the embodiment of the present disclosure supplements the data export process. By encrypting the data list and the data file during the data export process, the security of the data migration process can be ensured, and risks such as data leakage and tampering can be reduced. The data export and import method provided by the embodiment of the present disclosure and the data export and import method provided by the above embodiment belong to the same general concept. For technical details not described in detail in this embodiment, reference can be made to the above embodiment, and the same technical features have the same beneficial effects in this embodiment and the above embodiment.

[0117] Figure 9The flowchart of a data import method provided by an embodiment of the present disclosure. The embodiments of the present disclosure are applicable to the situation of data import, for example, applicable to the situation of importing data into various heterogeneous storages. This method can be executed by a data import device, which can be implemented in the form of software and / or hardware, and the device can be configured in an electronic device, such as configured in a server.

[0118] As Figure 9 shown, the data import method provided in this embodiment may include:

[0119] S910. In response to a data import instruction for a first data range, obtain an import file corresponding to the first data range; wherein, the import file includes a data list and a data file.

[0120] In the embodiments of the present disclosure and the above embodiments, the meanings represented by the same terms can be considered the same. For example, the first data range, the second data range, and the third data range in the embodiments of the present disclosure respectively have the same meanings as those represented by the first data range, the second data range, and the third data range in the data export process, and will not be enumerated here.

[0121] Refer to again Figure 2 , the export file in the data export process can be completely transferred from the external storage 1 in the source environment to the external storage 2 in the target environment to complete data transfer. In the embodiments of the present disclosure, the export file can be obtained from the external storage 2 in response to a data import instruction for a first data range. And, the export file can also be referred to as an import file during the data import process. Wherein, the import file may include a data list and a data file, and the determination process of the data list and the data file can refer to the data export method provided in the embodiments of the present disclosure.

[0122] In the embodiments of the present disclosure, the import logic can be defined in advance according to the associations and dependencies between the data in the first data range. Wherein, the import logic can define the import order, import dependencies, etc. Correspondingly, in response to a data import instruction, the import logic corresponding to the first data range defined in advance can be obtained.

[0123] S920. According to the data list, reconstruct a second data range with a hierarchical relationship; wherein, the second data range includes the data ranges obtained by hierarchically splitting the first data range in advance.

[0124] Wherein, the process of hierarchically splitting the first data range to obtain the second data range can refer to the data export method provided in the embodiments of the present disclosure. Since the data list records each second data range and the hierarchical relationship between the data ranges, a second data range with a hierarchical relationship can be reconstructed through the records in the data list.

[0125] In some implementations, when the data list is in a tree structure, a second data range with a hierarchical relationship can be reconstructed according to the field information included in each node of the data list. Exemplarily, referring again to Figure 5 , the data import process only exemplarily shows the reconstruction process of the second data range with a hierarchical relationship corresponding to Service A. Among them, the reconstruction process of the second data range with a hierarchical relationship can be considered the same as the splitting process of the second data range, belonging to the determination process from a large data range to a small data range.

[0126] S930. Import the data in the data file corresponding to the third data range; wherein, the third data range includes the second data range at the bottom layer of the hierarchical relationship.

[0127] In the embodiments of the present disclosure, the data files corresponding to each third data range can be read at least partially in parallel according to the import logic, and the data in the data files can be written into the storage service to implement the import of the data in the data files corresponding to the third data range. As Figure 2 shown, the storage service 2 in the target environment can also include application services, RDS, Redis, ES, etc. In some implementations, importing the data in the data file corresponding to the third data range may include: obtaining the import order of the data file corresponding to the third data range; wherein, the import order includes a serial import order and / or a parallel import order; importing the data in the data file corresponding to the third data range according to the import order. Among them, if there is no restriction on the import order among the data in the third data range, the import logic can indicate that the import order of the data in the data files corresponding to these third ranges is a parallel import order; if there is a restriction on the import order among the data in the third data range, the import logic can indicate that the data in the data files corresponding to these third ranges is a serial import order. By importing the data in the data files corresponding to the third data range according to the import order, at least partial parallel import of the data in the data files corresponding to the third data range can be achieved. By at least partially parallel importing the data in the data files corresponding to the third data range according to the export logic, not only can the efficiency of data import be ensured, but also the consistency and integrity of the association relationship during data import can be ensured.

[0128] Referring again to Figure 2 , in some implementations, the data files corresponding to the third data range may include business files and embedded data files. Among them, during the data import process, the business file can directly obtain the corresponding file through transfer / download and write the file into the storage service 2. Among them, during the data import process of the embedded database file, data can be first read from the embedded database file and then the read data can be written into the storage service 2.

[0129] Exemplarily, Figure 3 FIG. is a signaling schematic diagram of a data migration service provided by an embodiment of the present disclosure. Figure 3 The target environment shown in Figure 2 corresponds to the target environment shown in, and both can be deployed with a storage service 2, a data import service, and an external storage 2, and the data import service can be used to execute the data import method provided by the embodiment of the present disclosure.

[0130] As Figure 3 shown, the signaling transfer during the data import process may include: First, the data import service may obtain an import file from the external storage 2 and pull out the data list in the import file; then the data import service may reconstruct a second data range with a hierarchical relationship according to the data list; among them, the second data range located at the bottom layer of the hierarchical relationship may be called the third data range; then the data import service may pull at least part of the data files corresponding to each third data range in parallel according to the import logic; after that, the data import service may perform operations such as parsing and conversion on the pulled data files to obtain data such as business files and / or structured business data; finally, the data import service may import the data such as business files and / or business data into the storage service 2.

[0131] In some optional implementation manners, the data import method may further include: in response to an import range selection instruction, determining a third data range from the second data range. Referring again to Figure 2 , after reconstructing the second data range with a hierarchical relationship according to the data list, the data range to be imported may also be selected from the second data range according to the import range selection instruction. Among them, the selected data range may correspond to any split-level second data range, and thus the data range located at the bottom layer of the selected second data range may be used as the third data range to be imported.

[0132] In these optional implementation manners, by determining the third data range to be imported according to the import range selection instruction, the data within the first data range can be imported in batches and segments, thereby increasing the flexibility of data import.

[0133] In some optional implementation manners, the import file may further include a first directory and a second directory; among them, the first directory is generated according to the third data range corresponding to the data file; the second directory is generated according to the data list, the first directory, and the file name of the data file; the data method may further include: obtaining the data list, the first directory, and the file name by querying the second directory in the import file; performing at least part of the parallel import of the data in the data file corresponding to the third data range, including: performing at least part of the parallel import of the data in the corresponding data file by querying the first directory and the file name.

[0134] Among them, the generation processes of the first directory and the second directory can refer to the data export method. In these alternative implementation manners, by adopting a hierarchical directory manner to construct the import file, rapid query of the data list file and the data file can be realized, which is beneficial to improving the data import efficiency.

[0135] In some further implementation manners, since different data within the first data range can be exported according to different splitting logics and export logics, at least one data list can be generated. Correspondingly, the import file may further include a third directory; wherein, the third directory can be constructed according to the first directory corresponding to the same data list. Correspondingly, the data method may include: obtaining the data list, the third directory, the first directory and the file name by querying the second directory in the import file; at least partially parallel importing the data in the data file corresponding to the third data range may include: under the third directory corresponding to the data list, querying the first directory and the file name, and at least partially parallel importing the data in the corresponding data file.

[0136] In these further implementation manners, it is possible to realize querying the data file and importing the data from the corresponding directory for different data lists, which is beneficial to improving the efficiency of parallel data import and can support the export of data in complex data export scenarios.

[0137] In addition, refer to Figure 4 , in addition to the data list, the second directory may further include a transfer file. By setting the transfer file, the lightweight of the data list can be ensured, which is convenient for quickly reconstructing the hierarchical data range based on the data list during the subsequent data import process to improve the data import efficiency.

[0138] Refer to again Figure 2 , in some alternative implementation manners, it may further include: generating a data import log in response to the import of the data in the data file corresponding to the third data range. In these alternative implementation manners, by intercepting the input and output of the access to the storage service 2 interface during the data import process, the actual data import log can be dynamically generated. Among them, the data import log may include multi-dimensional data export information, for example, it may include: data entity name, data import details and quantity (such as data fields, data query methods and ranges, etc.) and data access duration, etc. By generating the import log, not only can the data import situation be provided for the user to refer to, but also it is helpful for anomaly location during the data import process, and the data import experience can be improved.

[0139] In the technical solution of the embodiment of the present disclosure, in response to a data import instruction for a first data range, an import file corresponding to the first data range may be obtained; wherein, the import file includes a data list and a data file; according to the data list, a second data range with a hierarchical relationship is reconstructed; wherein, the second data range includes the data ranges obtained by hierarchically splitting the first data range in advance; the data in the data file corresponding to the third data range is imported; wherein, the third data range includes the second data range located at the bottom layer of the hierarchical relationship. During the data import process, by reconstructing the split data range based on the data list, it is beneficial to parallel import the data in the split range, thereby ensuring the efficiency of data import; in addition, during the data import process, by exporting based on the corresponding import logic, the integrity of data import during the parallel import process can be ensured.

[0140] In addition, the data import method provided by the embodiment of the present disclosure and the data export method provided by the above embodiment belong to the same inventive concept. The technical details not described in detail in this embodiment can be referred to the above embodiment, and the same technical features have the same beneficial effects in this embodiment and the above embodiment.

[0141] In the data import method provided by the embodiment of the present disclosure and the above embodiment, each optional solution can be combined. The data import method provided in this embodiment describes the data import process in detail. By adopting the method of distributed task scheduling for parallel data import, it can tolerate errors caused by unstable basic components during the data import process, significantly reduce the data import time, and has high stability and is easy to maintain. In addition, during the export process, the data reading ability of a preset reading tool can be called to import the data, which can reduce the development difficulty of the data import service.

[0142] In the data import method provided in this embodiment, importing the data in the data file corresponding to the third data range may include: determining a corresponding data import task according to the third data range; wherein, the data export task is stored in the second task queue; the data import task is pulled from the second task queue by a distributed thread and executed to import the data in the data file corresponding to the third data range.

[0143] Exemplarily, Figure 2The distributed task scheduling component can also be deployed in the data import service. Among them, the third data range can be read out through the range reading component, and the third data range is input into the distributed task scheduling component, so that the distributed task scheduling component can create a data import task corresponding to the third data range. Among them, the data import task can be persisted to the second task queue according to the corresponding import logic. Among them, the distributed thread can be created through the existing thread creation method, and the distributed thread can be called by the distributed task scheduling component to actively pull the data import task from the shared second task queue and execute it. Thus, at least partial parallel import of the data within the third data range can be realized. After the data import task is executed, the status of the data import task can also be updated through the distributed task scheduling component.

[0144] In addition, as Figure 2 three cascaded task generators (such as Figure 2 the generators 1-3 in

[0145] can also be deployed in Figure 10 and the distributed task scheduling component can cascade each generator. After the second data range with a hierarchical relationship is reconstructed, the second data range from the first level to the third level can be read through the range reader and input into the generators 1-3 respectively, so that the generators 1-3 can create data import tasks corresponding to the second data range from the first level to the third level respectively. In addition, the distributed task scheduling component can also use the data import task of the first level output by the generator 1 as the upper-level input of the generator 2, and use the data import task of the second level output by the generator 2 as the upper-level input of the generator 3, so that a data import task with a hierarchical relationship can be created within the distributed task scheduling component. Figure 10 Exemplarily, Figure 10 is a schematic diagram of the creation logic of the data import task in a data import method provided by an embodiment of the present disclosure. Refer to Figure 2 Each level of creation logic can include reading the second data range of the corresponding level from the data list and creating a data import task for the data within the read second data range. Among them,

[0146] Correspondingly, during the data import process, the distributed task scheduling component can support the dependent triggering of data import tasks with a hierarchical relationship. For example, it supports marking the parent data import task as completed when all child data import tasks are completed; for another example, it supports recursively marking the parent data import task as failed when a child data import task fails; and for yet another example, executing the parent data import task can trigger the execution of child data import tasks, etc.

[0147] In addition, the distributed task scheduling component can also provide the following task scheduling capabilities: support for querying the status of data import tasks; support for timeout monitoring during the execution of data import tasks; support for limited error retries during the execution of data import tasks; support for distributed execution of data import tasks. Among them, supporting the distributed execution of data import tasks can include: supporting dynamically adjusting the number of data import tasks executed in parallel; in the case of a data import task failure, re-executing the data import task on other threads through an error retry mechanism, etc.

[0148] By adopting the method of distributed task scheduling for parallel data import, it is possible to tolerate errors caused by unstable basic components during the data import process, significantly reduce the data import time, and has high stability and is easy to maintain.

[0149] See again Figure 2 In the data import method provided in this embodiment, importing the data in the data file corresponding to the third data range may include: importing the data in the data file corresponding to the third data range through a preset reading tool. See again Figure 2 In some implementation manners, for example, when the imported file is a hierarchical directory file, the range reading component can further read the directory path (i.e., the data storage path) of the data file corresponding to the data in the third data range after reading the third data range; correspondingly, the third data range and the data storage path can be sent to the data import component through the distributed task scheduling component. The data import component can directly call the reading interfaces provided by the preset reading tools (such as data readers, file reading components, and data reading components) to execute the data import process, which can reduce the development difficulty of the data import service.

[0150] In addition, Figure 2 The components filled in the boxes in Figure 2 such as splitters 1-3, data export components, data import components, and range reading components can be considered as components that need to be developed for business needs. The components not filled in the boxes in

[0151] The technical solution of the embodiment of the present disclosure describes the data import process in detail. By adopting the method of distributed task scheduling for parallel data import, it can tolerate errors caused by unstable basic components during the data import process, significantly reduce the data import time, and has high stability and is easy to maintain. In addition, during the export process, the data reading ability of a preset reading tool can be called to import data, which can reduce the development difficulty of the data import service. The data export and import method provided by the embodiment of the present disclosure and the data export and import method provided by the above embodiment belong to the same general concept. Technical details not described in detail in this embodiment can be referred to the above embodiment, and the same technical features have the same beneficial effects in this embodiment and the above embodiment.

[0152] The various alternative solutions in the data export and import methods provided in the embodiments of the present disclosure and the above embodiments can be combined. The data import method provided in this embodiment supplements the data import process. In the embodiment of the present disclosure, the data list and data file in the import file can be in an encrypted state. By decrypting the data list and data file during the import process, the security of the data migration process can be ensured, and risks such as data leakage and tampering can be reduced.

[0153] The data import method provided in this embodiment, after obtaining the import file, may further include: decrypting the import file to obtain a data list and a data file. Among them, the encrypted data list and data file can be decrypted by using a decryption algorithm corresponding to the encryption algorithm to obtain the data list and data file, so as to perform subsequent data import operations.

[0154] See again Figure 8 , the data import method provided in this embodiment may further include: in response to a data import instruction, obtaining a preset private key and an encrypted root key; decrypting the import file may include: decrypting the encrypted root key by using the preset private key to obtain the root key; decrypting the encrypted data list by using the root key to obtain the data list; obtaining a sub-key corresponding to a third data range from the data list; decrypting the encrypted data file corresponding to the third data range by using the sub-key to obtain the data file.

[0155] Among them, the preset private key can be regarded as the private key in the public key-private key pair generated by relevant components in the target environment, and corresponds to the preset public key disclosed above. Among them, an asymmetric decryption algorithm corresponding to the asymmetric encryption algorithm of the encryption root key can be used to decrypt the encrypted root key through the preset private key to obtain the root key. Among them, a symmetric decryption algorithm corresponding to the symmetric encryption algorithm of the encrypted data list can be used to decrypt the encrypted data list through the root key pair to obtain the data list. Then, the data list can be parsed to obtain sub-keys corresponding to each third data range. And, the encrypted data file can be decrypted through the symmetric decryption algorithm corresponding to the symmetric encryption algorithm of the encrypted data file through the sub-key corresponding to it to obtain the data file. Furthermore, the data import service can perform a data import operation through the data list and the data file.

[0156] The technical solution of the embodiments of the present disclosure supplements the data import process. In the embodiments of the present disclosure, the data list and the data file in the import file can be in an encrypted state. By decrypting the data list and the data file during the import process, the security of the data migration process can be ensured, and risks such as data leakage and tampering can be reduced. The data export and import method provided by the embodiments of the present disclosure and the data export and import method provided by the above embodiments belong to the same general inventive concept. Technical details not described in detail in this embodiment can be referred to the above embodiments, and the same technical features have the same beneficial effects in this embodiment and the above embodiments.

[0157] Figure 11 It is a schematic structural diagram of a data export device provided by an embodiment of the present disclosure. The data export device provided in this embodiment is applicable to the situation of data export, for example, applicable to the situation of exporting various heterogeneous data.

[0158] Such as Figure 11 shown, the data export device provided by the embodiments of the present disclosure may include:

[0159] The splitting module 1101 is configured to perform hierarchical splitting on the first data range in response to a data export instruction for the first data range to obtain a second data range with a hierarchical relationship;

[0160] The list determination module 1102 is configured to determine a data list according to the second data range with a hierarchical relationship;

[0161] The data export module 1103 is configured to export the data within the third data range to obtain a data file; wherein, the third data range includes the second data range located at the bottom layer of the hierarchical relationship;

[0162] The export file determination module 1104 is configured to determine an export file according to the data list and the data file.

[0163] In some alternative implementation manners, the splitting module can be used for:

[0164] Performing hierarchical splitting on the service data range according to the splitting logic corresponding to the service data range.

[0165] In some alternative implementation manners, the list determination module can be used for:

[0166] Determining a data list in a tree structure according to a second data range and a hierarchical relationship to which the second data range belongs; wherein, nodes in the tree structure are used to represent the second data range.

[0167] In some alternative implementation manners, the data export module can be used for:

[0168] Obtaining an export order of the third data range; wherein, the export order includes a serial export order and / or a parallel export order;

[0169] Exporting data within the third data range according to the export order.

[0170] In some alternative implementation manners, the data export module can be used for:

[0171] Determining a corresponding data export task according to the third data range; wherein, the data export task is stored in a first task queue;

[0172] Pulling and executing the data export task from the first task queue through a distributed thread to export data within the third data range.

[0173] In some alternative implementation manners, data within the third data range includes service files and / or structured service data;

[0174] The data export module can be used for exporting data within the third data range based on the following process to obtain a data file:

[0175] Storing the service file to obtain a data file; and / or, writing the service data into an embedded database file to obtain a data file.

[0176] In some alternative implementation manners, the data export module can be used for:

[0177] Exporting data within the third data range through a preset storage tool.

[0178] In some alternative implementation manners, the data export device may further include:

[0179] An encryption module, configured to encrypt the data list and the data file;

[0180] An export file determination module, which can be used to determine an export file according to the encrypted data list and the encrypted data file.

[0181] In some optional implementation manners, the data export device may further include:

[0182] A public key acquisition module, which is used to acquire a preset public key;

[0183] A key generation module, which is used to generate a root key in response to a data export instruction; and generate a sub-key corresponding to the third data range in response to splitting to obtain the third data range; wherein, the sub-key is stored in the data list;

[0184] Correspondingly, the encryption module can be used for:

[0185] Encrypt the data file corresponding to the third data range through the sub-key to obtain an encrypted data file;

[0186] Encrypt the data list through the root key to obtain an encrypted data list;

[0187] Encrypt the root key through the preset public key to obtain an encrypted root key; wherein, the encrypted root key is transmitted to the target environment along with the export file.

[0188] In some optional implementation manners, the export file determination module can be used for:

[0189] Generate a first directory according to the third data range corresponding to the data file;

[0190] Generate a second directory according to the data list, the first directory, and the file name of the data file;

[0191] Determine the export file according to the first directory, the second directory, the data list, and the data file.

[0192] In some optional implementation manners, the data export device may further include:

[0193] A first log generation module, which is used to generate a data export log in response to the export of the data in the third data range.

[0194] The data export device provided by the embodiments of the present disclosure can execute the data export method provided by any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects for executing the method.

[0195] It should be noted that the various units and modules included in the above device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of mutual distinction and are not used to limit the protection scope of the embodiments of the present disclosure.

[0196] Figure 12 FIG. 4 is a schematic structural diagram of a data import device provided by an embodiment of the present disclosure. The data import device provided in this embodiment is applicable to data import scenarios, for example, applicable to the scenario of importing data into various heterogeneous storages.

[0197] As Figure 12 shown, the data import device provided by the embodiment of the present disclosure may include:

[0198] An acquisition module 1201, configured to acquire an import file corresponding to a first data range in response to a data import instruction for the first data range; wherein, the import file includes a data list and a data file;

[0199] A reconstruction module 1202, configured to reconstruct a second data range with a hierarchical relationship according to the data list; wherein, the second data range includes the data range obtained by hierarchically splitting the first data range in advance;

[0200] A data import module 1203, configured to import the data in the data file corresponding to a third data range; wherein, the third data range includes the second data range at the bottom layer of the hierarchical relationship.

[0201] In some optional implementation manners, the data import module may be configured to:

[0202] Obtain the import order of the data file corresponding to the third data range; wherein, the import order includes a serial import order and / or a parallel import order;

[0203] Import the data in the data file corresponding to the third data range according to the import order.

[0204] In some optional implementation manners, the data import module may be configured to:

[0205] Determine a corresponding data import task according to the third data range; wherein, the data export task is stored in the second task queue;

[0206] Pull the data import task from the second task queue through a distributed thread and execute it to import the data in the data file corresponding to the third data range.

[0207] In some optional implementation manners, the data import module may be configured to:

[0208] Import the data in the data file corresponding to the third data range through a preset reading tool.

[0209] In some alternative implementation manners, the data import device may further include:

[0210] A decryption module, configured to decrypt the import file after obtaining the import file, so as to obtain a data list and a data file.

[0211] In some alternative implementation manners, the obtaining module may be configured to:

[0212] In response to a data import instruction, obtain a preset private key and an encrypted root key;

[0213] The decryption module may be configured to:

[0214] Decrypt the encrypted root key through the preset private key to obtain the root key;

[0215] Decrypt the encrypted data list through the root key to obtain the data list;

[0216] Obtain a sub-key corresponding to the third data range from the data list;

[0217] Decrypt the encrypted data file corresponding to the third data range through the sub-key to obtain the data file.

[0218] In some alternative implementation manners, the import file further includes a first directory and a second directory; wherein, the first directory is generated according to the third data range corresponding to the data file; the second directory is generated according to the data list, the first directory, and the file name of the data file;

[0219] The obtaining module may be configured to: obtain the data list, the first directory, and the file name by querying the second directory in the import file;

[0220] The data import module may be configured to:

[0221] Import the data in the corresponding data file by querying the first directory and the file name.

[0222] In some alternative implementation manners, the data import module may further include:

[0223] A range determination module, configured to determine a third data range from the second data range in response to an import range selection instruction.

[0224] In some alternative implementation manners, the data import module may further include:

[0225] A second log generation module, configured to generate a data import log in response to the import of data in a data file corresponding to a third data range.

[0226] The data import device provided by the embodiments of the present disclosure can execute the data import method provided by any embodiment of the present disclosure, and has function modules and beneficial effects corresponding to the execution of the method.

[0227] It should be noted that the various units and modules included in the above device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of the functional units are only for the convenience of mutual distinction and do not limit the protection scope of the embodiments of the present disclosure.

[0228] The data export and import system provided in this embodiment may include any data export device provided by the embodiments of the present disclosure and any data import device provided by the embodiments of the present disclosure. In some implementation manners, the data export and import system may include: a splitting module, a list determination module, a data export module, an export file determination module, an acquisition module, a reconstruction module, and a data import module. Among them, the splitting module may include a splitter as Figure 2 shown; the data export module may include a distributed task scheduling component, a data export component, a data storage, a file writing component, and a data writing component as Figure 2 shown; the export file determination module may include an encryption component as Figure 2 shown; the data import module may include a range reading component, a generator, a distributed task scheduling component, a data import component, a data reader, a file reading component, and a data reading component as Figure 2 shown.

[0229] In addition, the system may further include: a logic definition module, configured to define the splitting logic, export logic, and import logic of the first data range.

[0230] In actual business, data usually does not exist in isolation, but has associations and dependencies with other data. During the data migration process, it is necessary to ensure the consistency and integrity of these associations and dependencies. In the related art, high requirements are imposed on developers of data export services and data import services. Developers need to have an in-depth understanding of business logic and perform careful design and coding to avoid problems such as data disorder and loss of association relationships during the data migration process. However, due to the complexity and diversity of actual business logic, the development difficulty is large and the risk is high.

[0231] In the data export and import system provided by the embodiments of the present disclosure, by deploying a logic definition module, during the development of the data export service and the data import service, each business team can be supported to define the splitting logic, export logic, and import logic by themselves, so as to decouple the cross-team collaboration problem in the development process of the data export service and the data import service, greatly compress the development schedule, and improve the development efficiency.

[0232] Exemplarily, Figure 13 FIG. 5 is a schematic diagram of the structural layering of a data export and import system provided by an embodiment of the present disclosure. Figure 13 The shown system structure layering diagram corresponds to Figure 2 the data export and import service architecture shown in FIG. 6. Refer to Figure 13 , the system structure layering may include: a migration logic layer, a migration tool layer, a splitting and reconstruction layer, a data migration layer, an encryption and decryption layer, and a file management layer.

[0233] Among them, the migration logic layer may include a splitting logic, a data export logic, and a data import logic, and can be defined through a logic definition module.

[0234] Among them, the migration tool layer may include Figure 2 the data storage, file writing component, data writing component, data reader, file reader, and data reading component shown in FIG. 7.

[0235] Among them, the splitting and reconstruction layer may include a splitter, a range reading component, and a generator as shown in FIG. 8, which can be respectively used to record the second data range with a hierarchical relationship to the data list, and read the second data range with a hierarchical relationship from the data list. In addition, the splitting and reconstruction layer can also send the range splitting result or range reading result to the distributed task scheduling component, so that the distributed task scheduling component creates and executes a data export task or a data import task with a hierarchical relationship. Among them, the data migration layer may include Figure 2 the data export component and the data import component shown in FIG. 9, and the data export component and the data import component can, through the distributed task scheduling component, call the storage or reading capabilities provided by the migration tool layer to execute the corresponding data export task and data import task. Among them, the distributed task scheduling component can record the relevant task scheduling situation to Figure 13 the task scheduling storage shown in FIG. 10.

[0236] Among them, the encryption and decryption layer may include Figure 2The encryption component and decryption component shown in []. During the data export process, a data list and data files can be generated. The encryption component can encrypt the data list and data files that can be generated, and then an export file can be generated based on the encrypted data list and data files. Among them, the export file can be stored in an external storage through the file management layer. During the data import process, the import file can be read from the external storage, and the decryption component can decrypt the data list and data files in the import file to perform the data import process.

[0237] In addition, Figure 13 It may also include a data audit layer, which can be used to dynamically generate a data export log or data import log of the actual data during the data export process or data import process by intercepting the input and output of accessing the storage service interface.

[0238] Exemplarily, Figure 14 It is a schematic diagram of the migration pipeline in a data export and import system provided by an embodiment of the present disclosure. Refer to Figure 14 , each migration pipeline can include at least one splitting module and at least one data migration module; among them, the splitting module can contain Figure 13 the splitter in the split and reconstruction layer in []. Among them, the data migration module can be equivalent to Figure 13 the data migration layer shown in []. For each business data range, multiple migration pipelines can be configured, and the number and execution logic of the splitting module and data migration module in each pipeline can be different. Among them, modules with the same execution logic in different migration pipelines can be merged. For example Figure 14 As shown in the three migration pipelines in [], since the execution logic of each splitting module is the same, the splitting modules can be merged.

[0239] In addition, there can also be a dependency relationship between the migration pipelines of different services, that is, the export logic and import logic between different business data ranges can also be defined in advance through a logic definition module. Exemplarily, assuming that during the data import process, the data import logic corresponding to Business A depends on the complete data corresponding to Business B, Business C, and Business D in the target environment; the data import logic corresponding to Business B and Business C depends on the complete data corresponding to Business D in the target environment. Then, during the data import process, the execution order of the migration pipelines needs to be: Migration pipeline of Business D → Parallel execution of migration pipelines of Business B and Business C → Migration pipeline of Business A.

[0240] Exemplarily, Figure 15 It is a schematic block diagram of the data reading and writing process in a data export and import system provided by an embodiment of the present disclosure. Refer to Figure 15, storage services can be divided into direct-attached storage services and non-direct-attached storage services. Among them, direct-attached storage services can directly read / write data with data export services / data import services; non-direct-attached storage services can perform data reading / writing through the remote procedure call (RPC) provided by the service layer corresponding to the storage service. In addition, Figure 15 For modules and their related functions not described in Figure 15 , reference can be made to the descriptions of the various modules / components disclosed above, and details will not be elaborated here.

[0241] The data export / import system provided by the embodiments of the present disclosure can execute the data export / import method and data export method provided by any embodiment of the present disclosure, and has functional modules and beneficial effects corresponding to the execution of the methods.

[0242] It should be noted that the various units and modules included in the above device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of the functional units are only for the convenience of mutual distinction and do not limit the protection scope of the embodiments of the present disclosure.

[0243] Next, refer to Figure 16 , which shows a schematic structural diagram of an electronic device (such as Figure 16 the terminal device or server in Figure 16 ) 1600 suitable for implementing the embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet Computers), PMPs (Portable Multimedia Players), in-vehicle terminals (such as in-vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 16 The electronic device shown in Figure 16 is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present disclosure.

[0244] As Figure 16 shown, the electronic device 1600 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 1601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1602 or the program loaded from the storage device 1608 into the random access memory (RAM) 1603. In the RAM 1603, various programs and data required for the operation of the electronic device 1600 are also stored. The processing device 1601, the ROM 1602, and the RAM 1603 are connected to each other through a bus 1604. The input / output (I / O) interface 1605 is also connected to the bus 1604.

[0245] Generally, the following devices can be connected to the I / O interface 1605: an input device 1606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 1607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1609. The communication device 1609 can allow the electronic device 1600 to communicate with other devices wirelessly or wiredly to exchange data. Although Figure 16 the electronic device 1600 with various devices is shown, it should be understood that it is not required to implement or have all the shown devices. Instead, more or fewer devices can be implemented or had.

[0246] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes program codes for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication device 1609, or installed from the storage device 1608, or installed from the ROM 1602. When the computer program is executed by the processing device 1601, the above functions defined in the data export method or the data import method of the embodiment of the present disclosure are executed.

[0247] The electronic device provided by the embodiment of the present disclosure and the data export method and the data import method provided by the above embodiment belong to the same inventive concept. Technical details not described in detail in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0248] The embodiment of the present disclosure provides a storage medium storing computer-executable instructions, and the computer-executable instructions can be used to execute the data export method or the data import method provided by the above embodiment when executed by a computer processor.

[0249] It should be noted that the above storage medium in the present disclosure may be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or a flash memory (FLASH), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium may be any tangible medium that contains or stores executable instructions that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable executable instructions. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium, and the computer-readable signal medium may send, propagate, or transmit executable instructions for use by or in conjunction with an instruction execution system, apparatus, or device. The executable instructions contained on the storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0250] In some embodiments, the client and the server may communicate using any currently known or future-developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), the Internet (e.g., the Internet), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0251] The above storage medium may be included in the above electronic device; or it may exist separately and not be assembled into the electronic device.

[0252] The above storage medium carries one or more executable instructions, and when the one or more executable instructions are executed by the electronic device, the electronic device is caused to:

[0253] In response to a data export instruction for a first data range, perform hierarchical splitting on the first data range to obtain a second data range with a hierarchical relationship; determine a data list according to the second data range with a hierarchical relationship; export the data within a third data range to obtain a data file; wherein, the third data range includes the second data range located at the bottom layer of the hierarchical relationship; determine an export file according to the data list and the data file.

[0254] Alternatively, the above computer-readable medium carries one or more executable instructions, which, when executed by the electronic device, cause the electronic device to:

[0255] In response to a data import instruction for a first data range, obtain an import file corresponding to the first data range; wherein, the import file includes a data list and a data file; reconstruct a second data range with a hierarchical relationship according to the data list; wherein, the second data range includes the data range obtained by pre-hierarchically splitting the first data range; import the data in the data file corresponding to the third data range; wherein, the third data range includes the second data range located at the bottom layer of the hierarchical relationship.

[0256] The executable instructions for performing the operations of the present disclosure can be written in one or more programming languages or combinations thereof. The above programming languages include, but are not limited to, object-oriented programming languages - such as Java, Smalltalk, C++; and also include conventional procedural programming languages - such as the "C" language or similar programming languages. The executable instructions can be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network - including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, by using an Internet service provider to connect through the Internet).

[0257] The embodiments of the present disclosure also provide a computer program product, including a computer program, which when executed by a processor can implement the data export method or the data import method provided in any one of the embodiments of the present disclosure.

[0258] In the process of implementing the computer program product, computer program code for performing the operations of the present disclosure may be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network - including a local area network (LAN) or a wide area network (WAN) - or, alternatively, may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0259] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a portion of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions denoted in the blocks may occur in a different order than that denoted in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system that performs the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0260] The units described in the embodiments of the present disclosure may be implemented in software or in hardware. Among them, the names of the units and modules do not constitute a limitation to the units and modules themselves in some cases.

[0261] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that can be used include: Field Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), Application Specific Standard Parts (ASSP), System on Chip (SOC), Complex Programmable Logic Device (CPLD), and so on.

[0262] In the context of this disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0263] According to one or more embodiments of this disclosure, a data export method is provided, the method including:

[0264] In response to a data export instruction for a first data range, hierarchically splitting the first data range to obtain a second data range with a hierarchical relationship;

[0265] Determining a data list according to the second data range with the hierarchical relationship;

[0266] Exporting data within a third data range to obtain a data file; wherein the third data range includes the second data range located at the bottom layer of the hierarchical relationship;

[0267] Determining an export file according to the data list and the data file.

[0268] According to one or more embodiments of this disclosure, a data export method further includes:

[0269] In some alternative implementations, determining the data list according to the second data range with a hierarchical relationship includes:

[0270] Determining the data list in a tree structure according to the second data range and the hierarchical relationship to which the second data range belongs; wherein, the nodes in the tree structure are used to represent the second data range.

[0271] According to one or more embodiments of the present disclosure, a data export method is provided, which further includes:

[0272] In some alternative implementations, exporting the data within the third data range includes:

[0273] Obtaining the export order of the third data range; wherein, the export order includes a serial export order and / or a parallel export order;

[0274] Exporting the data within the third data range according to the export order.

[0275] According to one or more embodiments of the present disclosure, a data export method is provided, which further includes:

[0276] In some alternative implementations, exporting the data within the third data range includes:

[0277] Determining a corresponding data export task according to the third data range; wherein, the data export task is stored in a first task queue;

[0278] Pulling the data export task from the first task queue by a distributed thread and executing it to export the data within the third data range.

[0279] According to one or more embodiments of the present disclosure, a data export method is provided, which further includes:

[0280] In some alternative implementations, the data within the third data range includes business files and / or structured business data;

[0281] The process of exporting the data within the third data range to obtain the data file includes:

[0282] Storing the business file to obtain the data file; and / or, writing the business data into an embedded database file to obtain the data file.

[0283] According to one or more embodiments of the present disclosure, a data export method is provided, which further includes:

[0284] In some alternative implementations, exporting the data within the third data range includes:

[0285] Exporting the data within the third data range through a preset storage tool.

[0286] According to one or more embodiments of the present disclosure, a data export method is provided, further including:

[0287] In some alternative implementations, determining the export file according to the data list and the data file includes:

[0288] Encrypting the data list and the data file;

[0289] Determining the export file according to the encrypted data list and the encrypted data file.

[0290] According to one or more embodiments of the present disclosure, a data export method is provided, further including:

[0291] In some alternative implementations, obtaining a preset public key;

[0292] Responding to the data export instruction to generate a root key;

[0293] Responding to splitting to obtain the third data range, generating a sub-key corresponding to the third data range; wherein, the sub-key is stored in the data list;

[0294] Correspondingly, encrypting the data list and the data file includes:

[0295] Encrypting the data file corresponding to the third data range through the sub-key to obtain the encrypted data file;

[0296] Encrypting the data list through the root key to obtain the encrypted data list;

[0297] Encrypting the root key through the preset public key to obtain the encrypted root key; wherein, the encrypted root key is transmitted to the target environment along with the export file.

[0298] According to one or more embodiments of the present disclosure, a data export method is provided, further including:

[0299] In some alternative implementations, determining the export file according to the data list and the data file includes:

[0300] Generating a first directory according to the third data range corresponding to the data file;

[0301] Generate a second directory according to the data list, the first directory, and the file name of the data file;

[0302] Determine the export file according to the first directory, the second directory, the data list, and the data file.

[0303] According to one or more embodiments of the present disclosure, a data export method is provided, further including:

[0304] In some alternative implementation manners, in response to the export of data within the third data range, generate a data export log.

[0305] According to one or more embodiments of the present disclosure, a data import method is provided, the method including:

[0306] In response to a data import instruction for the first data range, obtain an import file corresponding to the first data range; wherein, the import file includes a data list and a data file;

[0307] According to the data list, reconstruct a second data range with a hierarchical relationship; wherein, the second data range includes data ranges obtained by hierarchically splitting the first data range in advance;

[0308] Import data in a data file corresponding to a third data range; wherein, the third data range includes the second data range at the bottom layer of the hierarchical relationship.

[0309] According to one or more embodiments of the present disclosure, a data import method is provided, further including:

[0310] In some alternative implementation manners, the importing of data in a data file corresponding to a third data range includes:

[0311] Obtain an import order of a data file corresponding to the third data range; wherein, the import order includes a serial import order and / or a parallel import order;

[0312] Import data in a data file corresponding to a third data range according to the import order.

[0313] According to one or more embodiments of the present disclosure, a data import method is provided, further including:

[0314] In some alternative implementation manners, the importing of data in a data file corresponding to a third data range includes:

[0315] Determine a corresponding data import task according to the third data range; wherein, the data export task is stored in a second task queue;

[0316] Pull the data import task from the second task queue through distributed threads and execute it to import the data in the data file corresponding to the third data range.

[0317] According to one or more embodiments of the present disclosure, a data import method is provided, further including:

[0318] In some optional implementation manners, the importing of the data in the data file corresponding to the third data range includes:

[0319] Import the data in the data file corresponding to the third data range through a preset reading tool.

[0320] According to one or more embodiments of the present disclosure, a data import method is provided, further including:

[0321] In some optional implementation manners, after obtaining the import file, it further includes:

[0322] Decrypt the import file to obtain the data list and the data file.

[0323] According to one or more embodiments of the present disclosure, a data import method is provided, further including:

[0324] In some optional implementation manners, in response to the data import instruction, obtain a preset private key and an encrypted root key;

[0325] The decrypting of the import file includes:

[0326] Decrypt the encrypted root key through the preset private key to obtain the root key;

[0327] Decrypt the encrypted data list through the root key to obtain the data list;

[0328] Obtain a sub-key corresponding to the third data range from the data list;

[0329] Decrypt the encrypted data file corresponding to the third data range through the sub-key to obtain the data file.

[0330] According to one or more embodiments of the present disclosure, a data import method is provided, further including:

[0331] In some optional implementation manners, the import file further includes a first directory and a second directory; wherein, the first directory is generated according to the third data range corresponding to the data file; the second directory is generated according to the data list, the first directory, and the file name of the data file;

[0332] The method further includes:

[0333] By querying the second directory in the import file, obtaining the data list, the first directory, and the file name;

[0334] The importing of the data in the data file corresponding to the third data range includes:

[0335] By querying the first directory and the file name, importing the data in the corresponding data file.

[0336] According to one or more embodiments of the present disclosure, a data import method is provided, which further includes:

[0337] In some alternative implementation manners, in response to an import range selection instruction, determining the third data range from the second data range.

[0338] According to one or more embodiments of the present disclosure, a data import method is provided, which further includes:

[0339] In some alternative implementation manners, in response to the import of the data in the data file corresponding to the third data range, generating a data import log.

[0340] According to one or more embodiments of the present disclosure, a data export device is provided, and the device includes:

[0341] A splitting module, configured to hierarchically split the first data range in response to a data export instruction for the first data range, to obtain a second data range with a hierarchical relationship;

[0342] A list determination module, configured to determine a data list according to the second data range with the hierarchical relationship;

[0343] A data export module, configured to export the data within a third data range to obtain a data file; wherein, the third data range includes the second data range at the bottom layer of the hierarchical relationship;

[0344] An export file determination module, configured to determine an export file according to the data list and the data file.

[0345] According to one or more embodiments of the present disclosure, a data import device is provided, and the device includes:

[0346] An acquisition module, configured to acquire an import file corresponding to the first data range in response to a data import instruction for the first data range; wherein, the import file includes a data list and a data file;

[0347] A reconstruction module, configured to reconstruct a second data range with a hierarchical relationship according to the data list; wherein, the second data range includes a data range obtained by hierarchically splitting the first data range in advance.

[0348] A data import module, configured to import data in a data file corresponding to a third data range; wherein, the third data range includes the second data range at the bottom layer of the hierarchical relationship.

[0349] According to one or more embodiments of the present disclosure, a data export and import system is provided, which includes a data export device provided by an embodiment of the present disclosure and a data import device provided by an embodiment of the present disclosure.

[0350] The above description is only a preferred embodiment of the present disclosure and an explanation of the applied technical principles. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosure concept. For example, a technical solution formed by mutually replacing the above features with (but not limited to) technical features having similar functions disclosed in the present disclosure.

[0351] In addition, although the operations are depicted in a particular order, this should not be construed as requiring that the operations be performed in the particular order shown or in sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although a number of specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present disclosure. Certain features described in the context of separate embodiments may also be implemented combinatorially in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented separately or in any suitable sub-combination in multiple embodiments.

[0352] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A data export method, characterized in that: include: In response to the data export instruction of the first data range, the first data range is hierarchically split to obtain a second data range with a hierarchical relationship; Determine a data list according to the second data range having a hierarchical relationship; Exporting data within a third data range to obtain a data file; wherein the third data range includes the second data range located at the bottom of the hierarchical relationship; An export file is determined according to the data list and the data file.

2. The method according to claim 1, characterized in that The step of determining a data list according to the second data range having a hierarchical relationship includes: The data list of the tree structure is determined according to the second data range and the hierarchical relationship to which the second data range belongs; wherein the nodes in the tree structure are used to represent the second data range.

3. The method according to claim 1, characterized in that: The exporting of data within the third data range includes: Acquire a derivation order of the third data range; wherein the derivation order includes a serial derivation order and / or a parallel derivation order; The data within the third data range is exported according to the export order.

4. The method according to claim 1, characterized in that The exporting of data within the third data range includes: Determine a corresponding data export task according to the third data range; wherein the data export task is stored in the first task queue; The data export task is pulled from the first task queue through a distributed thread and executed to export the data within the third data range.

5. The method according to claim 1, characterized in that: The data within the third data range includes business documents and / or structured business data; The process of exporting the data within the third data range to obtain the data file includes: Storing the business file to obtain the data file; And / or, writing the business data into an embedded database file to obtain the data file.

6. The method according to claim 1, characterized in that The exporting of data within the third data range includes: The data within the third data range is exported through a preset storage tool.

7. The method according to claim 1, characterized in that The step of determining the export file according to the data list and the data file comprises: encrypting the data list and the data file; The export file is determined according to the encrypted data list and the encrypted data file.

8. The method according to claim 7, characterized in that The method further comprises: Get the preset public key; In response to the data export instruction, generating a root key; In response to obtaining the third data range through splitting, generating a subkey corresponding to the third data range; wherein the subkey is stored in the data list; Accordingly, encrypting the data list and the data file includes: Encrypting the data file corresponding to the third data range by using the subkey to obtain the encrypted data file; Encrypting the data list by using the root key to obtain the encrypted data list; The root key is encrypted by the preset public key to obtain an encrypted root key; wherein the encrypted root key is transmitted to the target environment along with the export file.

9. The method according to claim 1, characterized in that: The step of determining the export file according to the data list and the data file comprises: Generate a first directory according to the third data range corresponding to the data file; Generate a second directory according to the data list, the first directory and the file name of the data file; The export file is determined according to the first directory, the second directory, the data list and the data file.

10. The method according to claim 1, characterized in that Also includes: In response to the exporting of the data of the third data range, a data export log is generated.

11. A data import method, characterized in that: include: In response to a data import instruction for a first data range, acquiring an import file corresponding to the first data range; wherein the import file includes a data list and a data file; Reconstructing a second data range with a hierarchical relationship according to the data list; wherein the second data range includes a data range obtained by pre-hierarchically splitting the first data range; Import the data in the data file corresponding to the third data range; wherein the third data range includes the second data range located at the bottom of the hierarchical relationship.

12. The method according to claim 11, characterized in that The importing of data in the data file corresponding to the third data range includes: Obtaining an import order of data files corresponding to the third data range; wherein the import order includes a serial import order and / or a parallel import order; According to the importing sequence, the data in the data file corresponding to the third data range is imported.

13. The method according to claim 11, characterized in that The importing of data in the data file corresponding to the third data range includes: Determine a corresponding data import task according to the third data range; wherein the data export task is stored in the second task queue; The data import task is pulled from the second task queue through a distributed thread and executed to import the data in the data file corresponding to the third data range.

14. The method according to claim 11, characterized in that The importing of data in the data file corresponding to the third data range includes: The data in the data file corresponding to the third data range is imported through a preset reading tool.

15. The method according to claim 11, characterized in that After obtaining the import file, the method further includes: The imported file is decrypted to obtain the data list and the data file.

16. The method according to claim 15, characterized in that The method further comprises: In response to the data import instruction, obtaining a preset private key and an encrypted root key; The decrypting the imported file comprises: Decrypting the encrypted root key using the preset private key to obtain the root key; Decrypting the encrypted data list using the root key to obtain the data list; Acquire a subkey corresponding to the third data range from the data list; The encrypted data file corresponding to the third data range is decrypted using the subkey to obtain the data file.

17. The method according to claim 11, characterized in that The import file further includes a first directory and a second directory; wherein the first directory is generated according to the third data range corresponding to the data file; and the second directory is generated according to the data list, the first directory and the file name of the data file; The method further comprises: By querying the second directory in the imported file, the data list, the first directory and the file name are obtained; The importing of data in the data file corresponding to the third data range includes: By querying the first directory and the file name, the data in the corresponding data file is imported.

18. The method according to claim 11, characterized in that Also includes: In response to an import range selection instruction, the third data range is determined from the second data range.

19. The method according to claim 11, characterized in that Also includes: In response to the import of data in the data file corresponding to the third data range, a data import log is generated.

20. A data export device, characterized in that: include: a splitting module, configured to perform hierarchical splitting on the first data range in response to a data export instruction of the first data range, to obtain a second data range having a hierarchical relationship; A list determination module, used for determining a data list according to the second data range having a hierarchical relationship; A data export module, used for exporting data in a third data range to obtain a data file; wherein the third data range includes the second data range located at the bottom of the hierarchical relationship; The export file determination module is used to determine the export file according to the data list and the data file.

21. A data import device, characterized in that: include: An acquisition module, configured to acquire an import file corresponding to a first data range in response to a data import instruction of the first data range; wherein the import file includes a data list and a data file; A reconstruction module, configured to reconstruct a second data range having a hierarchical relationship according to the data list; wherein the second data range includes a data range obtained by pre-hierarchically splitting the first data range; A data import module is used to import data in a data file corresponding to a third data range; wherein the third data range includes the second data range located at the bottom of the hierarchical relationship.

22. A data export and import system, characterized in that: It includes the data export device as described in claim 20 and the data import device as described in claim 21.

23. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the data export method as described in any one of claims 1-10, or implement the data import method as described in any one of claims 11-19.

24. A storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute the data export method as described in any one of claims 1 to 10, or to implement the data import method as described in any one of claims 11 to 19.

25. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the data export method as described in any one of claims 1 to 19, or implements the data import method as described in any one of claims 11 to 18.

Images