Data import and export method and device, computer equipment and storage medium

Abstract

The invention relates to a data import and export method and device, computer equipment and a storage medium. The method comprises the steps of obtaining to-be-imported data in response to an import instruction, creating at least one import task according to the to-be-imported data, adding the import task into a message queue, processing the at least one import task in parallel through the message queue so as to import the to-be-imported data into a database, and outputting the to-be-imported data in response to an export instruction. The method comprises the following steps: acquiring a pre-configured task template, creating an export task according to the task template, fragmenting the export task through a pre-deployed distributed scheduling framework, and executing each fragmented task fragment to export corresponding data from a database. By adopting the method, the data import and export efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data import and export method, device, computer equipment and storage medium. Background Art

[0002] With the development of data processing technology, data import and export technology has emerged. Data import and export refers to the operation of exchanging data between the SQL (Structured Query Language) Server database system and external systems. Data import is the process of querying or specifying data from an external data source and inserting it into the data table of SQL Server, that is, introducing data from other systems into the SQL Server database. Data export is the process of converting data in the SQL Server database into a data format specified by the user, that is, introducing data from the SQL Server database to other systems. This operation is very important for database administrators and data engineers because it allows them to effectively manage and move data between different systems to ensure data integrity and consistency.

[0003] In traditional data import and export solutions, data is generally processed synchronously and serially, which makes the data import and export inefficient. Summary of the invention

[0004] Based on this, it is necessary to provide a data import and export method, device, computer equipment and storage medium that can improve the efficiency of data import and export in response to the above technical problems.

[0005] A data import and export method, comprising:

[0006] In response to the import instruction, obtaining the data to be imported;

[0007] Creating an import task according to the data to be imported, the import task including at least one;

[0008] Adding the import task to a message queue, and processing at least one import task in parallel through the message queue to import the data to be imported into the database;

[0009] In response to the export instruction, obtaining a pre-configured task template, and creating an export task according to the task template;

[0010] The export task is sharded through a pre-deployed distributed scheduling framework, and each task shard is executed to export the corresponding data from the database.

[0011] In one embodiment, in response to the import instruction, obtaining the data to be imported includes:

[0012] Download the pre-configured import template from the object storage service according to the import instructions;

[0013] Get the data to be imported according to the import template.

[0014] In one embodiment, the data to be imported includes multiple items, and creating an import task according to the data to be imported includes:

[0015] Create corresponding import tasks according to each piece of data to be imported;

[0016] Processing at least one import task in parallel through a message queue to import the data to be imported into a database, including:

[0017] Perform data verification, data analysis and data processing on the data to be imported corresponding to each import task, and import the processed data into the database.

[0018] In one embodiment, in response to an export instruction, obtaining a pre-configured task template, and creating an export task according to the task template include:

[0019] In response to the export instruction, the business scenario to which the data to be exported belongs is determined according to the export instruction through the distributed scheduling framework, and a task template is determined according to the business scenario;

[0020] Determine the fields to be exported and the pre-configured data processing strategy based on the task template;

[0021] Create an export task based on the fields to be exported and the data processing strategy.

[0022] In one embodiment, the export task is segmented by a pre-deployed distributed scheduling framework, and each segmented task segment is executed to export corresponding data from the database, including:

[0023] The distributed scheduling framework is used to obtain the corresponding data to be exported from the database according to the fields to be exported, and the export tasks are sharded according to the data to be exported;

[0024] Allocate each task slice obtained after sharding to each node in the distributed scheduling framework;

[0025] Each node in the distributed scheduling framework processes the data to be exported according to the data processing strategy to export the corresponding data from the database.

[0026] In one embodiment, the data processing strategy includes any one or more of the following:

[0027] Data filtering, data encryption, decryption and data desensitization.

[0028] In one embodiment, the data import and export method further includes:

[0029] The processed data is written into a data table and the data table is stored in the object storage service so that the terminal can query the data table from the object storage service.

[0030] A data import and export device, comprising:

[0031] A first acquisition module, configured to acquire data to be imported in response to an import instruction;

[0032] A creation module, used to create an import task according to the data to be imported, and the import task includes at least one;

[0033] An import module, used to add an import task to a message queue, and process at least one import task in parallel through the message queue to import the data to be imported into the database;

[0034] A second acquisition module is used to obtain a pre-configured task template in response to the export instruction, and create an export task according to the task template;

[0035] The export module is used to slice the export task through the pre-deployed distributed scheduling framework and execute each sliced ​​task to export the corresponding data from the database.

[0036] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0037] In response to the import instruction, obtaining the data to be imported;

[0038] Creating an import task according to the data to be imported, the import task including at least one;

[0039] Adding the import task to a message queue, and processing at least one import task in parallel through the message queue to import the data to be imported into the database;

[0040] In response to the export instruction, obtaining a pre-configured task template, and creating an export task according to the task template;

[0041] The export task is sharded through a pre-deployed distributed scheduling framework, and each task shard is executed to export the corresponding data from the database.

[0042] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0043] In response to the import instruction, obtaining the data to be imported;

[0044] Creating an import task according to the data to be imported, the import task including at least one;

[0045] Adding the import task to a message queue, and processing at least one import task in parallel through the message queue to import the data to be imported into the database;

[0046] In response to the export instruction, obtaining a pre-configured task template, and creating an export task according to the task template;

[0047] The export task is sharded through a pre-deployed distributed scheduling framework, and each task shard is executed to export the corresponding data from the database.

[0048] The above-mentioned data import and export method, device, computer equipment and storage medium, by storing the created import task in the message queue, and consuming the messages in the message queue to realize asynchronous processing of each import task, solves the problem of low data import efficiency caused by synchronous serial execution of each import task in the traditional technology. In addition, when processing the export task, the present application executes each export task through the distributed scheduling framework, and performs sharding processing on the export task through the sharding execution capability of the distributed scheduling framework, thereby improving the efficiency of data export. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 An application environment diagram of a data import and export method in an embodiment;

[0050] Figure 2 A schematic diagram of a flow chart of a data import and export method in one embodiment;

[0051] Figure 3 A schematic diagram of a process of data import in one embodiment;

[0052] Figure 4 A schematic diagram of a process of data importing and exporting in one embodiment;

[0053] Figure 5 It is a structural block diagram of a data import and export device in an embodiment;

[0054] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] Please refer to Figure 1 , Figure 1A schematic diagram of an application environment of a data import and export method provided by an exemplary embodiment of the present application. Figure 1 As shown, the application environment includes a distributed scheduling framework 100, and the distributed scheduling framework 100 includes servers 101, 102, 103 and 104. The servers can be communicatively connected through a network 105 to implement the data import and export method of the present application.

[0057] Server 101 is used to respond to an import instruction, obtain the data to be imported, create an import task based on the data to be imported, the import task includes at least one, add the import task to a message queue, and process at least one import task in parallel through the message queue to import the data to be imported into the database, respond to an export instruction, obtain a pre-configured task template, create an export task based on the task template, shard the export task through a pre-deployed distributed scheduling framework, and execute each task shard after sharding to export corresponding data from the database.

[0058] Server 102, server 103 and server 104 are used to execute each task slice after slicing to export corresponding data from the database.

[0059] Among them, each server can be implemented as an independent server or a server cluster composed of multiple servers.

[0060] The network 105 is used to realize the network connection between various servers. Specifically, the network 105 may include various types of wired or wireless networks.

[0061] In one embodiment, Figure 2 As shown, a data import and export method is provided, which is applied to Figure 1 The server in the example is used to illustrate the following steps:

[0062] Step S11, in response to the import instruction, obtaining the data to be imported.

[0063] The import instruction refers to an instruction to import data into the system. The data to be imported refers to the data uploaded by the user. The data to be imported may include multiple fields and the values ​​of each field. For example, the data to be imported may be a basic information table of students, which may include fields such as the student's gender, height, age, and student ID number, as well as the values ​​of each field.

[0064] Specifically, the user can input the corresponding data to be imported in the terminal interface, and click the virtual import button to trigger the above-mentioned import instruction, and the server obtains the data to be imported uploaded by the user.

[0065] Step S12: creating an import task according to the data to be imported, wherein the import task includes at least one.

[0066] The import task refers to the task of importing the data to be imported into the system.

[0067] Specifically, the data to be imported includes multiple pieces, and the server creates an import task corresponding to each piece of data to be imported, wherein each import task may include information such as the corresponding data to be imported and the import time.

[0068] Step S13, adding the import task to the message queue, and processing at least one import task in parallel through the message queue to import the data to be imported into the database.

[0069] Specifically, the present application pre-sets a message queue for storing various import tasks.

[0070] Step S14, in response to the export instruction, obtaining a pre-configured task template, and creating an export task according to the task template.

[0071] Among them, the export instruction is an instruction to transfer the data in the database to the memory and then convert it into an Excel table.

[0072] A task template is a template required to create an export task. The task template is pre-configured with information such as the fields to be exported.

[0073] Specifically, the present application pre-configures a plurality of task templates, and pre-configures information required for creating an export task in the task templates so that corresponding export tasks can be subsequently created according to the task templates.

[0074] Step S15, slicing the export task through a pre-deployed distributed scheduling framework, and executing each of the sliced ​​task slices to export corresponding data from the database.

[0075] Among them, the distributed scheduling framework is a system that implements task scheduling in a distributed architecture. It involves three key components: distribution, task scheduling and configuration center.

[0076] Distributed: This means that the platform is deployed in a distributed manner, and each node can be horizontally expanded statelessly and infinitely, that is, elastic and dynamic expansion. This architecture allows the system to expand according to demand, improving the scalability and reliability of the system.

[0077] Task scheduling: involves task status management, sending and receiving task scheduling requests, specific task allocation, and specific task execution. This requires the system to be able to perceive the operating status of the entire cluster and schedule and execute tasks according to the configuration.

[0078] Configuration Center: This component can sense the status of the entire cluster and is also the registration place for task information. The role of the configuration center is to provide a centralized management point for managing and coordinating task information in the cluster to ensure the correct allocation and execution of tasks.

[0079] Specifically, the distributed scheduling framework of the present application may be a scheduleX task scheduling framework. ScheduleX is an open source distributed task scheduling framework that uses a distributed architecture and can improve the processing capability and availability of the system through horizontal expansion.

[0080] The above-mentioned server can be one of the nodes in the distributed scheduling framework, and the server and other nodes jointly complete the above-mentioned data export. The export instruction can also be received by the above-mentioned server, and after the export task is created, the export task is sent to the node in the distributed scheduling framework for execution.

[0081] Sharding the export tasks means splitting or grouping the export tasks. For example, when the number of export tasks is large, the export tasks can be sliced ​​to obtain multiple task slices, and each task slice is dispatched to each node in the distributed scheduling framework for execution.

[0082] In one of the embodiments, in response to an import instruction, obtaining the data to be imported includes: downloading a pre-configured import template from an object storage service according to the import instruction, and obtaining the data to be imported according to the import template.

[0083] Specifically, the present application pre-configures an import template, and stores the configured import template in an object storage service OSS (Object Storage Service) or a database.

[0084] Furthermore, the present application pre-sets a data import SDK (Software Development Kit). When the user clicks the virtual import button, the server calls the SDK's API (Application Programming Interface), creates an import task through the SDK, and executes the import task.

[0085] Specifically, the SDK first downloads the import template from the OSS or database, obtains information such as the fields to be imported according to the import template, and obtains the data to be imported according to the import template.

[0086] Among them, the data import SDK of the present application can be connected to any application for use to realize data import.

[0087] In one of the embodiments, the data to be imported includes multiple pieces, and an import task is created based on the data to be imported, including: creating a corresponding import task based on each piece of data to be imported, and processing at least one import task in parallel through a message queue to import the data to be imported into a database, including: performing data verification, data analysis and data processing on the data to be imported corresponding to each import task, and importing the processed data into the database.

[0088] When executing the import task, the SDK obtains the import template and the data to be imported, wherein the import template includes the fields to be imported and the verification rules of each field, and performs data verification on the imported data according to the verification rules, such as verification of data length.

[0089] Data processing means that the data cannot be used directly and needs to be converted into a different format before being imported into the system, such as converting English into Chinese, converting ID numbers into names, etc.

[0090] Furthermore, the present application can also provide prompts for data that failed to be imported.

[0091] Furthermore, the present application can also generate import details, which refer to information such as how many data items have been consumed in the message queue and how many data items have failed to be imported.

[0092] Please refer to Figure 3 , Figure 3 FIG. 1 is a flowchart of data import in one embodiment. Figure 3 In the import task, after the user inputs the data to be imported, the server calls the SDK API to create and process the import task. When processing the import task, the import template is obtained, and data verification, data analysis and data processing are performed according to the import template. The verification includes basic verification and data verification. Basic verification refers to preliminary verification of data, such as data length, etc. Data verification refers to verification of different rules in different business scenarios. After a series of processing, the processed data is stored in the database to complete the data import.

[0093] The beneficial effect of this embodiment is that by adopting the message queue, asynchronous execution of messages is achieved, thereby improving the concurrency of message processing.

[0094] In one of the embodiments, in response to an export instruction, a preconfigured task template is obtained, and an export task is created according to the task template, including: in response to the export instruction, determining the business scenario to which the data to be exported belongs according to the export instruction through a distributed scheduling framework, and determining the task template according to the business scenario, determining the fields to be exported and the preconfigured data processing strategy according to the task template, and creating the export task according to the fields to be exported and the data processing strategy.

[0095] In the present application, multiple business scenarios are pre-set, and a corresponding task template is configured for each business scenario. Different task templates include corresponding fields to be exported and data processing strategies corresponding to each field.

[0096] The export instruction may specifically include an identifier of a business scenario or a scenario name. The server obtains a corresponding task template according to the identifier of the business scenario or the scenario name in the export instruction, and creates an export task according to the task template.

[0097] Specifically, each export task includes fields to be exported and corresponding data processing strategies, wherein the data processing strategy refers to a processing strategy for the exported data, such as data conversion.

[0098] The beneficial effect of this embodiment is that corresponding task templates can be configured according to different business scenarios, and data can be exported according to different task templates, so that data export can be templated and configured, thereby improving the efficiency of data export.

[0099] In one of the embodiments, the export task is sharded through a pre-deployed distributed scheduling framework, and each task slice after sharding is executed to export corresponding data from a database, including: obtaining corresponding data to be exported from a database according to the field to be exported through the distributed scheduling framework, and sharding the export task according to the data to be exported, allocating each task slice obtained after sharding to each node in the distributed scheduling framework, and processing the data to be exported according to the data processing strategy by each node in the distributed scheduling framework to export the corresponding data from the database.

[0100] Specifically, the present application can shard the export task according to the number of data to be exported. For example, assuming that the data to be exported includes 1 million pieces, the export task can be sharded, each task shard includes 100,000 pieces of data, and each task shard is assigned to a corresponding node for execution, and each node only needs to export 100,000 pieces of data.

[0101] Through this embodiment, when the amount of data is large, the export task can be processed in pieces, solving the problem of insufficient memory resources when the amount of data is large. Because data export is specifically the process of converting data from the database to the memory and then to the Excel table, it is necessary to ensure that there are sufficient memory resources to perform data export. When a distributed scheduling framework is adopted, the function of the distributed scheduling framework for sharding execution can be used to assign the export task to each node for execution, so as to solve the problem of insufficient memory resources when a single machine executes.

[0102] Please refer to Figure 4 , Figure 4 FIG. 1 is a flow chart of data export in one embodiment. Figure 4In the figure, the distributed task scheduling center creates an export task and executes the export task. The process in the dotted box is the internal data processing process in the distributed scheduling center. When the distributed scheduling center executes the export task, it obtains the data to be exported according to the export task through the business DAO, and further processes the data to be exported by calling the executor, specifically according to the pre-configured data processing strategy, and finally writes the processed data to the OSS file to complete the data export, where the OSS file refers to the object in the object storage service. In OSS, an object is the basic unit for storing data, and can also be regarded as an OSS file. Business DAO refers to a data access object, which is an application programming interface used to interact with a database.

[0103] In addition, the distributed task scheduling center can also support export status updates and export record queries.

[0104] Furthermore, when it is necessary to export offline data, the data can be exported by calling the offline data export execution engine. Specifically, after obtaining the export task, splice SQL (Structured Query Language), splice the decryption function for the encrypted fields marked in the task template, splice the desensitization function according to the plaintext and desensitized information selected by the business user, splice the filter conditions and export fields, and after the splicing is completed, submit the SQL to write the processed data into the OSS file.

[0105] In one embodiment, the data processing strategy includes any one or more of the following:

[0106] Data filtering, data encryption, decryption and data desensitization.

[0107] Among them, the data processing strategy may specifically include filtering conditions, encryption and decryption functions, and desensitizing functions. When the server performs data filtering, it filters the exported data according to the filtering conditions, encrypts or decrypts the exported data by calling the encryption and decryption function, and desensitizes the exported data by calling the desensitizing function.

[0108] In one of the embodiments, the method further includes: writing the processed data into a data table, and storing the data table into an object storage service, so that the terminal can query the data table from the object storage service.

[0109] Specifically, the processed data to be exported is filled into an Excel table, and the Excel table is written into an OSS file for user query.

[0110] In one embodiment, Figure 5As shown, a data import and export device is provided, comprising: a first acquisition module 51, a creation module 52, an import module 53, a second acquisition module 54 and an export module 55, wherein:

[0111] A first acquisition module 51 is used to acquire the data to be imported in response to the import instruction;

[0112] A creation module 52, used to create an import task according to the data to be imported, and the import task includes at least one;

[0113] An import module 53, used to add an import task to a message queue, and process at least one import task in parallel through the message queue to import the data to be imported into the database;

[0114] A second acquisition module 54 is used to acquire a pre-configured task template in response to the export instruction, and create an export task according to the task template;

[0115] The export module 55 is used to slice the export task through a pre-deployed distributed scheduling framework, and execute each sliced ​​task to export corresponding data from the database.

[0116] In one embodiment, the first acquisition module 51 may download a pre-configured import template from the object storage service according to the import instruction, and acquire the data to be imported according to the import template.

[0117] In one embodiment, the data to be imported includes multiple pieces, and the creation module 52 can create corresponding import tasks according to each piece of data to be imported. The import module 53 can perform data verification, data analysis and data processing on the data to be imported corresponding to each import task, and import the processed data into the database.

[0118] In one embodiment, the second acquisition module 54 can respond to the export instruction, determine the business scenario to which the data to be exported belongs according to the export instruction through the distributed scheduling framework, determine the task template according to the business scenario, determine the fields to be exported and the pre-configured data processing strategy according to the task template, and create an export task according to the fields to be exported and the data processing strategy.

[0119] In one embodiment, the export module 55 can obtain the corresponding data to be exported from the database according to the fields to be exported through a distributed scheduling framework, and slice the export task according to the data to be exported, and distribute the task slices obtained after slicing to each node in the distributed scheduling framework. The data to be exported is processed by each node in the distributed scheduling framework according to the data processing strategy to export the corresponding data from the database.

[0120] In one embodiment, the data processing strategy includes any one or more of the following:

[0121] Data filtering, data encryption, decryption and data desensitization.

[0122] In one of the embodiments, the export module 55 may also write the processed data into a data table, and store the data table into the object storage service, so that the terminal can query the data table from the object storage service.

[0123] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. . The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data import and export method is implemented.

[0124] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented: in response to an import instruction, data to be imported is obtained, an import task is created according to the data to be imported, the import task includes at least one, the import task is added to a message queue, and at least one import task is processed in parallel through the message queue to import the data to be imported into a database, in response to an export instruction, a pre-configured task template is obtained, an export task is created according to the task template, the export task is sharded through a pre-deployed distributed scheduling framework, and each task shard after sharding is executed to export corresponding data from the database.

[0125] In one embodiment, when the processor executes a computer program to implement the above-mentioned step of obtaining the data to be imported in response to the import instruction, the following steps are specifically implemented: according to the import instruction, a pre-configured import template is downloaded from the object storage service, and the data to be imported is obtained according to the import template.

[0126] In one embodiment, the processor executes a computer program to implement the above-mentioned data to be imported including multiple items, and when creating an import task step according to the data to be imported, the following steps are specifically implemented:

[0127] According to each piece of data to be imported, a corresponding import task is created. When the processor executes a computer program to implement the above-mentioned step of processing at least one import task in parallel through a message queue to import the data to be imported into a database, the following steps are specifically implemented: data verification, data analysis and data processing are performed on the data to be imported corresponding to each import task, and the processed data is imported into the database.

[0128] In one embodiment, the processor executes a computer program to implement the above-mentioned response to the export instruction, obtains a pre-configured task template, and creates an export task step according to the task template, and specifically implements the following steps: responding to the export instruction, determining the business scenario to which the data to be exported belongs according to the export instruction through the distributed scheduling framework, and determining the task template according to the business scenario, determining the fields to be exported and the pre-configured data processing strategy according to the task template, and creating an export task according to the fields to be exported and the data processing strategy.

[0129] In one embodiment, when a processor executes a computer program to implement the above-mentioned slicing of the export task through a pre-deployed distributed scheduling framework, and executes each task slice after slicing to export the corresponding data step from the database, the following steps are specifically implemented: obtaining the corresponding data to be exported from the database according to the field to be exported through the distributed scheduling framework, and slicing the export task according to the data to be exported, allocating each task slice obtained after slicing to each node in the distributed scheduling framework, and processing the data to be exported according to the data processing strategy through each node in the distributed scheduling framework to export the corresponding data from the database.

[0130] In one embodiment, the data processing strategy includes any one or more of the following: data filtering, data encryption, decryption, and data desensitization.

[0131] In one embodiment, when the processor executes the computer program, it further specifically implements the following steps: writing the processed data into a data table, and storing the data table in an object storage service so that the terminal can query the data table from the object storage service.

[0132] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: in response to an import instruction, data to be imported is obtained, an import task is created according to the data to be imported, and the import task includes at least one, the import task is added to a message queue, and at least one import task is processed in parallel through the message queue to import the data to be imported into a database, in response to an export instruction, a pre-configured task template is obtained, an export task is created according to the task template, the export task is sharded through a pre-deployed distributed scheduling framework, and each task shard after sharding is executed to export corresponding data from the database.

[0133] In one embodiment, when a computer program is executed by a processor to implement the above-mentioned step of obtaining the data to be imported in response to an import instruction, the following steps are specifically implemented: according to the import instruction, a pre-configured import template is downloaded from the object storage service, and the data to be imported is obtained according to the import template.

[0134] In one embodiment, the computer program is executed by the processor to implement the above-mentioned data to be imported includes multiple items. When creating an import task step according to the data to be imported, the following steps are specifically implemented: creating a corresponding import task according to each item of data to be imported. When the computer program is executed by the processor to implement the above-mentioned step of processing at least one import task in parallel through a message queue to import the data to be imported into a database, the following steps are specifically implemented: performing data verification, data analysis and data processing on the data to be imported corresponding to each import task, and importing the processed data into the database.

[0135] In one embodiment, a computer program is executed by a processor to implement the above-mentioned response to an export instruction, obtain a pre-configured task template, and create an export task step according to the task template, and specifically implement the following steps: respond to the export instruction, determine the business scenario to which the data to be exported belongs according to the export instruction through a distributed scheduling framework, and determine the task template according to the business scenario, determine the fields to be exported and the pre-configured data processing strategy according to the task template, and create an export task according to the fields to be exported and the data processing strategy.

[0136] In one embodiment, when a computer program is executed by a processor to implement the above-mentioned slicing of the export task through a pre-deployed distributed scheduling framework, and executing each task slice after slicing to export the corresponding data step from the database, the following steps are specifically implemented: obtaining the corresponding data to be exported from the database according to the field to be exported through the distributed scheduling framework, and slicing the export task according to the data to be exported, allocating the task slices obtained after slicing to each node in the distributed scheduling framework, and processing the data to be exported according to the data processing strategy through each node in the distributed scheduling framework to export the corresponding data from the database.

[0137] In one embodiment, the data processing strategy includes any one or more of the following: data filtering, data encryption, decryption, and data desensitization.

[0138] In one embodiment, when the computer program is executed by the processor, the following steps are specifically implemented: writing the processed data into a data table, and storing the data table in an object storage service so that the terminal can query the data table from the object storage service.

[0139] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program, and the above-mentioned computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0140] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0141] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A data import and export method, characterized in that: The method comprises: In response to the import instruction, obtaining the data to be imported; Creating an import task according to the data to be imported, the import task including at least one; Adding the import task to a message queue, and processing the at least one import task in parallel through the message queue to import the data to be imported into a database; In response to the export instruction, obtaining a pre-configured task template, and creating an export task according to the task template; The export task is sliced ​​through a pre-deployed distributed scheduling framework, and each of the sliced ​​task slices is executed to export corresponding data from the database.

2. The method according to claim 1, characterized in that The step of obtaining the data to be imported in response to the import instruction includes: Downloading a pre-configured import template from the object storage service according to the import instruction; The data to be imported is obtained according to the import template.

3. The method according to claim 2, characterized in that The data to be imported includes a plurality of items, and the step of creating an import task according to the data to be imported includes: Create corresponding import tasks according to the data to be imported described in each item; The processing of the at least one import task in parallel through the message queue to import the to-be-imported data into a database includes: Data verification, data analysis and data processing are performed on the data to be imported corresponding to each of the import tasks, and the processed data is imported into the database.

4. The method according to claim 1, characterized in that: The step of obtaining a pre-configured task template in response to the export instruction and creating an export task according to the task template includes: In response to the export instruction, determining, through the distributed scheduling framework, the business scenario to which the data to be exported belongs according to the export instruction, and determining the task template according to the business scenario; Determine the fields to be exported and the pre-configured data processing strategy according to the task template; The export task is created according to the to-be-exported fields and the data processing strategy.

5. The method according to claim 4, characterized in that The export task is sliced ​​by using a pre-deployed distributed scheduling framework, and each of the sliced ​​task slices is executed to export corresponding data from the database, including: Obtaining corresponding data to be exported from the database according to the field to be exported through the distributed scheduling framework, and slicing the export task according to the data to be exported; Allocate each task slice obtained after slicing to each node in the distributed scheduling framework; The data to be exported is processed according to the data processing strategy by each node in the distributed scheduling framework to export corresponding data from the database.

6. The method according to claim 5, characterized in that The data processing strategy includes any one or more of the following: Data filtering, data encryption, decryption and data desensitization.

7. The method according to claim 5, characterized in that The method further comprises: The processed data is written into a data table, and the data table is stored in an object storage service so that a terminal can query the data table from the object storage service.

8. A data import and export device, characterized in that: The device comprises: A first acquisition module, configured to acquire data to be imported in response to an import instruction; A creation module, used to create an import task according to the data to be imported, wherein the import task includes at least one; An import module, used for adding the import task to a message queue, and processing the at least one import task in parallel through the message queue to import the data to be imported into a database; A second acquisition module is used to acquire a pre-configured task template in response to the export instruction, and create an export task according to the task template; The export module is used to slice the export task through a pre-deployed distributed scheduling framework and execute each sliced ​​task to export corresponding data from the database.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images