A data transfer method and related device

Abstract

The present application provides a data transfer method and related devices, which relate to the field of software technology, and obtain a data transfer configuration, which includes a data source configuration, a data template, and a storage configuration; verifying the data source based on the data source configuration, and extracting the first original data from the data source after the verification is passed; using the data template to compare and verify the first original data; determining the target data to be transferred in the first original data based on the comparison and verification results, and importing the target data into the target database based on the storage configuration. The present application adds a data quality inspection function in the data transfer process to ensure that the data can be strictly verified and cleaned before the data is imported, which can improve the quality and reliability of the data and provide strong support for the user's business decision-making.

Description

Technical Field

[0001] The present application relates to the field of software technology, and in particular to a data transfer method and related devices. Background Art

[0002] Data plays a vital role in users' daily work, but the sources of data are diverse, and may be stored in file systems, databases, or from third-party service interfaces. This diversity and dispersion of data makes users face many challenges when organizing, viewing, and applying data.

[0003] In order to overcome these difficulties, how to conduct comprehensive quality inspection and transfer of data has become an urgent problem that needs to be solved at this stage. Summary of the invention

[0004] In view of the above problems, the present application provides a data transfer method and related devices to achieve the purpose of comprehensive quality inspection and transfer of data. The specific scheme is as follows:

[0005] A first aspect of the present application provides a data transfer method, the data transfer method comprising:

[0006] Obtain data transfer configuration, which includes data source configuration, data template and storage configuration;

[0007] Verifying the data source based on the data source configuration, and extracting first original data from the data source after the verification passes;

[0008] Using the data template to compare and verify the first original data;

[0009] The target data to be transferred in the first original data is determined according to the comparison and verification result, and the target data is imported into the target database based on the storage configuration.

[0010] In a possible implementation, the data source configuration includes a connection configuration and a verification strategy of the data source; and verifying the data source based on the data source configuration and extracting the first original data from the data source after the verification passes, includes:

[0011] In the case where the data source is an original database, verifying whether the original database is in a connectable state and whether the first data table to be extracted exists based on the connection configuration; if the original database is in a connectable state and the first data table exists, establishing a connection with the original database and generating an SQL query statement according to the verification strategy; running the SQL query statement, and extracting the first original data from the first data table through the connection with the original database;

[0012] In the case where the data source is an interface, verifying whether the interface is in a responsive state based on the connection configuration; if the interface is in a responsive state, constructing an HTTP request according to the verification strategy and sending it to the interface; receiving response data returned by the interface, the response data including a status code and a response body; if the status code indicates that the interface responds successfully, extracting the first original data from the response body;

[0013] When the data source is a file, verify whether the file is in a downloadable and parsable state; if the file is in a downloadable and parsable state, perform a parsing operation on the file according to the verification strategy to obtain the first original data therefrom.

[0014] In a possible implementation, before comparing and verifying the first original data using the data template, the data transfer method further includes:

[0015] A preprocessing operation is performed on the first raw data, where the preprocessing operation includes one or more of data cleaning, conversion, and aggregation.

[0016] In a possible implementation, the data template includes mandatory configuration and data format of the field to be detected; and using the data template to compare and verify the first original data includes:

[0017] Determine the target field content in the first original data that matches the field to be detected;

[0018] The target field content is checked for mandatory nature according to the mandatory configuration, and the target field content is checked for format according to the data format.

[0019] In a possible implementation, the data template further includes the business logic configuration of the field to be detected; and the comparing and verifying the first original data using the data template further includes:

[0020] Perform business logic verification on the target field content according to the business logic configuration.

[0021] In a possible implementation, the importing the target data into the target database based on the storage configuration includes:

[0022] Determining an import type of the target data;

[0023] If the import type is new addition, a new second data table is created in the target database, and the target data is imported into the second data table;

[0024] If the import type is incremental update, determine the third data table to which the first original data belongs in the target database; compare the target data with the second original data in the third data table to determine the duplicate data in the target data that is the same as the second original data, and add a unique identifier to the duplicate data; import the other non-duplicate data in the target data except the duplicate data into the third data table;

[0025] If the import type is full update, determine the fourth data table to which the first original data belongs in the target database; back up the fourth data table and then delete all data therein; import the target data into the fourth data table.

[0026] In a possible implementation, the data transfer method further includes:

[0027] Perform data management, data display and data comparison on the target database.

[0028] In a possible implementation, the data transfer method further includes:

[0029] The target database is monitored, and when sensitive data is detected in the target database, the sensitive data is hidden and an alarm message is output.

[0030] A second aspect of the present application provides a data transfer device, the data transfer device comprising:

[0031] A configuration acquisition module is used to acquire a data transfer configuration, wherein the data transfer configuration includes a data source configuration, a data template, and a storage configuration;

[0032] A data extraction module, configured to verify the data source based on the data source configuration, and extract first original data from the data source after the verification is passed;

[0033] A data verification module, used to compare and verify the first original data using the data template;

[0034] The data import module is used to determine the target data to be transferred in the first original data according to the comparison and verification result, and import the target data into the target database based on the storage configuration.

[0035] A third aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0036] The memory is used to store computer programs;

[0037] The processor is used to execute the computer program so that the electronic device can implement the data transfer method of the first aspect or any implementation method of the first aspect.

[0038] A fourth aspect of the present application provides a computer program product, comprising computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the data transfer method of the first aspect or any implementation of the first aspect.

[0039] A fifth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can use the data transfer method of the first aspect or any implementation method of the first aspect.

[0040] By means of the above technical scheme, the present application provides a data transfer method and related devices, which obtain a data transfer configuration, wherein the data transfer configuration includes a data source configuration, a data template, and a storage configuration; the data source is verified based on the data source configuration, and the first original data is extracted from the data source after the verification is passed; the first original data is compared and verified using the data template; the target data to be transferred in the first original data is determined according to the comparison and verification result, and the target data is imported into the target database based on the storage configuration. The present application adds a data quality inspection function in the data transfer process to ensure that the data can be strictly verified and cleaned before the data is imported, which can improve the quality and reliability of the data and provide strong support for the user's business decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0042] Figure 1 A schematic diagram of a data transfer method provided in an embodiment of the present application;

[0043] Figure 2 A partial flow chart of a data transfer method provided in an embodiment of the present application;

[0044] Figure 3 Another schematic diagram of a data transfer method provided in an embodiment of the present application;

[0045] Figure 4 A schematic diagram of the structure of a data transfer device provided in an embodiment of the present application;

[0046] Figure 5A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0048] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0049] The terms "first", "second" etc. in the specification of the application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable in appropriate circumstances, and this is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0050] In order to meet the diverse needs of different enterprises and organizations for data management and application, this application provides a comprehensive data transfer solution, which will not only support domestic databases to respond to the strategic requirements of information security and independent control, but also have the following features:

[0051] 1. Compatibility of multiple data sources. Some existing data transfer solutions do not support all domestic databases, and these domestic databases occupy an important position in the domestic market, especially in some key industries and fields. Therefore, the data transfer of these domestic databases cannot be supported, which undoubtedly limits their application and promotion in the domestic market. In this regard, the present application provides a data transfer method that can support domestic databases, ensuring that users can import a variety of data into the domestic system.

[0052] 2. Data extraction and processing capabilities. Existing data transfer solutions often only support a single data source and are unable to aggregate data in interfaces and files. However, in actual applications, user data is often scattered across multiple different data sources, including databases, interfaces, and files. Therefore, the present application provides a data transfer method that can support multiple data sources, ensuring that users can easily aggregate data from different data sources, which will greatly improve the user's data management efficiency and reduce the complexity of data integration.

[0053] 3. Data management, display and data comparison capabilities. Currently, most data migration tools only complete the function of data migration, but users need to use database connection tools to view the migrated data, which will bring the following problems: 1) Data tables are scattered, and users sometimes cannot clearly know where the data tables are migrated; 2) Most of the names of data tables are in English, and it is difficult to understand the meaning of the names; 3) There are many types of databases, and some commonly used filtering statements vary greatly between different databases, which will increase the user's learning cost. In this regard, the data transfer method provided in this application can manage and display the imported data, and provide users with filtering and comparison functions, which not only facilitates the user's use, but also reduces the cost of use, and greatly improves the user's work efficiency.

[0054] 4. Comprehensive data quality inspection function. Current data transfer solutions often lack data quality inspection functions, which causes users to find that some data does not meet their needs after the data is imported. This situation not only wastes the user's time and energy, but may also have a negative impact on business decisions. In this regard, the data transfer method provided in this application adds a data quality inspection function to ensure that the data can be strictly verified and cleaned before the data is imported, which will greatly improve the quality and reliability of the data and provide strong support for users' business decisions.

[0055] 5. Efficient and reliable data service. Most database migration tools on the market are slow to transfer tables or libraries with large amounts of data, and there are failures in sharing. In this regard, the data transfer method provided by this application can ensure that the migration speed is very fast while ensuring the successful migration of data, which will greatly save the user's migration time.

[0056] 6. User customized service. The data transfer method provided in this application can provide customized services. Users can directly filter and query fields on the database display page, or filter data through custom SQL statements. This reduces the cost of use for users who are not proficient in SQL statements. On the other hand, custom SQL statements also greatly increase the freedom of query. Users can quickly filter and query the data they need according to their own situation.

[0057] 7. Data security and privacy protection. Data security is the most important point in daily work, but most of the database migration tools on the market are not just simple migrations, and do not consider their security too much. In this regard, the data transfer method provided by this application can encrypt and migrate the fields that users expect to be encrypted (such as ID card number, bank card number, account password, etc.), which can protect the privacy of users.

[0058] See also Figure 1 , Figure 1A flow chart of a data transfer method provided in an embodiment of the present application. Figure 1 As shown, a data transfer method provided in an embodiment of the present application may include steps S10 to S40, and these steps are described in detail below.

[0059] S10, obtaining data transfer configuration, which includes data source configuration, data template and storage configuration.

[0060] In an embodiment of the present application, before executing each data transfer, the user can input a data transfer configuration, which includes a data source configuration, a data template, and a storage configuration, wherein the data source configuration is used to indicate the data source to be connected, the data template is a template for subsequent quality inspection of the extracted data, and the storage configuration is used to indicate the target database for the data to be imported.

[0061] In actual applications, data templates can not only be reused in different projects of the same user to improve the efficiency of data management, but can also be shared with other users to promote team collaboration and convenience of data exchange. By creating and managing data templates, users can establish a more unified monitoring system, better control data quality, and ensure the accuracy and availability of subsequent imported data.

[0062] S20, verifying the data source based on the data source configuration, and extracting first original data from the data source after the verification passes.

[0063] In the embodiment of the present application, the data source configured by the user can be source verified and data quality checked based on the data source configuration. Specifically, during source verification, it is necessary to verify whether the data source provided by the user is valid, such as whether the database can be connected normally, whether the interface can be accessed normally, whether the file can be downloaded normally, etc.; during data quality inspection, the data source will be comprehensively checked, including the format, integrity, consistency, etc. of the data. Only data sources that pass the quality inspection can enter the subsequent data extraction stage. This strict data quality inspection process ensures that the data imported into the target database is high-quality, reliable, and meets the needs of the user.

[0064] After the data source is verified, the original data to be transferred (ie, the first original data) can be extracted from the data source.

[0065] In a possible implementation, verification and data extraction can be performed for different types of data sources. In this regard, an embodiment of the present application provides a data transfer method, wherein the data source configuration includes a connection configuration and a verification strategy of the data source, and step S20 "verifying the data source based on the data source configuration, and extracting the first original data from the data source after the verification passes" may include the following steps:

[0066] In the case where the data source is an original database, verify whether the original database is in a connectable state and whether the first data table to be extracted exists based on the connection configuration; if the original database is in a connectable state and the first data table exists, establish a connection with the original database according to the verification strategy and generate an SQL query statement; run the SQL query statement to extract the first original data from the first data table through the connection with the original database;

[0067] In the case where the data source is an interface, verify whether the interface is in a responsive state based on the connection configuration; if the interface is in a responsive state, construct an HTTP request according to the verification strategy and send it to the interface; receive response data returned by the interface, the response data including a status code and a response body; if the status code indicates that the interface responds successfully, extract the first original data from the response body;

[0068] When the data source is a file, verify whether the file is in a downloadable and parsable state; if the file is in a downloadable and parsable state, perform a parsing operation on the file according to the verification strategy to obtain the first original data therefrom.

[0069] Specifically, in the process of data quality inspection, it is very important to verify the accessibility of the data source. For different types of data sources, the embodiment of the present application can provide a multi-source data extraction algorithm to extract various types of data. For ease of understanding, the following three aspects are explained from the perspectives of database type, interface type, and file type, providing users with a one-stop data solution:

[0070] 1. The data source is the original database. For the database, first ensure that it can be successfully connected to. To this end, parse the connection configuration to obtain the database address, port, user name, password and other connection information, and then verify whether the original database is in a connectable state. Once the connection is successful, it means that the original database is in a connectable state. At this time, further confirm whether the data table to be extracted (i.e. the first data table) exists to avoid errors caused by the non-existence of the table during subsequent queries.

[0071] If the original database is in a connectable state and the first data table exists, a connection is established with the original database according to the verification strategy corresponding to the original database, and an SQL query statement is generated. Then, the SQL query statement is run to extract the first original data from the first data table through the connection with the original database. Specifically, according to the database type (such as MySQL, PostgreSQL, Oracle, Renda Jincang, Damang, etc.) and connection configuration of the original database, a database connection library or driver matching the type of the original database is called to establish a connection with the original database. After the connection is successful, an SQL query statement is generated according to the first data table (which can be a single table or multiple tables) and specific query conditions. The SQL query statement can be a simple SELECT statement or a complex JOIN, GROUP BY, HAVING and other statements. Then, the SQL query statement is run to extract the first original data from the first data table based on the connection established with the original database. In addition, in order to optimize query performance, the embodiment of the present application also supports caching and reuse of query statements.

[0072] 2. The data source is an interface. For interfaces (such as GET, POST, etc.), it is necessary to verify whether the interface can respond to requests normally, including checking the URL, request parameters, request method (such as GET, POST, etc.) of the interface, the status code of the interface response, and the return data format. To this end, the URL, request parameters, and request method are parsed from its connection configuration, and they are checked to determine whether the interface is in a responsive state. If the interface is in a responsive state, an HTTP request (including URL, request header, request body, and other parameters) is constructed according to the interface's verification strategy and sent to the interface. Further, the response data returned by the interface (including the status code and response body) is received. If the status code indicates that the interface responds successfully, the first original data is extracted from the response body. Specifically, for different response formats (such as JSON, XML, etc.), the corresponding parsing library can be called to parse the response body to extract the first original data from it.

[0073] 3. The data source is a file. For a file, first ensure that the file can be downloaded normally and the downloaded file can be correctly parsed. To this end, it is possible to verify whether the file is in a downloadable state and a parsable state. Specifically, the file can be verified by checking the download link and file format of the file to verify whether the file is in a downloadable state, and by checking whether the code logic of the file is correct to verify whether the file is in a parsable state. Further, when the file is in a downloadable state and a parsable state, the file is parsed according to its verification strategy to obtain the first original data. Specifically, the file format is read to call the built-in file parsing library or tool of the program to parse the file. That is to say, after reading the content of the file, the corresponding content can be parsed according to different file formats. For example, for Excel files, worksheets, rows, columns and cell data can be parsed. For another example, for TXT files, text content can be read by row or column. For another example, for JSON files, the hierarchical structure and key-value pairs of JSON objects can be parsed. Of course, there are other corresponding content parsing methods for other file formats, which will not be repeated in the embodiments of the present application.

[0074] On this basis, the first original data may also be subjected to preprocessing operations, including one or more of data cleaning, conversion, and aggregation. Specifically, the first original data may be converted into a specified data format (such as JSON, XML, etc.) according to user requirements, and necessary cleaning, conversion, and aggregation operations may be performed. This allows the data required by users in their daily work to be cleaned and collected together, providing users with more convenient, efficient, and accurate data services, thereby greatly improving the work efficiency and business level of users.

[0075] Therefore, this application provides users with a flexible and efficient data solution by supporting data extraction from multiple database types, file types, and interface types. During the data extraction process, connection configuration, query construction, content parsing, data conversion and other operations can be performed according to user needs to ensure the accuracy and availability of the data. At the same time, it also supports operations such as data cleaning, conversion and aggregation, providing users with a more convenient data processing and analysis experience.

[0076] S30: Use the data template to compare and verify the first original data.

[0077] In the embodiment of the present application, when the user configures the data template as needed, multiple key elements in the data template can be defined. First, the data type to which the data template applies needs to be defined (which can be user information, government data, product data, etc.), and second, the fields to be detected need to be defined in the data template (these fields are usually key information points closely related to business logic), and when defining the fields to be detected, it is also possible to specify which fields are required to ensure the integrity and accuracy of the data. In addition, the data format of the fields to be detected, such as date, number, text, etc., can also be defined in the data template to ensure the consistency and readability of the data.

[0078] Based on the data template, the extracted first original data is compared and verified to determine whether there is data in the first original data that does not meet the requirements of the data template.

[0079] In a possible implementation, the data template includes the mandatory configuration and data format of the field to be detected. Figure 2 , Figure 2 A partial flow chart of a data transfer method provided in an embodiment of the present application. Figure 2 As shown, in the embodiment of the present application, step S30 "using the data template to compare and verify the first original data" may include steps S301 to S302, and these steps are described in detail below.

[0080] S301, determining the target field content matching the field to be detected in the first original data.

[0081] S302, performing mandatory check on the target field content according to the mandatory configuration, and performing format check on the target field content according to the data format.

[0082] To ensure data accuracy and consistency, the structure, content and format of the first original data are verified using a data template. Specifically, in an embodiment of the present application, the fields in the first original data are matched with the fields to be detected in the data template to determine whether the fields in the first original data completely correspond to the fields to be detected, including field names, field order and number of fields, etc. After field matching, the content of the field in the first original data that matches the field to be detected, i.e., the target field content, can be determined.

[0083] In addition, based on the mandatory configuration of the field to be detected in the data template, it can be determined whether the field to be detected is a mandatory item. Based on this, for the mandatory configuration of the field to be detected in the data template, it can be checked whether all these fields are contained in the first original data, and whether the target field content matched by these fields has a valid value, so as to perform mandatory verification. If the mandatory field to be detected is missing in the first original data, or the content of the matched target field is empty, the mandatory verification result is a verification failure. On the contrary, if all the mandatory fields to be detected are contained in the first original data, and the content of the matched target field has a valid value, the mandatory verification result is a verification success.

[0084] In addition, based on the data format of the field to be detected in the data template, the format check can be performed on the matching target field content, specifically including checking the data type (such as integer, floating point number, string, etc.), length limit, special character limit, etc. If there is field content whose data format is not satisfied in the first original data, the format check result is a check failure, otherwise, if there is no field content whose data format is not satisfied in the first original data, the format check result is a check success.

[0085] In practical applications, considering the application of data in business, the first original data can also be checked for business logic. To this end, the data template also includes the business logic configuration of the field to be detected. Accordingly, step S30 "using the data template to compare and check the first original data" can also be as follows:

[0086] Perform business logic verification on the target field content according to the business logic configuration.

[0087] Specifically, in an embodiment of the present application, based on the business logic configuration of the field to be detected in the data template, a business logic check can be performed on the matching target field content, such as whether the land area is greater than 0, whether the process task status is legal, etc.

[0088] In actual applications, corresponding information can also be output according to the results of the comparison and verification. Specifically, if the first original data fully meets the requirements of the data template (i.e., the mandatory verification, format verification, and business logic verification are all successful), the quality inspection passes; if the first original data does not meet the requirements (i.e., at least one of the mandatory verification, format verification, and business logic verification fails), detailed error information is output, including the field name that does not meet the requirements, the error type, etc., so that the user can check and correct it.

[0089] In addition, to achieve performance optimization, for large-scale data comparison and verification, performance can be improved through technical means such as program parallel processing and caching. For example, data can be divided into multiple batches for parallel processing, or verified data can be cached to avoid repeated verification.

[0090] In the data extraction stage, the embodiment of the present application can flexibly extract data from multiple data sources such as files, databases, interfaces, etc. through a multi-source data extraction algorithm. This flexibility enables users to choose the data source that best suits them according to their needs. At the same time, it supports a customized data filtering function, and users can filter out data that meets their own business needs. In addition, considering the security and privacy of the data, multiple encryption algorithms can be provided for users to choose during the extraction process. Users can encrypt sensitive fields, and multiple security measures are taken during data transmission to ensure the security of user data.

[0091] S40, determining the target data to be transferred in the first original data according to the comparison and verification result, and importing the target data into the target database based on the storage configuration.

[0092] In the embodiment of the present application, after obtaining the comparison and verification result of the first original data, the first original data can be transferred as the target data when the comparison and verification result is a successful verification. In order to achieve availability optimization, the data in the first original data that meets the requirements of the data template can also be transferred as the target data when the comparison and verification result is a verification failure. Specifically, in order to ensure that the quality inspection task will not fail due to a small part of non-standard data, the embodiment of the present application can define a fault-tolerant mechanism, and the user can set it according to his actual situation. According to the user's settings, the data that failed the comparison and verification is extracted, and these data are handed over to the user to judge and modify, while the rest of the data can be imported normally. Based on the storage configuration, the database type, import type, and connection information (such as host name, port number, user name, password, etc.) of the imported database (i.e., the target database) can be determined, and then, the target database is verified to exist and whether it can be connected normally. Further, after determining that the target database exists and can be connected normally, consider the user's choice to encrypt those sensitive fields (the encryption algorithm can choose RSA, MD5, etc.), and import the target data into the target database after encryption is completed.

[0093] In actual applications, data cleaning can be performed before importing data into the target database. Specifically, data that failed the comparison and verification or duplicate data in the first original data can be provided to the user for selection. The user can choose to delete the data or modify the data on the page to ensure that the data imported into the database is reasonable and can be used normally.

[0094] The cleaned data can also be formatted and converted into a unified standard format for easy storage. In addition, to ensure data security, consistency checks can be performed on the data before and after import to ensure that the data has not been tampered with during the import process. We also provide a variety of encryption algorithms that can encrypt sensitive fields specified by users to ensure that sensitive information is not leaked.

[0095] In a possible implementation, different import types can be used to store data. Figure 3 , Figure 3 Another flow chart of a data transfer method provided in an embodiment of the present application. Figure 3 As shown, an embodiment of the present application provides a data transfer method, wherein in step S40, "importing target data into a target database based on the storage configuration" may include steps S401 to S404, and these steps are described in detail below.

[0096] S401, determining the import type of target data.

[0097] In the embodiment of the present application, the import type of the first original data can be divided into new additions to the database (referring to the target data being brand new and not existing in the database before. This type of import usually occurs in new business scenarios or projects, and a new data structure needs to be established to carry the new data), incremental updates (referring to only importing new or changed data based on the target data) and full updates (referring to the need to replace all data in the original table. This type of import usually occurs in scenarios where the database needs to be completely updated or reconstructed, such as data migration, system upgrades, etc.).

[0098] S402: If the import type is new addition, create a new second data table in the target database, and import the target data into the second data table.

[0099] In an embodiment of the present application, if the import type is new addition to the database, a new data table needs to be created to store the target data. To this end, the table structure is designed based on the table structure information of the first data table (such as the table name, field name, data type, constraints, etc.). It should be noted that the design of the table structure should comply with the database design specifications and meet business needs.

[0100] According to the designed table structure, use SQL statements to create a new table in the target database, namely the second data table. When creating the second data table, you need to ensure that the target database connection is normal and that there are sufficient permissions to create a new table. Furthermore, after the second data table is successfully created, the target data can be imported into the second data table. There are many ways to import target data. You can use the INSERT INTO statement of the SQL statement to insert data one by one, or you can use batch insertion to import a large amount of data at one time. When importing data, you need to pay attention to the accuracy and completeness of the data to avoid import failures due to errors in data format or content.

[0101] Finally, after the data import is completed, the second data table needs to be verified and tested, including checking the accuracy and completeness of the data to ensure that the second data table can work properly.

[0102] S403, if the import type is incremental update, determine the third data table to which the first original data belongs in the target database; compare the target data with the second original data in the third data table to determine the duplicate data in the target data that is the same as the second original data, and add a unique identifier for the duplicate data; import the other non-duplicate data in the target data except the duplicate data into the third data table.

[0103] In an embodiment of the present application, if the import type is incremental update, first determine the data table (i.e., the third data table) to which the first original data belongs in the target database, and compare the target data with the original data in the third data table (i.e., the second original data) to determine whether there is a duplicate primary key or unique key. If so, determine that there is duplicate data in the target data that is the same as the second original data, add a unique identifier after it, and subsequently provide it to the user for modification or deletion.

[0104] Import the non-duplicate data in the target data except the duplicate data into the third data table. When importing data, ensure that the imported fields match the fields in the third data table and comply with the data types and constraints.

[0105] After the data is imported, the third data table needs to be verified and tested, including checking the accuracy and completeness of the data to ensure that the third data table can work properly and that the newly added or changed data has been correctly imported.

[0106] S404, if the import type is full update, determine the fourth data table to which the first original data belongs in the target database; back up the fourth data table and delete all data therein; import the target data into the fourth data table.

[0107] In the embodiment of the present application, if the import type is a full update, in order to prevent data loss caused by unexpected situations during the update process, the original table needs to be backed up, and the backup can be achieved by exporting the table data to a file or creating a table snapshot. In this regard, first determine the data table (i.e., the fourth data table) to which the first original data belongs in the target database, and after backing up the fourth data table, delete all the data therein. When deleting data, it is necessary to operate with caution to ensure that only the data in the original table is deleted without affecting other tables or database objects.

[0108] After the data in the fourth data table is deleted, the target data is imported into the fourth data table. The import operation is similar to adding data to the database. When importing data, it is necessary to ensure the accuracy and completeness of the data and to comply with the structure and constraints of the original table.

[0109] After the data is imported, the fourth data table needs to be verified and tested, including checking the accuracy and completeness of the data, and whether it is consistent with the backup data (optional), to ensure that the fourth data table can work properly and all data has been imported correctly.

[0110] In a possible implementation, data management and analysis tools can also be provided to help users better manage and utilize the imported data. These tools can help users perform operations such as data query, statistical analysis, and visualization, providing strong support for users' business decisions. In this regard, a data transfer method provided in an embodiment of the present application also includes the following steps:

[0111] Perform data management, data display and data comparison on the target database.

[0112] In the embodiment of the present application, when managing data in the target database, the target database can be classified according to the database type, and the user can be provided with query and deletion functions, so that the user can quickly find the data he needs in daily use.

[0113] When displaying data in the target database, it can support the display of relational data, non-relational data, live video data, and file-based video data. And through the built-in SQL engine, you can search, modify, and delete database-type data on the page without having to learn different SQL syntax. The following is how the SQL engine is implemented:

[0114] Obtain the database type and database connection method, and intelligently match the database connection driver to be used; intelligently generate SQL query statements that match the database according to the query conditions selected by the user or query through the SQL statements filled in by the user; encounter unreasonable query statements and conflicting filter conditions and return specific error information to the front-end page.

[0115] When comparing data in the target database, it supports comparison display of relational databases and non-relational databases. Through the comparison tool, users can see the differences and similarities between tables on the page. The following is how to implement the comparison tool:

[0116] Database connection: Supports connections with a variety of relational and non-relational databases, ensuring that users can seamlessly access and extract the required data.

[0117] Data extraction: Once the connection is established, the user-specified data table or collection can be extracted from the database. For relational databases, information such as table structure, field type, index, etc. can be extracted. For non-relational databases, data such as document structure and key-value pairs in the collection can be extracted.

[0118] Comparison tool: It can intelligently analyze the extracted data and show the differences and similarities between them. For relational databases, it can compare table structures, field differences, data row differences, etc. For non-relational databases, it can compare document structures, key-value pair differences, etc.

[0119] Visual interface: The comparison results are displayed through a user-friendly visual interface. Users can intuitively see the differences and similarities between different data tables or collections on the page, including structural differences, data differences, etc. The interface design is clear and easy for users to understand and operate.

[0120] In a possible implementation, sensitive data in the target database may also be monitored and alarmed. In this regard, a data transfer method provided in an embodiment of the present application further includes the following steps:

[0121] The target database is monitored, and when sensitive data is detected in the target database, the sensitive data is hidden and an alarm message is output.

[0122] In the embodiment of the present application, the import process of the target data can be monitored. If abnormal data is detected in the target database, the abnormal data can be extracted and provided to the user, and the user can modify or delete the abnormal data.

[0123] In addition, the data in the target database can be continuously monitored. If sensitive data appears in the target database, the sensitive data can be hidden and an alarm can be issued to the user, who can then selectively process the sensitive data. Specifically, users can set up security red lines to determine which fields are sensitive (such as account passwords, ID numbers, bank card numbers, etc.), and regularly scan the target database to monitor whether there is data in the sensitive fields involved in the security red lines (i.e. sensitive data) in the target database. If sensitive data exists, the sensitive data will be hidden first and an alarm will be issued to the user. The user can modify or delete the sensitive data. Of course, if it is a false alarm, the user can also mark the data so that there will be no false alarms again in the future.

[0124] Of course, in actual applications, a feedback mechanism can also be established. Once users find sensitive data that is not compliant or not monitored, they can provide feedback, and developers will make further improvements based on user feedback.

[0125] Through the above measures, the accuracy, reliability and security of the data can be ensured, providing a solid foundation for subsequent data analysis and decision-making.

[0126] Based on the above description, the data transfer method provided by the embodiment of the present application has the following advantages:

[0127] 1. Support relational databases (such as Mysql, Postgresql, Oracle), non-relational databases (such as MongoDB) and domestic databases (such as Renda Jincang, DAMO, etc.). Currently, many database migration tools on the market are only adapted for migration between relational data, but most of them do not support domestic databases. This application not only supports domestic databases, but also supports the conversion and storage of non-relational and relational databases.

[0128] 2. It supports filtering, grouping, merging and other operations on data when extracting data from the database. It also supports users to write SQL statements to filter data. Most products on the market only support the migration of original data intact, but the original data may not be completely needed by users. Users need to perform additional filtering operations after obtaining the data. This product provides customized data filtering operations at the initial stage of data extraction, which greatly improves the availability of data.

[0129] 3. This application can support the optimization of large data volumes during data extraction, and greatly shorten the table import time through parallel write operations and high-performance caching during import.

[0130] 4. Support quality inspection and import of data of types such as files, databases, and interfaces. Most of the database import tools currently available on the market are for database-to-database use, and most of the file type imports only support Excel. This application supports parsing and importing data from file data sources such as Excel, TXT, JSON, etc., supports importing between common databases on the market, and supports obtaining data that needs to be imported through interface requests. In addition, the diversified data sources can provide users with more choices and are more in line with users' daily usage scenarios.

[0131] 5. Provide data table management function, support online viewing of imported data, and provide customized query methods and data comparison functions. Currently, the data import tools on the market only support simple data import, but do not support online viewing and comparison of data, as well as data definition query functions. This application allows users to easily find the data table they need from a large number of historical tasks, and can easily find the data they need from the table.

[0132] 6. Provide data encryption. Users can choose which fields to encrypt and the corresponding encryption algorithm, so that the imported information can be desensitized.

[0133] 7. Provide data monitoring function and scan the database regularly to ensure that the data in the database is within the security red line.

[0134] A data transfer method provided in an embodiment of the present application is introduced above, and a device for executing the above data transfer method will be introduced below.

[0135] See also Figure 4 , Figure 4 This is a schematic diagram of the structure of a data transfer device provided in an embodiment of the present application. Figure 4 As shown, the data transfer device provided in the embodiment of the present application includes:

[0136] Configuration acquisition module 10, used to acquire data transfer configuration, the data transfer configuration includes data source configuration, data template and storage configuration;

[0137] The data extraction module 20 is used to verify the data source based on the data source configuration, and extract the first original data from the data source after the verification is passed;

[0138] A data verification module 30, configured to compare and verify the first original data using a data template;

[0139] The data import module 40 is used to determine the target data to be transferred in the first original data according to the comparison and verification result, and import the target data into the target database based on the storage configuration.

[0140] In a possible implementation, the data source configuration includes a connection configuration and a verification strategy of the data source; the data extraction module 20 is specifically used to:

[0141] In the case where the data source is an original database, verify whether the original database is in a connectable state and whether the first data table to be extracted exists based on the connection configuration; if the original database is in a connectable state and the first data table exists, establish a connection with the original database according to the verification strategy and generate an SQL query statement; run the SQL query statement to extract the first original data from the first data table through the connection with the original database;

[0142] In the case where the data source is an interface, verify whether the interface is in a responsive state based on the connection configuration; if the interface is in a responsive state, construct an HTTP request according to the verification strategy and send it to the interface; receive response data returned by the interface, the response data including a status code and a response body; if the status code indicates that the interface responds successfully, extract the first original data from the response body;

[0143] When the data source is a file, verify whether the file is in a downloadable and parsable state; if the file is in a downloadable and parsable state, perform a parsing operation on the file according to the verification strategy to obtain the first original data therefrom.

[0144] In a possible implementation, the data extraction module 20 is further configured to:

[0145] A preprocessing operation is performed on the first original data, where the preprocessing operation includes one or more of data cleaning, conversion, and aggregation.

[0146] In a possible implementation, the data template includes mandatory configuration and data format of the field to be detected; the data verification module 30 is specifically used to:

[0147] Determine the target field content that matches the field to be detected in the first original data; perform a mandatory check on the target field content according to the mandatory configuration; and perform a format check on the target field content according to the data format.

[0148] In a possible implementation, the data template further includes the business logic configuration of the field to be detected; the data verification module 30 is further used to:

[0149] Perform business logic verification on the target field content according to the business logic configuration.

[0150] In a possible implementation, the data import module 40 for importing the target data into the target database based on the storage configuration is specifically used to:

[0151] Determine the import type of the target data; if the import type is new addition, create a new second data table in the target database and import the target data into the second data table; if the import type is incremental update, determine the third data table to which the first original data belongs in the target database; compare the target data with the second original data in the third data table to determine the duplicate data in the target data that is the same as the second original data, and add a unique identifier to the duplicate data; import the other non-duplicate data in the target data except the duplicate data into the third data table; if the import type is full update, determine the fourth data table to which the first original data belongs in the target database; back up the fourth data table and delete all data therein; import the target data into the fourth data table.

[0152] In a possible implementation, the data import module 40 is further used for:

[0153] Perform data management, data display and data comparison on the target database.

[0154] In a possible implementation, the data import module 40 is further used for:

[0155] The target database is monitored, and when sensitive data is detected in the target database, the sensitive data is hidden and an alarm message is output.

[0156] It should be noted that the detailed functions of each module in the embodiment of the present application can be found in the corresponding public part of the above-mentioned data transfer method embodiment, and will not be repeated here.

[0157] The present application also provides an electronic device in an embodiment. Figure 5 , Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptop computers, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 5 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0158] like Figure 5As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 to a random access memory (RAM) 503. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 503. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0159] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a memory card, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0160] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the data transfer methods provided in the embodiments of the present application.

[0161] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any data transfer method provided in an embodiment of the present application.

[0162] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0163] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0164] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0165] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A data transfer method, characterized in that: The data transfer method comprises: Obtain data transfer configuration, which includes data source configuration, data template and storage configuration; Verifying the data source based on the data source configuration, and extracting first original data from the data source after the verification passes; Using the data template to compare and verify the first original data; The target data to be transferred in the first original data is determined according to the comparison and verification result, and the target data is imported into the target database based on the storage configuration.

2. The data transfer method according to claim 1, characterized in that: The data source configuration includes a connection configuration and a verification strategy of the data source; the verifying the data source based on the data source configuration and extracting the first original data from the data source after the verification passes, includes: In the case where the data source is an original database, verifying whether the original database is in a connectable state and whether the first data table to be extracted exists based on the connection configuration; if the original database is in a connectable state and the first data table exists, establishing a connection with the original database and generating an SQL query statement according to the verification strategy; running the SQL query statement, and extracting the first original data from the first data table through the connection with the original database; In the case where the data source is an interface, verifying whether the interface is in a responsive state based on the connection configuration; if the interface is in a responsive state, constructing an HTTP request according to the verification strategy and sending it to the interface; receiving response data returned by the interface, the response data including a status code and a response body; if the status code indicates that the interface responds successfully, extracting the first original data from the response body; When the data source is a file, verify whether the file is in a downloadable and parsable state; if the file is in a downloadable and parsable state, perform a parsing operation on the file according to the verification strategy to obtain the first original data therefrom.

3. The data transfer method according to claim 1, characterized in that: Before comparing and verifying the first original data using the data template, the data transfer method further includes: A preprocessing operation is performed on the first raw data, where the preprocessing operation includes one or more of data cleaning, conversion, and aggregation.

4. The data transfer method according to claim 1, characterized in that: The data template includes mandatory configuration and data format of the field to be detected; and using the data template to compare and verify the first original data includes: Determine the target field content in the first original data that matches the field to be detected; The target field content is checked for mandatory nature according to the mandatory configuration, and the target field content is checked for format according to the data format.

5. The data transfer method according to claim 4, characterized in that: The data template also includes the business logic configuration of the field to be detected; and the comparison and verification of the first original data using the data template also includes: Perform business logic verification on the target field content according to the business logic configuration.

6. The data transfer method according to claim 1, characterized in that: The step of importing the target data into a target database based on the storage configuration includes: Determining an import type of the target data; If the import type is new addition to the database, a new second data table is created in the target database, and the target data is imported into the second data table; If the import type is incremental update, determine the third data table to which the first original data belongs in the target database; compare the target data with the second original data in the third data table to determine the duplicate data in the target data that is the same as the second original data, and add a unique identifier to the duplicate data; import the other non-duplicate data in the target data except the duplicate data into the third data table; If the import type is full update, determine the fourth data table to which the first original data belongs in the target database; back up the fourth data table and then delete all data therein; import the target data into the fourth data table.

7. The data transfer method according to claim 1, characterized in that: The data transfer method further includes: Perform data management, data display and data comparison on the target database.

8. The data transfer method according to claim 1, characterized in that: The data transfer method further includes: The target database is monitored, and when sensitive data is detected in the target database, the sensitive data is hidden and an alarm message is output.

9. A data transfer device, characterized in that: The data transfer device comprises: A configuration acquisition module is used to acquire a data transfer configuration, wherein the data transfer configuration includes a data source configuration, a data template, and a storage configuration; A data extraction module, configured to verify the data source based on the data source configuration, and extract first original data from the data source after the verification is passed; A data verification module, used to compare and verify the first original data using the data template; The data import module is used to determine the target data to be transferred in the first original data according to the comparison and verification result, and import the target data into the target database based on the storage configuration.

10. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the data transfer method as described in any one of claims 1 to 8.

Images