Data import method, device, equipment and computer storage medium

Abstract

The present invention relates to the technical field of financial technology (Fintech), and discloses a data import method, which includes: determining a target model mode for data to be imported, and controlling the application to read and write the backup cluster according to the target model mode; importing the master data in the data warehouse to the master cluster, and detecting whether the master cluster has completely received the master data; if the master cluster has completely received the master data, controlling the application to only read and write the master cluster, and importing the backup data in the data warehouse to the backup cluster, detecting whether the backup cluster has completely received the backup data; if the backup cluster has completely received the backup data, controlling the application to restore the default read-write mode. The present invention also discloses a data import device, equipment and a computer storage medium. The present invention improves the stability of the service when importing data.

Description

Technical Field

[0001] The present invention relates to the field of testing technology of financial technology (Fintech), and in particular to a data import method, device, equipment and computer storage medium. Background Art

[0002] With the development of computer technology, more and more technologies (big data, distributed, blockchain, artificial intelligence, etc.) are being applied in the financial field. The traditional financial industry is gradually transforming to financial technology (Fintech), but due to the security and real-time requirements of the financial industry, higher requirements are also placed on technology. Due to the explosive growth of data on the Internet, there is an urgent need to store massive data. However, since traditional MySQL (relational database management system) and Oracle (relational database) can only provide good transaction capabilities and read and write performance for millions of data, when the data exceeds millions, its performance drops sharply. In addition, for MySQL, if the client wants to obtain better performance, it is necessary to divide the db (database) into libraries and tables, which invisibly increases the complexity of the business and is inconvenient for subsequent maintenance and expansion, thus making the service stability worse. Therefore, how to improve the stability of the service when importing data has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The main purpose of the present invention is to propose a data import method, device, equipment and computer storage medium, aiming to solve how to improve the stability of the service during data import, which has become a technical problem that needs to be solved urgently.

[0004] To achieve the above object, the present invention provides a data import method, which is applied to a data import system, wherein the data import system includes an application, a main cluster, a standby cluster and a data warehouse, and the data import method includes the following steps:

[0005] Determine a target model mode for data to be imported, and control the application to read and write the standby cluster according to the target model mode;

[0006] Importing the master data in the data warehouse into the master cluster, and detecting whether the master cluster has completely received the master data;

[0007] If the primary cluster has completely received the primary data, control the application to read and write only the primary cluster, and import the backup data in the data warehouse into the backup cluster, and detect whether the backup cluster has completely received the backup data, wherein the data warehouse includes primary data and backup data;

[0008] If the standby cluster has completely received the standby data, the application is controlled to restore the default read-write mode.

[0009] Optionally, the step of importing the master data in the data warehouse into the master cluster includes:

[0010] Data in the data warehouse is cleaned to obtain target data in a data format supported by the primary cluster, the target data is used as primary data, and the target data is imported into the primary cluster.

[0011] Optionally, the step of importing the target data into the main cluster includes:

[0012] Encapsulating the data source of the target data to obtain encapsulated data;

[0013] The encapsulated data is stored in a HDFS temporary directory file, and the HDFS temporary directory file is moved to the main cluster.

[0014] Optionally, the step of performing data cleaning on the data in the data warehouse to obtain target data in a data format supported by the primary cluster includes:

[0015] Determine the intermediate data or the full historical data in the data warehouse, and perform data cleaning on the intermediate data or the full historical data to obtain cleaned data;

[0016] The data format supported by the main cluster is determined, the format of the cleaned data is converted according to the data format supported by the main cluster, and the cleaned data after the format conversion is used as the target data having the data format supported by the main cluster.

[0017] Optionally, the step of importing the standby data in the data warehouse into the standby cluster includes:

[0018] The storage format of the standby cluster is determined, and the standby data in the data warehouse is format-converted according to the storage format, and the format-converted standby data is imported into the standby cluster.

[0019] Optionally, the step of determining the target model mode for data import includes:

[0020] Detecting a field value in the application, and determining an available model mode from a plurality of preset model modes according to the field value, and detecting whether the available model mode is a read-only standby cluster mode;

[0021] If so, the available model mode is used as the target model mode for data import.

[0022] Optionally, if the standby cluster has completely received the standby data, the step of controlling the application to restore the default read-write mode includes:

[0023] If the standby cluster has completely received the standby data, then detecting whether the integrity of the data of the active cluster is qualified, and detecting whether the integrity of the data of the standby cluster is qualified;

[0024] If the data detection of the primary cluster is qualified and the data detection of the standby cluster is qualified, the application is controlled to restore the default read-write mode.

[0025] In addition, to achieve the above-mentioned purpose, the present invention also provides a data importing device, the data importing device comprising:

[0026] A determination module, used to determine a target model mode for data import, and control the application program to read and write the standby cluster according to the target model mode;

[0027] A detection module, used for importing the master data in the data warehouse into the master cluster, and detecting whether the master cluster has completely received the master data;

[0028] An import module, used to control the application to read and write only the primary cluster if the primary cluster has completely received the primary data, and import the backup data in the data warehouse to the backup cluster, and detect whether the backup cluster has completely received the backup data, wherein the data warehouse includes primary data and backup data;

[0029] The recovery module is used to control the application to recover the default read-write mode if the standby cluster has completely received the standby data.

[0030] In addition, to achieve the above-mentioned purpose, the present invention also provides a data import device, which includes: a memory, a processor, and a data import program stored in the memory and executable on the processor, and when the data import program is executed by the processor, the steps of the data import method described above are implemented.

[0031] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer storage medium, on which a data import program is stored, and when the data import program is executed by a processor, the steps of the data import method described above are implemented.

[0032] The present invention determines the target model mode of data to be imported, controls the application to read and write the standby cluster according to the target model mode; imports the main data in the data warehouse into the main cluster, detects whether the main cluster has completely received the main data; if the main cluster has completely received the main data, controls the application to read and write only the main cluster, and imports the standby data in the data warehouse into the standby cluster, detects whether the standby cluster has completely received the standby data; if the standby cluster has completely received the standby data, controls the application to restore the default read-write mode. By controlling the application to read and write only the standby cluster according to the determined target model mode, and importing the main data in the data warehouse into the main cluster, and after the main data is completely imported, the application is allowed to read and write only the main cluster, import the standby data in the data warehouse into the standby cluster, and restore the default read-write mode after the import is completed, thereby avoiding the phenomenon of simultaneous reading, writing and importing of the main cluster and the standby cluster in the prior art, making the service of the application unstable, and by reading the main cluster and the standby cluster separately, it is ensured that the data import is uninterrupted while the application can read the data normally, thereby improving the stability of the service. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a schematic diagram of the structure of a data import device in a hardware operating environment involved in an embodiment of the present invention;

[0034] Figure 2 This is a flow chart of a first embodiment of a data import method of the present invention;

[0035] Figure 3 A schematic diagram of a device module of a data import device of the present invention;

[0036] Figure 4 It is a schematic diagram of the process of the data import method of the present invention;

[0037] Figure 5 It is a schematic diagram of the data reading and writing process in the data import method of the present invention.

[0038] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0039] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0040] like Figure 1 As shown, Figure 1 It is a schematic diagram of the data import device structure of the hardware operating environment involved in the embodiment of the present invention.

[0041] The data import device in the embodiment of the present invention may be a PC or a server device, on which a Java virtual machine runs.

[0042] like Figure 1 As shown, the data import device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0043] Those skilled in the art will understand that Figure 1 The data import device structure shown in the figure does not constitute a limitation of the device, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0044] like Figure 1 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a data import program.

[0045] exist Figure 1 In the data import device shown, the network interface 1004 is mainly used to connect to the background server and communicate data with the background server; the user interface 1003 is mainly used to connect to the client (user end) and communicate data with the client; and the processor 1001 can be used to call the data import program stored in the memory 1005 and perform the operations in the following data import method.

[0046] Based on the above hardware structure, an embodiment of the data import method of the present invention is proposed.

[0047] Reference Figure 2 , Figure 2 This is a flow chart of a first embodiment of a data import method of the present invention, the method comprising:

[0048] Step S10, determining a target model mode for data to be imported, and controlling the application to read and write the standby cluster according to the target model mode;

[0049] In this embodiment, the data import method is applied to a data import system, and the data import system includes an application, a main cluster, a standby cluster and a data warehouse. And the relevant terms in this embodiment are explained as follows: Hive is a data warehouse tool based on Hadoop, which can perform data sorting, special query and analysis processing on data sets stored in files on HDFS, and provides a query language similar to SQL language - HiveQL, which can implement simple MR statistics through HQL statements. Hive converts HQL statements into MR tasks for execution. Hadoop is a software framework that can perform distributed processing on large amounts of data. And Hadoop includes four modules, namely Common: a common tool that supports other modules; HDFS: a distributed file system that provides high throughput access performance; YARN: a framework that provides job scheduling and cluster resource management; MapReduce: a large data parallel computing framework, referred to as MR. Hbase is an open source non-relational distributed database (NoSQL). And Hbase's data storage management includes two aspects: one is metadata management, and the other is data management. The metadata is the corresponding region information stored in the meta table by Hbase. The client will first read the meta table to find the server where the corresponding region is located, and then directly perform read and write operations through RPC. And Hbase data is stored in HDFS in a strict lexicographic order, with hfile as the smallest file unit.

[0050] And this embodiment is mainly based on TDSQL's QS multi-mode query, and 4 model modes can be set, including master read-only master database, slave read-only standby database, sync_all reads the master database first, if the timeout reads the standby database or the standby times out reads the master; and async_all reads the master standby database at the same time, and the first return is used first. Therefore, in this embodiment, it is necessary to first determine the model mode to be imported into the data, and use it as the target model mode, and when the target model mode is the slave read-only standby database, the application is controlled to read and write the standby cluster only according to the target model mode. Before importing data, data cleaning is required before mode selection. Because when doing big data applications, hive is generally used as a data warehouse, acting on the intermediate link of data processing, and hive is a batch append operation, which does not support random updates. Therefore, in the big data solution, hive (data warehouse) will be used as a data warehouse to store and clean intermediate data or historical full data, and when random online queries are required, SQL can be used to run MR tasks, multiple ods, dm and other tables can be calculated, and extracted and converted into the underlying storage format hfile required by hbase. Among them, the role of data cleaning is to provide the corresponding data source and provide data support for the operation of subsequent model modes. After data cleaning and selecting the target model mode, the master record field will be updated to importing, and the business system will be notified to adopt the target model mode, switch the online traffic to the backup cluster, and record the update status of the current master cluster, so as to avoid the phenomenon of disk IO, CPU and process resources competing with each other caused by the derivative of the main cluster, which affects the online business.

[0051] Step S20, importing the master data in the data warehouse into the master cluster, and detecting whether the master cluster has completely received the master data;

[0052] After the target model mode is determined, the client application will switch to read and write slave clusters, and the read and write operations of the current master cluster will be suspended. At the same time, the data that has been cleaned in the data warehouse (i.e., master data) will be imported into the master cluster in batches, that is, the data in hive (data warehouse) will be generated to the hdfs temporary directory of the master hbase cluster through the MR program, and the hfile temporary directory file will be moved to the master cluster. That is, the data source corresponding to the data in the data warehouse is uploaded to HDFS for storage, and then the data source on HDFS is read in the program, custom encapsulation is performed, RowKey is assembled, and the encapsulated data is written back to HDFS, and stored in the directory specified by HDFS in the form of HFile. Then the master cluster gc status and current merge status will be checked, and the master field status in the current database db will be updated to done (completed), and the read_type field will be updated to master. That is, it will detect whether the master cluster has completely received the master data, and verify the current cluster's operating status and resource usage, and then perform different operations according to different detection results. Among them, the master data is the data to be imported into the master cluster after data cleaning in the data warehouse.

[0053] Step S30: If the primary cluster has completely received the primary data, control the application to read and write only the primary cluster, and import the backup data in the data warehouse into the backup cluster, and detect whether the backup cluster has completely received the backup data, wherein the data warehouse includes primary data and backup data;

[0054] When it is found that the master cluster has completely received the master data, the target model mode will be changed actively, that is, from the slave read-only standby database to the master read-only master database, and the db slave field status will be updated to the importing status. Then, according to the changed model mode, the client application is switched to read and write the master cluster, and the read and write operations of the current slave cluster are suspended. Since the master cluster has now recovered and can provide services, the slave standby cluster is ready to perform input and import operations, that is, the standby data in the data warehouse is imported into the standby cluster through the MR program, that is, the hive data is assembled into the storage format required by the target slave cluster, and imported into the slave cluster hdfs temporary directory, and this hdfs temporary directory is moved to the slave standby cluster. Then check the GC status of the salve cluster and the current merge status, and update the TDSQL db slave status record to done. That is, check whether the standby cluster has completely received the standby data, and perform different operations according to different detection results. Among them, the data warehouse includes master data and standby data. Standby data is the data in the data warehouse to be imported into the standby cluster after data cleaning.

[0055] Step S40: If the standby cluster has completely received the standby data, control the application to restore the default read-write mode.

[0056] When it is determined that the standby cluster has completely received the standby data, the application can be controlled to restore the default read-write mode, that is, the master reads only the primary database, the slave reads only the standby database, sync_all reads the primary database first, and if it times out, reads the standby database or the standby reads the primary database after a timeout; and async_all reads the primary and standby databases at the same time, and returns first to use any of the four model modes. The specific default read-write mode can be defined according to user needs.

[0057] In addition, to help understand the principle of batch data import in this embodiment, an example is given below.

[0058] For example, Figure 4 As shown in the figure, the infrastructure of the data import system is DB online source, HIVE data warehouse (including HDFS (distributed system), Database), Yarn MR model (including mappers Mapper1, Mapper2 and Mapper3, and reducers Reducer1, Reducer2, Reducer3, Reducer4 and Reducer5), DB: TDSQL (distributed database), HBASE cluster source (including HDS temporary directory hfile, RS1, Region1, Region2, Region3) and client Online system. The steps for bulk data import are: ① perform data cleaning; ② update field values ​​read_type: slave, master: importing; ③ determine the salve mode of the client system; ④ generate the target hfile (including Complete bulkload and move files); ⑤ check the cluster and modify master: done; ⑥ determine the master mode and use the same operation steps for another cluster; ⑦ update the field values ​​read_type: master, slave: importing; ⑧ generate the target hfile (including Complete bulkload and movefiles); ⑨ check the cluster and modify slave: done; ⑩ the client system restores the default read-write mode.

[0059] In this embodiment, the steps for cluster read and write operations can be as follows: Figure 5As shown, QS (multi-mode query) reads data from the primary cluster or the standby cluster, and periodically accesses the storage memory to TDSQL (distributed database), while the primary cluster SS and ETL write data to the primary cluster, and similarly, the standby cluster SS and ETL write data to the standby cluster.

[0060] In this embodiment, by determining the target model mode for data import, the application is controlled to read and write the standby cluster according to the target model mode; the master data in the data warehouse is imported into the master cluster, and the master data is detected whether the master cluster has completely received the master data; if the master cluster has completely received the master data, the application is controlled to read and write only the master cluster, and the standby data in the data warehouse is imported into the standby cluster, and the standby data is detected whether the standby cluster has completely received the standby data; if the standby cluster has completely received the standby data, the application is controlled to restore the default read-write mode. By controlling the application to read and write only the standby cluster according to the determined target model mode, and importing the master data in the data warehouse into the master cluster, and after the master data is completely imported, the application is allowed to read and write only the master cluster, and the standby data in the data warehouse is imported into the standby cluster, and the default read-write mode is restored after the import is completed, thereby avoiding the phenomenon of simultaneous reading, writing and importing of the master cluster and the standby cluster in the prior art, which makes the service of the application unstable. By reading the master cluster and the standby cluster separately, it is ensured that the data import is uninterrupted while the application can read the data normally, thereby improving the stability of the service.

[0061] Further, based on the first embodiment of the data import method of the present invention, a second embodiment of the data import method of the present invention is proposed. This embodiment is a refinement of step S20 of the first embodiment of the present invention, which is a step of importing the master data in the data warehouse into the master cluster, and includes:

[0062] Step a: clean the data in the data warehouse to obtain target data in a data format supported by the main cluster, use the target data as the main data, and import the target data into the main cluster.

[0063] When doing big data applications, hive is generally used as a data warehouse, acting in the middle of data processing, and hive is a batch append operation and does not support random updates. Therefore, in the big data solution, hive (data warehouse) will be used as a data warehouse to store and clean intermediate data or historical full data, and when random online queries are required, you can use SQL to run MR tasks, perform data calculations on multiple ods, dm and other tables, and extract and convert them into the underlying storage format hfile required by hbase. Among them, the role of data cleaning is to provide the corresponding data source and provide data support for the subsequent operation of each model mode. Therefore, after data cleaning, after obtaining the target data in the data format supported by the main cluster, the target data can be imported into the main cluster as the main data.

[0064] In this embodiment, by cleaning the data in the data warehouse, and then importing the target data in a data format supported by the main cluster as the main data into the main cluster, the stability of the data imported into the main cluster is ensured.

[0065] Specifically, the step of importing the target data into the main cluster includes:

[0066] Step b, encapsulating the data source of the target data to obtain encapsulated data;

[0067] When importing target data into the main cluster, you can upload the data source of the target data to HDFS for storage through MR, then read the stored data source in HDFS and perform custom packaging to obtain the packaged data.

[0068] Step c: storing the encapsulated data in a HDFS temporary directory file, and moving the HDFS temporary directory file to the main cluster.

[0069] After the encapsulated data is obtained, it will be written back to HDFS and stored in the form of HFile in the directory file specified by HDFS, that is, the HDFS temporary directory file, and this HDFS temporary directory file will be moved to the main cluster to complete the operation of importing the target data into the main cluster.

[0070] In this embodiment, the target data is encapsulated, the encapsulated data is stored in a temporary directory file of HDFS, and then moved to the main cluster, thereby ensuring that the main cluster obtains effective integrity of the target data.

[0071] Specifically, the step of performing data cleaning on the data in the data warehouse to obtain target data in a data format supported by the main cluster includes:

[0072] Step d, determining the intermediate data or the historical full data in the data warehouse, and performing data cleaning on the intermediate data or the historical full data to obtain cleaned data;

[0073] In this embodiment, Hive is used as the data warehouse. Since Hive is a batch append operation and does not support random update operations, it is necessary to determine the intermediate data or historical full data already in the data warehouse and perform data cleaning on the intermediate data or historical full data, that is, to clean up the useless data of the primary cluster and the backup cluster, and use the remaining data in the data warehouse as cleaned data.

[0074] Step e: determining a data format supported by the main cluster, performing format conversion on the cleaned data according to the data format supported by the main cluster, and using the cleaned data after format conversion as target data.

[0075] After obtaining the cleaned data, it is also necessary to determine the data format that the main cluster can support, and convert the data format of the cleaned data into a data format supported by the main cluster, and use the cleaned data after format conversion as the target data with a data format supported by the main cluster.

[0076] In this embodiment, the data warehouse is cleaned to obtain cleaned data, and the data format of the cleaned data is converted into a data format supported by the main cluster, which is then used as the target data, thereby ensuring the accuracy of the obtained target data.

[0077] Furthermore, the step of importing the standby data in the data warehouse into the standby cluster includes:

[0078] Step f: determining the storage format of the standby cluster, performing format conversion on the standby data in the data warehouse according to the storage format, and importing the standby data after format conversion into the standby cluster.

[0079] In this embodiment, when importing the backup data in the data warehouse to the backup cluster, it is necessary to first determine the storage format supported by the backup cluster, convert the format of the backup data in the data warehouse to the storage format supported by the backup cluster, and then import the backup data after the format conversion into the backup cluster. The backup data is the data in the data warehouse after data cleaning and data import to the main cluster.

[0080] In this embodiment, by determining the storage format of the standby cluster, converting the format of the standby data in the data warehouse into the storage format of the standby cluster, and then importing the data into the standby cluster, the stable import of data into the standby cluster is ensured.

[0081] Furthermore, the step of determining the target model mode for data import includes:

[0082] Step g, detecting a field value in the application, and determining an available model mode from a plurality of preset model modes according to the field value, and detecting whether the available model mode is a read-only standby cluster mode;

[0083] In this implementation, different field values ​​are defined in the client system for different model modes, such as master, slave, sync_all, and async_all modes. For example, when the field value is master or slave, only one cluster provides services, and the other cluster is busy. Therefore, in this embodiment, the available model mode (i.e., the model mode corresponding to the current field value) can be determined from the preset multiple model modes according to the currently existing field values ​​in the application, and it can be detected whether the available model mode is a read-only standby cluster mode, and different operations can be performed according to different detection results.

[0084] Step h: If yes, the available model mode is used as the target model mode for data import.

[0085] When it is determined that the available model mode is the read-only standby cluster mode, the available model mode is used as the target model mode, and the steps of controlling the application to read-only and write the standby cluster according to the target model mode can be performed. However, if the available model mode is not the read-only standby cluster mode, but other model modes, the operations corresponding to the other model modes are performed. For example, if the other model mode is the read-only primary cluster, the primary cluster can be read-only and written according to the target model mode.

[0086] In this embodiment, the available model mode is determined according to the field value in the application program, and when the available model mode is a read-only cluster mode, the available model mode is used as the target model mode, thereby ensuring the accuracy of the acquired target model mode.

[0087] Further, if the standby cluster has completely received the standby data, the step of controlling the application to restore the default read-write mode includes:

[0088] Step m: if the standby cluster has completely received the standby data, then detecting whether the integrity of the data of the active cluster is qualified, and detecting whether the integrity of the data of the standby cluster is qualified;

[0089] When it is determined that the standby cluster has completely received the standby data, the main cluster and the standby cluster can be tested, that is, the integrity of the data imported into the main cluster is tested to see whether it is qualified, and the integrity of the data imported into the standby cluster is tested to see whether it is qualified, and different operations are performed according to different test results. It should be noted that in addition to testing the integrity of the main cluster data and the integrity of the standby cluster data, the operation status of the main cluster and the operation status of the standby cluster can also be tested.

[0090] Step n: if the data detection of the primary cluster is qualified and the data detection of the standby cluster is qualified, control the application to restore the default read-write mode.

[0091] When it is found that the data of the primary cluster is qualified and the data of the standby cluster is qualified, the application can be controlled to restore the default read-write mode. However, if the data of the primary cluster fails to be qualified or the data of the standby cluster fails to be qualified, the import operation needs to be performed again until the data of the primary cluster passes the test and the data of the standby cluster passes the test.

[0092] In this embodiment, the application is controlled to restore the default read-write mode only when the data in the standby cluster is completely imported and the data in the primary cluster and the standby cluster are qualified, thereby ensuring the integrity of the data import of the primary cluster and the standby cluster.

[0093] The present invention also provides a data import device, referring to Figure 3 , the data import device comprises:

[0094] A determination module A10, used to determine a target model mode for data import, and control the application program to read and write the standby cluster according to the target model mode;

[0095] A detection module A20 is used to import the master data in the data warehouse into the master cluster and detect whether the master cluster has completely received the master data;

[0096] An import module A30 is used to control the application to read and write only the primary cluster if the primary cluster has completely received the primary data, and import the backup data in the data warehouse to the backup cluster, and detect whether the backup cluster has completely received the backup data, wherein the data warehouse includes primary data and backup data;

[0097] The recovery module A40 is used to control the application to recover the default read-write mode if the standby cluster has completely received the standby data.

[0098] Optionally, the detection module A20 is further used for:

[0099] Data in the data warehouse is cleaned to obtain target data in a data format supported by the primary cluster, the target data is used as primary data, and the target data is imported into the primary cluster.

[0100] Optionally, the detection module A20 is further used for:

[0101] Encapsulating the data source of the target data to obtain encapsulated data;

[0102] The encapsulated data is stored in a HDFS temporary directory file, and the HDFS temporary directory file is moved to the main cluster.

[0103] Optionally, the detection module A20 is further used for:

[0104] Determine the intermediate data or the full historical data in the data warehouse, and perform data cleaning on the intermediate data or the full historical data to obtain cleaned data;

[0105] The data format supported by the main cluster is determined, the format of the cleaned data is converted according to the data format supported by the main cluster, and the cleaned data after the format conversion is used as the target data having the data format supported by the main cluster.

[0106] Optionally, the import module A30 is further used for:

[0107] The storage format of the standby cluster is determined, and the standby data in the data warehouse is format-converted according to the storage format, and the format-converted standby data is imported into the standby cluster.

[0108] Optionally, the determining module A10 is further configured to:

[0109] Detecting a field value in the application, and determining an available model mode from a plurality of preset model modes according to the field value, and detecting whether the available model mode is a read-only standby cluster mode;

[0110] If so, the available model mode is used as the target model mode for data import.

[0111] Optionally, the recovery module A40 is further used to:

[0112] If the standby cluster has completely received the standby data, then detecting whether the integrity of the data of the active cluster is qualified, and detecting whether the integrity of the data of the standby cluster is qualified;

[0113] If the data detection of the primary cluster is qualified and the data detection of the standby cluster is qualified, the application is controlled to restore the default read-write mode.

[0114] The methods executed by the above-mentioned program modules can refer to the various embodiments of the data import method of the present invention, and will not be described in detail here.

[0115] The invention also provides a computer storage medium.

[0116] The computer storage medium of the present invention stores a data import program, and when the data import program is executed by a processor, the steps of the data import method described above are implemented.

[0117] Among them, the method implemented when the data import program running on the processor is executed can refer to the various embodiments of the data import method of the present invention, and will not be repeated here.

[0118] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0119] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0120] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0121] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A data import method, characterized in that: The data import method is applied to a data import system, the data import system includes an application, a main cluster, a standby cluster and a data warehouse, and the data import method includes the following steps: Detecting a field value in the application, and determining an available model mode from a plurality of preset model modes according to the field value, and detecting whether the available model mode is a read-only write standby cluster mode; If yes, the available model mode is used as the target model mode for data import; Controlling the application to read and write the standby cluster according to the target model mode; Importing the master data in the data warehouse into the master cluster, and detecting whether the master cluster has completely received the master data; If the primary cluster has completely received the primary data, control the application to read and write only the primary cluster, and import the backup data in the data warehouse into the backup cluster, and detect whether the backup cluster has completely received the backup data, wherein the data warehouse includes primary data and backup data; If the standby cluster has completely received the standby data, the application is controlled to restore the default read-write mode.

2. The data import method according to claim 1, characterized in that: The step of importing the master data in the data warehouse into the master cluster includes: Data in the data warehouse is cleaned to obtain target data in a data format supported by the primary cluster, the target data is used as primary data, and the target data is imported into the primary cluster.

3. The data import method according to claim 2, characterized in that: The step of importing the target data into the main cluster includes: Encapsulating the data source of the target data to obtain encapsulated data; The encapsulated data is stored in a HDFS temporary directory file, and the HDFS temporary directory file is moved to the main cluster.

4. The data import method according to claim 2, characterized in that: The step of performing data cleaning on the data in the data warehouse to obtain target data in a data format supported by the main cluster includes: Determine the intermediate data or the full historical data in the data warehouse, and perform data cleaning on the intermediate data or the full historical data to obtain cleaned data; Determine a data format supported by the main cluster, perform format conversion on the cleaned data according to the data format supported by the main cluster, and use the cleaned data after the format conversion as target data.

5. The data import method according to claim 1, characterized in that: The step of importing the standby data in the data warehouse into the standby cluster includes: The storage format of the standby cluster is determined, and the standby data in the data warehouse is format-converted according to the storage format, and the format-converted standby data is imported into the standby cluster.

6. The data import method according to any one of claims 1 to 5, characterized in that: The step of controlling the application to restore the default read-write mode if the standby cluster has completely received the standby data includes: If the standby cluster has completely received the standby data, then detecting whether the integrity of the data of the active cluster is qualified, and detecting whether the integrity of the data of the standby cluster is qualified; If the data detection of the primary cluster is qualified and the data detection of the standby cluster is qualified, the application is controlled to restore the default read-write mode.

7. A data import device, characterized in that: The data import device comprises: A determination module, used to detect a field value in an application, and determine an available model mode from a plurality of preset model modes according to the field value, and detect whether the available model mode is a read-only write standby cluster mode; if so, use the available model mode as a target model mode for data import; and control the application to read and write the standby cluster according to the target model mode; A detection module, used for importing the master data in the data warehouse into the master cluster, and detecting whether the master cluster has completely received the master data; An import module, used to control the application to read and write only the primary cluster if the primary cluster has completely received the primary data, and import the backup data in the data warehouse to the backup cluster, and detect whether the backup cluster has completely received the backup data, wherein the data warehouse includes primary data and backup data; The recovery module is used to control the application to recover the default read-write mode if the standby cluster has completely received the standby data.

8. A data import device, characterized in that: The data import device comprises: a memory, a processor, and a data import program stored in the memory and executable on the processor, wherein the data import program, when executed by the processor, implements the steps of the data import method according to any one of claims 1 to 6.

9. A computer storage medium, characterized in that: The computer storage medium stores a data import program, and when the data import program is executed by the processor, the steps of the data import method according to any one of claims 1 to 6 are implemented.

Images