A DataX-based quantum platform data acquisition encryption method
By automatically adapting encryption strategies and chaining Transformer calls, the problem of insufficient linkage between DataX and the quantum encryption platform is solved, enabling efficient and secure cross-database synchronization of sensitive data and reducing the risks of manual configuration and exposure of intermediate data.
Patent Information
- Application Number
- CN202611253696.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-18
- Publication Date
- 2026-11-17
AI Technical Summary
The lack of automatic linkage between DataX and the quantum encryption platform means that encryption and decryption parameters need to be manually configured when synchronizing sensitive data across databases, which is prone to errors and difficult to adapt to different encryption algorithms and modes, increasing the risk of data exposure.
This approach combines automatic encryption strategy adaptation with Transformer chaining. By obtaining the encryption strategies of the source and target ends, it automatically maps fields and binds decryption and encryption plugins, constructs a field transformation chain and DataX task JSON configuration, and achieves data transformation.
It improves the level of configuration automation, enhances policy compatibility, reduces manual adjustments, improves the security and processing efficiency of sensitive data collection, and avoids the creation of plaintext intermediate tables.
Smart Images

Figure CN123122895A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data security technology, and in particular to a data acquisition encryption method based on the DataX quantum platform. Background Technology
[0002] With the continuous development of data platforms, digital government, and cross-system data sharing, utilizing ETL technology to achieve data acquisition and synchronization between multiple databases has become an important method for data integration. DataX uses Reader, Transformer, and Writer plugins to extract, transform, and load data, while the quantum encryption platform can configure encryption algorithms, ciphertext fields, and keys for database fields. However, in practical applications, the following problems commonly exist:
[0003] The lack of automatic linkage between DataX and the quantum encryption platform means that during the cross-database synchronization of sensitive data, manual querying of encryption strategies and configuration of encryption / decryption parameters are required for both the source and target ends. This increases the workload and susceptibility to errors in field or key configuration as the number of data sources and fields increases. Furthermore, the source and target ends may employ different encryption algorithms, keys, and operating modes. Existing data synchronization methods struggle to automatically adjust field mappings and encryption / decryption processing order based on the encryption status of both ends, potentially leading to non-compliance of encrypted data on the target end. When source-side ciphertext needs to be converted to another type of ciphertext on the target end, traditional methods typically require decryption followed by re-encryption. Decrypted data may pass through intermediate tables, files, or other storage stages, increasing the risk of sensitive data exposure during data acquisition.
[0004] Therefore, how to provide a data acquisition encryption method based on DataX for quantum platforms is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0005] One objective of this invention is to propose a data acquisition encryption method for a quantum platform based on DataX. This invention employs a combination of automatic encryption strategy adaptation and Transformer chained calls. By acquiring the source and target encryption strategies and identifying the encryption status of mapped fields, it automatically completes the replacement of plaintext field mappings to ciphertext field mappings and the binding of the df_decrypt and df_encrypt plugins. Furthermore, it constructs a field transformation chain and DataX task JSON configuration according to field processing relationships. Based on this, multiple Transformer plugins sequentially execute field decryption and encryption to achieve data conversion between different encryption strategies. This method boasts advantages such as high automation in configuration, strong strategy compatibility, and high security for sensitive data acquisition.
[0006] According to an embodiment of the present invention, a data acquisition encryption method based on DataX for a quantum platform includes the following steps:
[0007] S1. Configure the source-end encryption strategy and the target-end encryption strategy in the quantum encryption platform, and generate the master key based on quantum random numbers;
[0008] S2. Configure a data synchronization task in the data platform, establish a field mapping relationship between the source and target ends, and enable quantum encryption.
[0009] S3. Obtain the source encryption strategy and target encryption strategy according to the data synchronization task, identify the encryption status of the mapped fields, automatically replace the plaintext field mapping that needs to be encrypted with the ciphertext field mapping, and automatically bind the decryption plugin and encryption plugin according to the source encryption strategy and target encryption strategy, determine the plugin calling order, and form a field conversion chain and DataX task JSON configuration.
[0010] S4. DataX executes the DataX task JSON configuration, reads the source data through the Reader plugin, calls multiple Transformer plugins in the order of the field transformation chain, loads the plugin parameters to perform field decryption and encryption, and writes the processed data to the target end through the Writer plugin to obtain encrypted synchronized data.
[0011] S5. Verify the encrypted synchronization data to complete the data acquisition encryption.
[0012] Optionally, S1 specifically includes:
[0013] Configure the database connection information of the source and target ends in the quantum encryption platform, and obtain the table field information of the corresponding database;
[0014] Based on the table field information, the encryption field is determined, and the encryption algorithm, ciphertext field and running mode corresponding to the encryption field are configured to form the source-end encryption strategy and the target-end encryption strategy.
[0015] A master key associated with the encrypted field is generated based on quantum random numbers. The association between the master key and the encrypted field is established according to the database URL, table name, and field name. The source encryption strategy, target encryption strategy, and master key are stored in the configuration center.
[0016] Optionally, S2 specifically includes:
[0017] Set up a master quantum encryption switch and task-level quantum encryption switches corresponding to data synchronization tasks in the data platform;
[0018] Configure the source, target and field mapping relationships for the data synchronization task. When both the quantum encryption master switch and the task-level quantum encryption switch are in the on state, hide the ciphertext field of the configured encryption policy and retain the corresponding plaintext field as the field mapping configuration object.
[0019] The configured source, target, field mapping relationships, and quantum encryption switch status are combined to form a data synchronization task.
[0020] Optionally, S3 specifically includes:
[0021] Based on the data synchronization task, extract the data source information of the source and target ends, call the encryption strategy query interface of the quantum encryption platform, and obtain the source encryption strategy and target encryption strategy corresponding to the mapping field;
[0022] The source-end encryption strategy and the target-end encryption strategy are matched with the source field and the target field in the field mapping relationship, respectively. The running mode and encryption algorithm of the two ends of the field are read, and the encryption status of the source field and the target field is determined according to the running mode.
[0023] Compare the encryption states of the source field and the target field. Record the decryption process when the source field is encrypted, record the encryption process when the target field is encrypted, and record the decryption and encryption processes sequentially when both the source field and the target field are encrypted.
[0024] Read the preset encryption algorithm compatibility matrix, query the encryption algorithm compatibility matrix according to the encryption algorithms in the source encryption strategy and the encryption algorithms in the target encryption strategy, determine the execution combination of the decryption process and the encryption process, and obtain the field processing relationship;
[0025] Based on the source-end encryption strategy and the target-end encryption strategy, the plaintext field mapping in intermediate mode or ciphertext mode is replaced with the corresponding ciphertext field mapping to obtain the encrypted field mapping relationship.
[0026] The corresponding df_decrypt and df_encrypt plugins are associated based on the encrypted field mapping relationship and field processing relationship. The calling order of the Transformer plugins is arranged according to the field processing relationship to form a field transformation chain.
[0027] Configure the Reader and Writer plugins according to the encrypted field mapping relationship, and write the field conversion chain and corresponding plugin parameters into the DataX task configuration structure to form the DataX task JSON configuration.
[0028] Optionally, the step of replacing the plaintext field mapping in intermediate mode or ciphertext mode with the corresponding ciphertext field mapping based on the source-end encryption strategy and the target-end encryption strategy to obtain the encrypted field mapping relationship specifically includes:
[0029] Read the source field and target field in the field mapping relationship in sequence, and match the source field with the field information in the source encryption strategy;
[0030] When the running mode corresponding to the source field is intermediate mode or ciphertext mode, read the ciphertext field corresponding to the source field, replace the source field in the field mapping relationship with the ciphertext field, and record the decryption process corresponding to the ciphertext field.
[0031] The target field is matched with the field information in the target encryption policy. When the running mode corresponding to the target field is intermediate mode or ciphertext mode, the ciphertext field corresponding to the target field is read, the target field in the field mapping relationship is replaced with the ciphertext field, and the encryption process corresponding to the ciphertext field is recorded.
[0032] Compare the field mappings after the replacement. When the source field and target field in two field mappings are the same, merge the two field mappings into one to obtain the encrypted field mapping relationship.
[0033] Optionally, the step of associating the corresponding df_decrypt and df_encrypt plugins according to the encrypted field mapping relationship and field processing relationship, and arranging the calling order of the Transformer plugins according to the field processing relationship to form a field transformation chain, specifically includes:
[0034] Determine the field index corresponding to each field according to the field arrangement order in the encrypted field mapping relationship, and filter the records with fields that have decryption or encryption processing according to the field processing relationship;
[0035] For fields that have undergone decryption processing, establish the association between the field index and the df_decrypt plugin; for fields that have undergone encryption processing, establish the association between the field index and the df_encrypt plugin.
[0036] Read the encryption algorithm and key information corresponding to the decryption processing field in the source encryption strategy, determine the key parameters according to the key information, and write the field index, the corresponding decryption type and the key parameters into the df_decrypt plugin parameters;
[0037] Read the encryption algorithm and key information corresponding to the encryption processing field in the target encryption policy, determine the key parameters according to the key information, and write the field index, the corresponding encryption type and the key parameters into the df_encrypt plugin parameters;
[0038] When the same field records decryption and encryption processes in sequence, the two Transformer plugins are arranged in the order of df_decrypt plugin first and df_encrypt plugin second; when the field only records decryption or encryption processes, the corresponding single Transformer plugin is retained.
[0039] By assembling the Transformer plugins corresponding to each field according to the field arrangement order in the encrypted field mapping relationship, a field transformation chain is obtained.
[0040] Optionally, S4 specifically includes:
[0041] DataX loads the DataX task JSON configuration and uses the Reader plugin to read source field data according to the encrypted field mapping relationship to form a data record to be converted.
[0042] The Transformer plugins include the df_decrypt plugin and the df_encrypt plugin;
[0043] According to the plugin call order of the field transformation chain, the data record to be transformed is passed to the corresponding Transformer plugin in sequence. Based on the field index, encryption / decryption type and key parameters configured in each Transformer plugin, the corresponding field is decrypted or encrypted to obtain the transformed data record.
[0044] When the same field is configured for decryption and encryption in sequence, the decrypted field data is passed to the corresponding encryption plugin and re-encrypted according to the target encryption policy to obtain the target field data.
[0045] The Writer plugin writes the target field data into the corresponding field on the target end according to the encrypted field mapping relationship, thus obtaining encrypted synchronization data.
[0046] Optionally, when the same field is configured for decryption and encryption processes sequentially, the decrypted field data is passed to the corresponding encryption plugin and re-encrypted according to the target encryption strategy to obtain the target field data. Specifically, this includes:
[0047] Read the df_decrypt and df_encrypt plugins corresponding to the same field according to the field conversion chain, and obtain the field index that is common to both plugins;
[0048] The data record to be converted and the df_decrypt plugin parameters are passed to the evaluate method of the df_decrypt plugin. The source ciphertext field data is read from the data record to be converted according to the field index, and the corresponding decryption type and key parameters are read from the df_decrypt plugin parameters.
[0049] The source-end ciphertext field data is decrypted using the decryption type and key parameters, and the decrypted field data is written back to the position corresponding to the field index to form a decrypted data record.
[0050] The decrypted data record and df_encrypt plugin parameters are passed to the evaluate method of the df_encrypt plugin. The decrypted field data is read according to the same field index, and the corresponding encryption type and key parameters are read from the df_encrypt plugin parameters.
[0051] The decrypted field data is encrypted using the encryption type and key parameters, and the obtained target field data is written back to the position corresponding to the field index to form a re-encrypted transformed data record.
[0052] The transformed data record continues to be passed along the field transformation chain until the Transformer plugin call in the field transformation chain is completed.
[0053] Optionally, S5 specifically includes:
[0054] Read the encrypted synchronization data written by the target end, call the decryption interface of the quantum encryption platform according to the encryption strategy of the target end, and decrypt the ciphertext field in the encrypted synchronization data to obtain the verification field data;
[0055] The data in the verification field is compared for consistency with the data content of the corresponding field in the source.
[0056] When the data in the verification field matches the data in the corresponding field on the source end, the data collection and encryption are considered complete.
[0057] The beneficial effects of this invention are:
[0058] (1) This invention obtains the source encryption strategy and the target encryption strategy, matches the operation mode and encryption algorithm of the mapped field, and determines the execution combination of decryption and encryption processing through the encryption algorithm compatibility matrix. The plaintext field mapping in the intermediate mode or ciphertext mode is automatically replaced with the ciphertext field mapping. At the same time, the df_decrypt plugin and df_encrypt plugin are associated to form an encrypted field mapping relationship and field conversion chain, which reduces manual field adjustment and plugin parameter configuration in multi-source and multi-target data synchronization scenarios and improves the adaptability between different encryption algorithms and operation modes.
[0059] (2) This invention calls multiple Transformer plugins in the order of field transformation chain through DataX. In the same data record to be transformed, the df_decrypt plugin first performs source end ciphertext field decryption, and then the df_encrypt plugin performs re-encryption according to the target end encryption strategy. The corresponding field of the target end is written through the Writer plugin, so that the decrypted field data is continuously transmitted in the DataX data transformation process, reducing the establishment of plaintext intermediate tables and intermediate files. While ensuring that the target end encrypted synchronous data is consistent with the corresponding data of the source end, it improves the security and processing efficiency of the cross-database collection process of sensitive data. Attached Figure Description
[0060] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:
[0061] Figure 1 This is an overall flowchart of a quantum platform data acquisition encryption method based on DataX proposed in this invention;
[0062] Figure 2 This is a schematic diagram of the automatic adaptation of the encryption strategy proposed in this invention and the DataX encryption synchronization process;
[0063] Figure 3 This is a schematic diagram of the field transformation chain construction process proposed in this invention;
[0064] Figure 4 This is a schematic diagram of the data synchronization task configuration interface of the data platform proposed in this invention. Detailed Implementation
[0065] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.
[0066] refer to Figures 1-4A data acquisition encryption method based on DataX quantum platform includes the following steps:
[0067] S1. Configure the source-end encryption strategy and the target-end encryption strategy in the quantum encryption platform, and generate the master key based on quantum random numbers;
[0068] S2. Configure a data synchronization task in the data platform, establish a field mapping relationship between the source and target ends, and enable quantum encryption.
[0069] S3. Obtain the source encryption policy and target encryption policy according to the data synchronization task, identify the encryption status of the mapped fields, automatically replace the plaintext field mapping that needs to be encrypted with the ciphertext field mapping, and automatically bind the decryption plugin and encryption plugin according to the source encryption policy and target encryption policy, determine the plugin calling order, and form a field conversion chain and DataX task JSON configuration.
[0070] S4. DataX executes the DataX task JSON configuration, reads the source data through the Reader plugin, calls multiple Transformer plugins in the order of field transformation chain, loads plugin parameters to perform field decryption and encryption, and writes the processed data to the target end through the Writer plugin to obtain encrypted synchronized data.
[0071] S5. Verify the encrypted synchronization data to complete the data acquisition encryption.
[0072] In this embodiment, S1 specifically includes:
[0073] Configure the database connection information of the source and target ends in the quantum encryption platform, and obtain the table field information of the corresponding database;
[0074] Based on the table field information, determine the encryption field, configure the encryption algorithm, ciphertext field and running mode corresponding to the encryption field, and form the source-end encryption strategy and the target-end encryption strategy.
[0075] The master key is generated based on quantum random number and associated with the encrypted field. The association between the master key and the encrypted field is established according to the database URL, table name and field name. The source encryption strategy, target encryption strategy and master key are stored in the configuration center.
[0076] In the specific implementation process, the quantum encryption platform connects to the source database DB_A and the target database DB_B respectively. The JDBC URL of DB_A is set to "jdbc:mysql: / / 192.168.1.100:3306 / social_db", and the corresponding table name is set to "social_security". The JDBC URL of DB_B is set to "jdbc:mysql: / / 192.168.2.200:3306 / medical_db", and the corresponding table name is set to "medical_insurance". The quantum encryption platform reads the table field information of the two databases and selects "id_card" as the encryption field.
[0077] Configure the FPE encryption algorithm for the "id_card" field on both the source and target sides, setting the ciphertext field to "id_card_cipher" and the operating mode to intermediate mode. The quantum encryption platform generates master keys K1 and K2 based on quantum random numbers, with K1 set to "a7f3e9d2c4b1a8f7e6d5c4b3a2f1e0" and K2 set to "b8e4f0c1d3a5b7e9f0d2c4a6b8e0f1". Combine the database URL, table name, and field name to form the association identifier for each encrypted field. Write K1, K2, and the corresponding source and target encryption policies into the Nacos configuration center.
[0078] In this embodiment, S2 specifically includes:
[0079] Set up a master quantum encryption switch and task-level quantum encryption switches corresponding to data synchronization tasks in the data platform;
[0080] Configure the source, target and field mapping relationships for the data synchronization task. When both the quantum encryption master switch and the task-level quantum encryption switch are on, hide the ciphertext fields of the configured encryption policy and retain the corresponding plaintext fields as field mapping configuration objects.
[0081] The configured source, target, field mapping relationships, and quantum encryption switch status are combined to form a data synchronization task;
[0082] In the specific implementation process, refer to Figure 4In the data synchronization task configuration interface of the data platform, select the source and target ends and establish field mapping relationships, while simultaneously enabling the task-level quantum encryption switch. The data platform pre-enables the quantum encryption master switch, and simultaneously enables the task-level quantum encryption switch corresponding to the task when creating the "Social Security → Medical Insurance Data Synchronization" task; select the "social_security" table in DB_A for the source end, and select the "medical_insurance" table in DB_B for the target end.
[0083] The data platform frontend reads field information from two tables, displaying plaintext fields such as "id_card", "name", and "payment" in the field configuration area, while hiding the encrypted "id_card_cipher" field, which has a configured encryption policy. Operators set field mapping relationships such as "id_card→id_card", "name→name", and "payment→payment". Upon task submission, DB_A, social_security, DB_B, medical_insurance, the aforementioned field mapping relationships, and the states of the two quantum encryption switches are combined into a data synchronization task and passed to the subsequent DataX task JSON configuration processing.
[0084] In this embodiment, S3 specifically includes:
[0085] Extract the data source information of the source and target ends based on the data synchronization task, call the encryption strategy query interface of the quantum encryption platform, and obtain the source encryption strategy and target encryption strategy corresponding to the mapping field;
[0086] The source encryption strategy and the target encryption strategy are matched with the source field and the target field in the field mapping relationship, respectively. The running mode and encryption algorithm of the fields at both ends are read, and the encryption status of the source field and the target field is determined according to the running mode.
[0087] Compare the encryption states of the source field and the target field. Record the decryption process when the source field is encrypted, record the encryption process when the target field is encrypted, and record the decryption and encryption processes sequentially when both the source field and the target field are encrypted.
[0088] Read the preset encryption algorithm compatibility matrix, query the encryption algorithm compatibility matrix according to the encryption algorithms in the source encryption strategy and the encryption algorithms in the target encryption strategy, determine the execution combination of decryption and encryption processing, and obtain the field processing relationship;
[0089] Based on the source-end encryption strategy and the target-end encryption strategy, the plaintext field mapping in intermediate mode or ciphertext mode is replaced with the corresponding ciphertext field mapping to obtain the encrypted field mapping relationship.
[0090] Based on the mapping relationship and processing relationship of encrypted fields, the corresponding df_decrypt and df_encrypt plugins are associated, and the calling order of the Transformer plugins is arranged according to the processing relationship to form a field transformation chain;
[0091] Configure the Reader and Writer plugins according to the encrypted field mapping relationship, and write the field conversion chain and corresponding plugin parameters into the DataX task configuration structure to form the DataX task JSON configuration.
[0092] In the specific implementation process, after the data platform submits the data synchronization task, it extracts the data source information of the source and target ends based on DB_A, social_security, DB_B, and medical_insurance in the task. It then queries the field policies of the two tables through the encryption policy query interface of the quantum encryption platform. The source end “id_card” returns the ciphertext field “id_card_cipher”, FPE encryption algorithm, key information K1, and intermediate mode. The target end “id_card” returns the ciphertext field “id_card_cipher”, FPE encryption algorithm, key information K2, and intermediate mode.
[0093] The source encryption strategy is matched item by item with the source fields in the field mapping relationship, and the target encryption strategy is matched item by item with the target fields. "id_card" is in intermediate mode on the source side, recording decryption processing, and is also in intermediate mode on the target side, recording encryption processing. Therefore, this field forms a processing combination of "decryption processing → encryption processing". "name" and "payment" did not match any encryption strategy, so no decryption or encryption processing was recorded.
[0094] The encryption algorithm compatibility matrix establishes a row-column correspondence based on the source and target encryption algorithms, with each intersection position recording the corresponding field processing combination. For the government MySQL data source in the extended embodiment, when the source and target use different encryption algorithms, the corresponding intersection position records "source decryption → target encryption"; when the medical PostgreSQL data source uses dedicated encryption for mobile phone numbers and the target uses a different encryption algorithm, the corresponding intersection position records "decryption for mobile phone numbers → target encryption"; when the social security Oracle data source is not configured with source encryption, the corresponding position records "direct target encryption". After querying the encryption algorithm compatibility matrix, the field processing relationship corresponding to each mapping field is obtained.
[0095] For "id_card" in intermediate or encrypted mode, replace "id_card→id_card" with "id_card_cipher→id_card_cipher". Based on the field processing relationship, associate the source field with the df_decrypt plugin and the target field with the df_encrypt plugin, placing df_decrypt before df_encrypt to form a field transformation chain. Then, configure the Reader and Writer plugins according to the field order of "id_card_cipher, name, payment", and write the field transformation chain and plugin parameters into the DataX task configuration structure to form the DataX task JSON configuration.
[0096] This process continuously writes encryption policy queries, field mapping adjustments, and Transformer plugin configurations into the same task configuration generation process. When the source and target use different encryption algorithms, the corresponding field transformation chain is still determined according to the encryption algorithm compatibility matrix.
[0097] In this embodiment, based on the source-end encryption strategy and the target-end encryption strategy, the plaintext field mapping in intermediate mode or ciphertext mode is replaced with the corresponding ciphertext field mapping to obtain the encrypted field mapping relationship, specifically including:
[0098] Read the source field and target field in the field mapping relationship in sequence, and match the source field with the field information in the source encryption policy;
[0099] When the running mode corresponding to the source field is intermediate mode or ciphertext mode, read the ciphertext field corresponding to the source field, replace the source field in the field mapping relationship with the ciphertext field, and record the decryption process corresponding to the ciphertext field.
[0100] Match the target field with the field information in the target encryption policy. When the running mode corresponding to the target field is intermediate mode or ciphertext mode, read the ciphertext field corresponding to the target field, replace the target field in the field mapping relationship with the ciphertext field, and record the encryption process corresponding to the ciphertext field.
[0101] Compare the field mappings after the replacement. When the source field and target field in two field mappings are the same, merge the two field mappings into one to obtain the encrypted field mapping relationship.
[0102] In the specific implementation process, the field mapping relationships in the data synchronization task are read item by item. For the original mapping "id_card→id_card", the source field "id_card" is first matched with the source encryption strategy. The intermediate mode and the corresponding ciphertext field "id_card_cipher" are read. The source field is replaced with "id_card_cipher", and the decryption process is recorded in the field mapping.
[0103] Continue matching the target field "id_card" with the target encryption policy. Read the running mode, which is also intermediate mode, and the corresponding ciphertext field "id_card_cipher". Replace the target field with "id_card_cipher" and record the encryption process. The original field mapping is changed from "id_card→id_card" to "id_card_cipher→id_card_cipher".
[0104] For "name→name" and "payment→payment", the original field mapping remains unchanged if no intermediate mode or ciphertext mode encryption strategy is found. After all fields are replaced, field mapping pairs are formed using the source field and the target field, and each field mapping pair is compared item by item. When the source field and the target field of two field mapping pairs are the same, only one of them is retained, ultimately forming an encrypted field mapping relationship without duplicate ciphertext field mappings.
[0105] In this embodiment, the corresponding df_decrypt and df_encrypt plugins are associated based on the encryption field mapping relationship and the field processing relationship. The Transformer plugins are called in a specific order according to the field processing relationship to form a field transformation chain, which includes:
[0106] Determine the field index corresponding to each field according to the field order in the encrypted field mapping relationship, and filter the records with fields that have been decrypted or encrypted according to the field processing relationship;
[0107] For fields in a record that have undergone decryption processing, establish an association between the field index and the df_decrypt plugin; for fields in a record that have undergone encryption processing, establish an association between the field index and the df_encrypt plugin.
[0108] Read the encryption algorithm and key information corresponding to the decryption processing field in the source encryption strategy, determine the key parameters based on the key information, and write the field index, the corresponding decryption type, and the key parameters into the df_decrypt plugin parameters;
[0109] Read the encryption algorithm and key information corresponding to the encryption processing field in the target encryption policy, determine the key parameters based on the key information, and write the field index, the corresponding encryption type and key parameters into the df_encrypt plugin parameters;
[0110] When the same field records decryption and encryption processes in sequence, the two Transformer plugins are arranged in the order of df_decrypt plugin first and df_encrypt plugin second; when the field only records decryption or encryption processes, the corresponding single Transformer plugin is retained.
[0111] The Transformer plugins corresponding to each field are gathered according to the field order in the encrypted field mapping relationship to obtain the field transformation chain;
[0112] In the specific implementation process, the order in which the Reader plugin reads fields is determined according to the mapping relationship of encrypted fields. For example, “id_card_cipher”, “name” and “payment” are arranged in sequence, with corresponding field indices of 0, 1 and 2 respectively. After querying the field processing relationship, “id_card_cipher” records both decryption and encryption processing. Therefore, field index 0 is associated with the df_decrypt plugin and the df_encrypt plugin respectively. “name” and “payment” do not record decryption or encryption processing and are not associated with the Transformer plugin.
[0113] For the df_decrypt plugin, read the FPE encryption algorithm and key information K1 corresponding to "id_card" from the source encryption strategy. Determine the key parameters based on the key information, and write the field index 0, decryption type "FPE", and key parameters into the df_decrypt plugin parameters according to the DataX Transformer parameter structure. In the corresponding JSON configuration, set the plugin name to "df_decrypt", set columnIndex to 0, set the algorithm type in paras to "FPE", and write the corresponding key parameters.
[0114] For the df_encrypt plugin, the FPE encryption algorithm and key information K2 are read from the target encryption policy. Key parameters are formed based on the key information. Similarly, the field index 0, encryption type "FPE" and key parameters are written into the df_encrypt plugin parameters.
[0115] Since field index 0 records both decryption and encryption processing, the two Transformer plugins are arranged in the order of df_decrypt first, followed by df_encrypt. If a field only records encryption processing, only the df_encrypt plugin is configured. If a field records both decryption and encryption processing, the df_decrypt plugin is configured according to the decryption type corresponding to the source encryption strategy, and the df_encrypt plugin is configured according to the encryption type corresponding to the target encryption strategy. The plugin configurations for each field are compiled according to their order to obtain the field transformation chain.
[0116] In this embodiment, S4 specifically includes:
[0117] DataX loads the DataX task JSON configuration and uses the Reader plugin to read source field data according to the encrypted field mapping relationship, forming a data record to be converted.
[0118] Transformer plugins include the df_decrypt plugin and the df_encrypt plugin;
[0119] According to the plugin call order of the field transformation chain, the data record to be transformed is passed to the corresponding Transformer plugin in sequence. Based on the field index, encryption / decryption type and key parameters configured in each Transformer plugin, the corresponding field is decrypted or encrypted to obtain the transformed data record.
[0120] When the same field is configured for decryption and encryption in sequence, the decrypted field data is passed to the corresponding encryption plugin and re-encrypted according to the target encryption policy to obtain the target field data.
[0121] The Writer plugin writes the target field data to the corresponding field on the target end according to the encrypted field mapping relationship, thus obtaining encrypted synchronized data.
[0122] In the specific implementation process, after receiving the DataX task JSON configuration, DataX loads the mysqlreader plugin. mysqlreader connects to the "social_security" table of DB_A and reads "id_card_cipher", "name" and "payment" in sequence according to the encrypted field mapping relationship. The single read result is organized into a data record to be converted, where field index 0 corresponds to "id_card_cipher", field index 1 corresponds to "name", and field index 2 corresponds to "payment".
[0123] DataX sequentially passes the data records to be converted to the df_decrypt and df_encrypt plugins according to the field conversion chain. Each plugin reads the field index, encryption / decryption type, and key parameters from the JSON configuration, performs the conversion on the data corresponding to field index 0, and keeps the data at field indices 1 and 2 in the same data record for continued transmission.
[0124] After the Transformer plugin call is completed, the target field data that conforms to the target encryption policy is obtained at field index 0; mysqlwriter connects to the "medical_insurance" table of DB_B and writes the data records to the target end in the order of writing "id_card_cipher", "name" and "payment", forming encrypted synchronous data.
[0125] In this embodiment, when the same field is configured for decryption and encryption processes sequentially, the decrypted field data is passed to the corresponding encryption plugin, and encryption is re-executed according to the target encryption strategy to obtain the target field data. Specifically, this includes:
[0126] Read the df_decrypt and df_encrypt plugins corresponding to the same field according to the field transformation chain, and obtain the field index that is common to both plugins;
[0127] The data record to be converted and the parameters of the df_decrypt plugin are passed to the evaluate method of the df_decrypt plugin. The source ciphertext field data is read from the data record to be converted according to the field index, and the corresponding decryption type and key parameters are read from the parameters of the df_decrypt plugin.
[0128] The source ciphertext field data is decrypted using the decryption type and key parameters. The decrypted field data is then written back to the position corresponding to the field index to form a decrypted data record.
[0129] The decrypted data record and df_encrypt plugin parameters are passed to the evaluate method of the df_encrypt plugin. The decrypted field data is read according to the same field index, and the corresponding encryption type and key parameters are read from the df_encrypt plugin parameters.
[0130] The decrypted field data is encrypted using the encryption type and key parameters. The obtained target field data is then written back to the position corresponding to the field index, forming a re-encrypted transformed data record.
[0131] Continue passing the transformed data record along the field transformation chain until the Transformer plugin call in the field transformation chain is completed;
[0132] In practice, both the `df_decrypt` and `df_encrypt` plugins inherit from DataX's `Transformer` abstract class and implement the `evaluate` method. For the "id_card_cipher" field, the field transformation chain first reads the `df_decrypt` plugin to determine the common field index 0 for both this plugin and the subsequent `df_encrypt` plugin.
[0133] When executing the `df_decrypt` plugin, the current record to be converted (`record`) and the `df_decrypt` plugin parameters are passed to the `evaluate` method. The `evaluate` method reads `columnIndex=0`, retrieves the source ciphertext data corresponding to "id_card_cipher" from the 0th field of the `record`, and then reads the decryption type "FPE" and the corresponding source key parameter from the plugin parameters `paras`. After calling FPE decryption, the decrypted ID number is written back to the 0th field of the `record`, while the original "name" and "payment" fields remain unchanged, thus obtaining the decrypted data record.
[0134] Instead of writing the decrypted data record to the database or intermediate file, the same data record is directly passed to the df_encrypt plugin. The df_encrypt plugin's evaluate method reads the same columnIndex=0, retrieves the decrypted field data just written back, and reads the encryption type "FPE" and the corresponding key parameters from the df_encrypt plugin parameters; after performing FPE encryption, the resulting new ciphertext is written back to the 0th field of the record, forming the re-encrypted converted data record.
[0135] In the extended embodiment, when the source encryption strategy and the target encryption strategy use different encryption algorithms, the `df_decrypt` plugin loads the decryption type and key parameters corresponding to the source encryption strategy's `paras`, and subsequently, the `df_encrypt` plugin loads the encryption type and key parameters corresponding to the target encryption strategy's `paras`. The two plugins are executed sequentially according to the same data record transfer method. After all Transformer plugins in the field transformation chain are completed, the transformed data record is then handed over to the Writer plugin. In the above chained processing, the decrypted field data only exists in the memory processing stage where the current DataX data record resides; no intermediate decryption table or file is established, reducing the number of decrypted data persistence steps during data acquisition.
[0136] In this embodiment, S5 specifically includes:
[0137] Read the encrypted synchronization data written by the target end, call the decryption interface of the quantum encryption platform according to the encryption strategy of the target end, decrypt the ciphertext field in the encrypted synchronization data, and obtain the verification field data;
[0138] The data in the validation field is compared with the data content of the corresponding field in the source.
[0139] When the data in the verification field matches the data in the corresponding field on the source end, the data collection and encryption are considered complete.
[0140] In the specific implementation process, after the DataX task is completed, the data platform monitoring module reads the encrypted synchronization data written to the “medical_insurance” table in DB_B and extracts the “id_card_cipher” field; according to the target encryption strategy, it reads the FPE encryption algorithm and corresponding key information, and decrypts “id_card_cipher” through the decryption interface of the quantum encryption platform to obtain the verification field data.
[0141] The obtained verification field data is compared with the ID number of the corresponding record in the source "social_security" table one by one. The same data record with completely identical field content is used as the consistency condition. After all the records to be verified are compared, the records with consistent content are confirmed to have correct field conversion results. When the verification field data is consistent with the data content of the corresponding field in the source, the data collection and encryption is confirmed to be complete.
[0142] Example 1: To verify the feasibility of this invention in practice, it was applied to a cross-departmental sensitive data synchronization scenario on a government data sharing platform. This platform needs to synchronize basic personnel information from the social security system to the medical system, including business data such as ID numbers, names, and payment status. Since ID numbers are sensitive information, the source and target databases employ independent encryption strategies. Traditional synchronization methods require manual confirmation of field encryption status, modification of DataX task configurations, and decryption and re-encryption via an intermediate database. This presents problems such as complex configuration processes, the risk of plaintext data being lost to the system, and difficulty in adapting to different encryption rules.
[0143] In this application scenario, the encryption strategies for the source and target databases are first configured using the quantum encryption platform. The ID number field is set as an encrypted field, and the corresponding encryption algorithm, ciphertext field, and key information are associated with it. After the data platform creates a data synchronization task, only the plaintext field mapping relationship configurable by business personnel is displayed, while the ciphertext field is hidden. After the task is submitted, the intelligent JSON configuration generation project automatically calls the quantum encryption platform to obtain the encryption strategies for the source and target databases. Based on the field operation mode, it determines the decryption and encryption processes that need to be performed, automatically converts the original field mapping relationship into a ciphertext field mapping relationship, and generates a DataX task JSON configuration containing parameters for the Reader plugin, Writer plugin, and df_decrypt and df_encrypt plugins.
[0144] When DataX executes a task, the Reader plugin directly reads the ciphertext field data from the source end, while the Transformer plugin performs decryption and encryption processes sequentially according to the field transformation chain. The source ciphertext is first converted into decryptable field data by the df_decrypt plugin, then immediately passed to the df_encrypt plugin for re-encryption according to the target end's encryption policy. Finally, the Writer plugin writes the data to the target database. No plaintext intermediate table is established throughout this process, avoiding additional storage of sensitive data during synchronization.
[0145] To verify the data synchronization effect of the present invention, a comparative test was conducted between the traditional manual configuration synchronization method with the same data volume and database environment and the method of the present invention. The test data included 200,000 personnel information records, with fields including ID number, name, payment status and update time, etc. The test results are shown in Table 1.
[0146] Table 1. Comparison of Application Effects of Data Acquisition Encryption Methods Based on DataX Quantum Platform
[0147] Test metrics Traditional synchronization method Method of the present invention Improvement effect Average time spent on task configuration 31.5 minutes 6.2 minutes Reduced by 80.3% Average time for data synchronization 46.8 seconds 32.1 seconds Decreased by 31.4% Synchronous processing throughput 4270 messages / second 6230 messages / second An increase of 45.9% Plain text data intermediate landing point 1 place 0 Reduce by 100%
[0148] As shown in Table 1, the method of this invention can significantly reduce the workload of manual configuration during sensitive data synchronization. Traditional synchronization methods require manual querying of the encryption status of fields on the source and target ends and manual modification of the DataX task configuration. When the number of fields in the data table increases or different databases use different encryption rules, the task file needs to be adjusted repeatedly, resulting in a long configuration time. This invention automatically obtains the field encryption algorithm, ciphertext field, and operating mode through the policy query interface provided by the quantum encryption platform, and automatically adjusts the mapping relationship according to the field status, reducing the task configuration time from 31.5 minutes to 6.2 minutes, a reduction of approximately 80.3%.
[0149] Meanwhile, this invention controls the Transformer plugin call order through a field transformation chain, enabling source-side decryption and target-side encryption to be completed within the same data record transmission process. Traditional methods require writing the decrypted data to a temporary location before reading the data to perform target-side encryption. In contrast, this invention directly completes the field transformation within DataX, eliminating any plaintext intermediate data storage points. During testing, the traditional method generated one instance of plaintext temporary storage, while this invention generated no plaintext storage points, thus improving security during sensitive data synchronization.
[0150] In terms of data processing efficiency, the average data synchronization time of this invention was reduced from 46.8 seconds to 32.1 seconds, and the processing throughput was increased from 4270 records / second to 6230 records / second. This is because this invention automatically generates corresponding Transformer plugin configurations based on the source and target encryption strategies, avoiding repetitive processing caused by manual modification of task files. Simultaneously, the chained calls of the df_decrypt and df_encrypt plugins reduce the repeated data reading process. Test results show that this invention is applicable to data acquisition scenarios with multi-source heterogeneous data encryption rules, improving data synchronization efficiency and reducing manual configuration costs while ensuring data encryption consistency.
[0151] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A data acquisition encryption method based on DataX quantum platform, characterized in that, Includes the following steps: S1. Configure the source-end encryption strategy and the target-end encryption strategy in the quantum encryption platform, and generate the master key based on quantum random numbers; S2. Configure a data synchronization task in the data platform, establish a field mapping relationship between the source and target ends, and enable quantum encryption. S3. Obtain the source encryption strategy and target encryption strategy according to the data synchronization task, identify the encryption status of the mapped fields, automatically replace the plaintext field mapping that needs to be encrypted with the ciphertext field mapping, and automatically bind the decryption plugin and encryption plugin according to the source encryption strategy and target encryption strategy, determine the plugin calling order, and form a field conversion chain and DataX task JSON configuration. S4. DataX executes the DataX task JSON configuration, reads the source data through the Reader plugin, calls multiple Transformer plugins in the order of the field transformation chain, loads the plugin parameters to perform field decryption and encryption, and writes the processed data to the target end through the Writer plugin to obtain encrypted synchronized data. S5. Verify the encrypted synchronization data to complete the data acquisition encryption.
2. The data acquisition encryption method based on DataX for a quantum platform according to claim 1, characterized in that, S1 specifically includes: Configure the database connection information of the source and target ends in the quantum encryption platform, and obtain the table field information of the corresponding database; Based on the table field information, the encryption field is determined, and the encryption algorithm, ciphertext field and running mode corresponding to the encryption field are configured to form the source-end encryption strategy and the target-end encryption strategy. A master key associated with the encrypted field is generated based on quantum random numbers. The association between the master key and the encrypted field is established according to the database URL, table name, and field name. The source encryption strategy, target encryption strategy, and master key are stored in the configuration center.
3. The data acquisition encryption method based on DataX for a quantum platform according to claim 1, characterized in that, S2 specifically includes: Set up a master quantum encryption switch and task-level quantum encryption switches corresponding to data synchronization tasks in the data platform; Configure the source, target and field mapping relationships for the data synchronization task. When both the quantum encryption master switch and the task-level quantum encryption switch are in the on state, hide the ciphertext field of the configured encryption policy and retain the corresponding plaintext field as the field mapping configuration object. The configured source, target, field mapping relationships, and quantum encryption switch status are combined to form a data synchronization task.
4. The data acquisition encryption method based on DataX for a quantum platform according to claim 1, characterized in that, S3 specifically includes: Based on the data synchronization task, extract the data source information of the source and target ends, call the encryption strategy query interface of the quantum encryption platform, and obtain the source encryption strategy and target encryption strategy corresponding to the mapping field; The source-end encryption strategy and the target-end encryption strategy are matched with the source field and the target field in the field mapping relationship, respectively. The running mode and encryption algorithm of the two ends of the field are read, and the encryption status of the source field and the target field is determined according to the running mode. Compare the encryption states of the source field and the target field. Record the decryption process when the source field is encrypted, record the encryption process when the target field is encrypted, and record the decryption and encryption processes sequentially when both the source field and the target field are encrypted. Read the preset encryption algorithm compatibility matrix, query the encryption algorithm compatibility matrix according to the encryption algorithms in the source encryption strategy and the encryption algorithms in the target encryption strategy, determine the execution combination of the decryption process and the encryption process, and obtain the field processing relationship; Based on the source-end encryption strategy and the target-end encryption strategy, the plaintext field mapping in intermediate mode or ciphertext mode is replaced with the corresponding ciphertext field mapping to obtain the encrypted field mapping relationship. The corresponding df_decrypt and df_encrypt plugins are associated based on the encrypted field mapping relationship and field processing relationship. The calling order of the Transformer plugins is arranged according to the field processing relationship to form a field transformation chain. Configure the Reader and Writer plugins according to the encrypted field mapping relationship, and write the field conversion chain and corresponding plugin parameters into the DataX task configuration structure to form the DataX task JSON configuration.
5. The data acquisition encryption method based on DataX for a quantum platform according to claim 4, characterized in that, The step of replacing plaintext field mappings in intermediate or ciphertext modes with corresponding ciphertext field mappings based on the source-side and target-side encryption strategies to obtain encrypted field mapping relationships specifically includes: Read the source field and target field in the field mapping relationship in sequence, and match the source field with the field information in the source encryption strategy; When the running mode corresponding to the source field is intermediate mode or ciphertext mode, read the ciphertext field corresponding to the source field, replace the source field in the field mapping relationship with the ciphertext field, and record the decryption process corresponding to the ciphertext field. The target field is matched with the field information in the target encryption policy. When the running mode corresponding to the target field is intermediate mode or ciphertext mode, the ciphertext field corresponding to the target field is read, the target field in the field mapping relationship is replaced with the ciphertext field, and the encryption process corresponding to the ciphertext field is recorded. Compare the field mappings after the replacement. When the source field and target field in two field mappings are the same, merge the two field mappings into one to obtain the encrypted field mapping relationship.
6. The data acquisition encryption method based on DataX for a quantum platform according to claim 4, characterized in that, The process of associating the corresponding df_decrypt and df_encrypt plugins based on the encrypted field mapping and field processing relationships, and arranging the Transformer plugins in the order of invocation according to the field processing relationships to form a field transformation chain, specifically includes: Determine the field index corresponding to each field according to the field arrangement order in the encrypted field mapping relationship, and filter the records with fields that have decryption or encryption processing according to the field processing relationship; For fields that have undergone decryption processing, establish the association between the field index and the df_decrypt plugin; for fields that have undergone encryption processing, establish the association between the field index and the df_encrypt plugin. Read the encryption algorithm and key information corresponding to the decryption processing field in the source encryption strategy, determine the key parameters according to the key information, and write the field index, the corresponding decryption type and the key parameters into the df_decrypt plugin parameters; Read the encryption algorithm and key information corresponding to the encryption processing field in the target encryption policy, determine the key parameters according to the key information, and write the field index, the corresponding encryption type and the key parameters into the df_encrypt plugin parameters; When the same field records decryption and encryption processes in sequence, the two Transformer plugins are arranged in the order of df_decrypt plugin first and df_encrypt plugin second; when the field only records decryption or encryption processes, the corresponding single Transformer plugin is retained. By assembling the Transformer plugins corresponding to each field according to the field arrangement order in the encrypted field mapping relationship, a field transformation chain is obtained.
7. The data acquisition encryption method based on DataX for a quantum platform according to claim 1, characterized in that, S4 specifically includes: DataX loads the DataX task JSON configuration and uses the Reader plugin to read source field data according to the encrypted field mapping relationship to form a data record to be converted. The Transformer plugins include the df_decrypt plugin and the df_encrypt plugin; According to the plugin call order of the field transformation chain, the data record to be transformed is passed to the corresponding Transformer plugin in sequence. Based on the field index, encryption / decryption type and key parameters configured in each Transformer plugin, the corresponding field is decrypted or encrypted to obtain the transformed data record. When the same field is configured for decryption and encryption in sequence, the decrypted field data is passed to the corresponding encryption plugin and re-encrypted according to the target encryption policy to obtain the target field data. The Writer plugin writes the target field data into the corresponding field on the target end according to the encrypted field mapping relationship, thus obtaining encrypted synchronization data.
8. The data acquisition encryption method based on DataX for a quantum platform according to claim 7, characterized in that, When the same field is configured for both decryption and encryption, the decrypted field data is passed to the corresponding encryption plugin, and encryption is re-executed according to the target encryption strategy to obtain the target field data. Specifically, this includes: Read the df_decrypt and df_encrypt plugins corresponding to the same field according to the field conversion chain, and obtain the field index that is common to both plugins; The data record to be converted and the df_decrypt plugin parameters are passed to the evaluate method of the df_decrypt plugin. The source ciphertext field data is read from the data record to be converted according to the field index, and the corresponding decryption type and key parameters are read from the df_decrypt plugin parameters. The source-end ciphertext field data is decrypted using the decryption type and key parameters, and the decrypted field data is written back to the position corresponding to the field index to form a decrypted data record. The decrypted data record and df_encrypt plugin parameters are passed to the evaluate method of the df_encrypt plugin. The decrypted field data is read according to the same field index, and the corresponding encryption type and key parameters are read from the df_encrypt plugin parameters. The decrypted field data is encrypted using the encryption type and key parameters, and the obtained target field data is written back to the position corresponding to the field index to form a re-encrypted transformed data record. The transformed data record continues to be passed along the field transformation chain until the Transformer plugin call in the field transformation chain is completed.
9. The data acquisition encryption method based on DataX for a quantum platform according to claim 1, characterized in that, S5 specifically includes: Read the encrypted synchronization data written by the target end, call the decryption interface of the quantum encryption platform according to the encryption strategy of the target end, and decrypt the ciphertext field in the encrypted synchronization data to obtain the verification field data; The data in the verification field is compared for consistency with the data content of the corresponding field in the source. When the data in the verification field matches the data in the corresponding field on the source end, the data collection and encryption are considered complete.
