Big data efficient asynchronous and synchronous decryption processing method, module and system

Abstract

The invention discloses a big data efficient asynchronous decryption processing method, module and system. The method comprises that data is synchronously pushed to a Kafka cluster; ETL processing onthe corresponding Topic by using the Kafka consumption terminal; if the number of the data pieces corresponding to the same Last Up Date Time is smaller than or equal to the set N pieces, real-time synchronous decryption is carried out; if the number of the data pieces corresponding to the same Last Up Date Time is greater than the set N pieces, the temporary storage library directly stores the data into a number bin specified table in the distributed database; setting a timing task, reading the to-be-decrypted data stored in the temporary library every time interval, and storing the to-be-decrypted data in a data warehouse specified table in the distributed database after decryption is completed; and continuously running the asynchronous decryption program to ensure that all the to-be-decrypted data can meet the service decryption requirement. According to the big data efficient asynchronous and synchronous decryption processing method, module and system, synchronous and asynchronousdecryption modes are compatible at the same time according to service characteristics, and the large throughput performance requirement is met; and the performance bottleneck is solved, a fault-tolerant mechanism is added, and the requirement for large-batch transaction decryption can be met.

Description

technical field [0001] The invention relates to the field of big data decryption, in particular to a method, module and system for high-efficiency and synchronous decryption of big data. Background technique [0002] Data security has a decisive impact on the survival and development of enterprises. Leakage and destruction of data assets will lead to irreparable economic losses and loss of core competitiveness of enterprises. However, most small and medium-sized enterprises often focus on the rapid development of business and ignore The importance of data security. Data security control has always been an important part of data governance, and data desensitization belongs to the category of data security control. For Internet companies and traditional industries, data security has always been an extremely important and sensitive topic. Data desensitization refers to the transformation of certain sensitive information through desensitization rules to achieve reliable protec...

AI Technical Summary

Problems solved by technology

When dealing with a large amount of data in a similar example scenario, if a temporary real-time decryption method is used, there will obviously be a serious performance bottleneck, and it will not be able to respond to a large number of demands in a short time

Images