Real-time big data stream privacy protection method and device based on Kafka

Abstract

The invention relates to a Kafka-based real-time big data stream privacy protection method and device. Comprising the steps of receiving real-time data streams; preposed sensitive information identification is carried out on the real-time data stream, the preposed sensitive information identification is used for identifying static sensitive information, hierarchical desensitization processing is carried out on the static sensitive information, and a first desensitization data stream is obtained; post-desensitization processing is carried out on the first desensitization data stream, the post-desensitization processing is used for identifying dynamic sensitive information in the first desensitization data stream according to processing service information of data users, desensitization processing is carried out on the dynamic sensitive information according to the data users, and second desensitization data streams provided for the data users are obtained; and sending the second desensitized data stream to a corresponding data user for data processing. Dynamic desensitization is carried out for different data users, it is ensured that the desensitized data stream can be normally subjected to service processing, it is avoided that excessive plaintext data is provided for the data users, and the privacy security of the data is ensured.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and in particular, to a method and device for privacy protection of real-time big data streams based on Kafka. Background Art

[0002] As a distributed stream processing platform, Kafka is widely used in real-time data stream processing due to its high throughput and reliability. However, Kafka itself lacks built-in support for privacy protection. Therefore, how to integrate an efficient privacy protection mechanism into the Kafka real-time data stream processing framework has become an urgent problem to be solved. Summary of the Invention

[0003] To solve the problems existing in the prior art, the embodiments of this specification provide a method and device for privacy protection of real-time big data streams based on Kafka, so as to achieve efficient and low-latency privacy protection, ensure the security of data during transmission, processing, and storage, and at the same time meet the performance requirements of large-scale data stream processing.

[0004] The specific technical solutions of the embodiments of this specification are as follows:

[0005] On the one hand, the embodiments of this specification provide a method for privacy protection of real-time big data streams based on Kafka, and the method includes:

[0006] Receiving a real-time data stream;

[0007] Performing pre-sensitive information identification on the real-time data stream, where the pre-sensitive information identification is used to identify static sensitive information, and performing hierarchical desensitization processing on the static sensitive information to obtain a first desensitized data stream;

[0008] Performing post-desensitization processing on the first desensitized data stream, where the post-desensitization processing is used to identify dynamic sensitive information in the first desensitized data stream according to the processing service information of at least one data user, and performing desensitization processing on the dynamic sensitive information according to each data user to obtain a second desensitized data stream provided to each data user;

[0009] Sending the second desensitized data stream to the corresponding data user for data processing.

[0010] Further, before performing post-desensitization processing on the first desensitized data stream, the method further includes:

[0011] Receiving the processing service information provided by the data user;

[0012] Generating a dynamic desensitization strategy corresponding to the data user according to the processing service information;

[0013] Performing post-desensitization processing on the first desensitized data stream further includes:

[0014] Extracting at least one field and the corresponding field value that are not desensitized in the first desensitized data stream, and using the extracted field as the field to be analyzed for desensitization;

[0015] Executing the dynamic desensitization policy on the field to be analyzed for desensitization, and determining whether the field to be analyzed for desensitization is a first sensitive field. If so, using the field value of this field as the dynamic sensitive information;

[0016] Performing desensitization processing on the dynamic sensitive information.

[0017] Further, executing the dynamic desensitization policy on the field to be analyzed for desensitization and determining whether the field to be analyzed for desensitization is a first sensitive field further includes:

[0018] Identifying the semantics of the field to be analyzed for desensitization and matching the semantics with the processing service information;

[0019] If the matching is not successful, determining whether the field to be analyzed for desensitization contains sensitive information;

[0020] If it contains sensitive information, the field to be analyzed for desensitization is the first sensitive field.

[0021] Further, when receiving the processing service information provided by the data user, the method further includes:

[0022] Analyzing the network security level of the data user;

[0023] Generating the dynamic desensitization policy corresponding to the data user according to the processing service information further includes:

[0024] Generating the dynamic desensitization policy corresponding to the data user according to the processing service information and the network security level;

[0025] Performing desensitization processing on the dynamic sensitive information further includes:

[0026] Performing hierarchical desensitization processing on the dynamic sensitive information corresponding to the network security level.

[0027] Further, the method further includes:

[0028] Receiving the first key provided by the data user;

[0029] After obtaining the second desensitized data stream provided to each data user, the method further includes:

[0030] Encrypt the corresponding second desensitized data stream using the first key to obtain an encrypted data stream;

[0031] Sending the second desensitized data stream to the corresponding data user for data processing further includes:

[0032] Send the encrypted data stream to the corresponding data user so that the data user can decrypt the encrypted data stream using the second key corresponding to the first key and then perform the data processing.

[0033] Further, the pre-sensitive information identification of the real-time data stream further includes:

[0034] Use a static sensitive identification model to identify each field in the real-time data stream, and determine the second sensitive field and the corresponding sensitive level;

[0035] Use the field value corresponding to the second sensitive field in the real-time data stream as the static sensitive information;

[0036] Perform hierarchical desensitization processing on the static sensitive information corresponding to the sensitive level.

[0037] Further, the hierarchical desensitization processing on the static sensitive information corresponding to the sensitive level further includes:

[0038] Determine the encryption algorithm corresponding to the sensitive level;

[0039] Use the encryption algorithm to encrypt the static sensitive information to obtain the first desensitized data stream including the encrypted static sensitive information.

[0040] On the other hand, an embodiment of this specification also provides a real-time big data stream privacy protection device based on Kafka, and the device includes:

[0041] APP reporting information receiving unit, configured to receive the service status of the user and the terminal information reported by the terminal APP when it is opened;

[0042] Data stream receiving unit, configured to receive a real-time data stream;

[0043] Pre-desensitization unit, configured to perform pre-sensitive information identification on the real-time data stream, where the pre-sensitive information identification is used to identify static sensitive information, perform hierarchical desensitization processing on the static sensitive information, and obtain a first desensitized data stream;

[0044] A post-desensitization unit for performing post-desensitization processing on the first desensitized data stream. The post-desensitization processing is used to identify dynamic sensitive information in the first desensitized data stream according to the processing service information of at least one data user, and desensitize the dynamic sensitive information separately according to the data user to obtain a second desensitized data stream provided to each data user;

[0045] A data distribution unit for sending the second desensitized data stream to the corresponding data user for data processing.

[0046] On the other hand, an embodiment of this specification also provides a computer device, including a memory, a processor, and a computer program stored on the memory. When the processor executes the computer program, the above method is implemented.

[0047] On the other hand, an embodiment of this specification also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above method is implemented.

[0048] Using the embodiment of this specification, the pre-sensitive information identification and post-desensitization processing can desensitize the data stream in all directions, and realize dynamic desensitization of the data stream according to the services processed by different data users. Thus, while ensuring that the data user can normally process the desensitized data stream, it is avoided to provide too much plaintext data to the data user, ensuring the privacy and security of the data. Description of the Drawings

[0049] In order to more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the embodiments of this specification. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0050] Figure 1 It shows a schematic flowchart of a real-time big data stream privacy protection method based on Kafka in an embodiment of this specification;

[0051] Figure 2 It shows a schematic flowchart of pre-sensitive information identification for the real-time data stream in an embodiment of this specification;

[0052] Figure 3 It shows a schematic flowchart of performing classification desensitization processing corresponding to the sensitive level on the static sensitive information in an embodiment of this specification;

[0053] Figure 4The figure shows a schematic flowchart of post-desensitization processing for the first desensitized data stream in the embodiments of this specification;

[0054] Figure 5 The figure shows a schematic flowchart of dynamic hierarchical desensitization in the embodiments of this specification;

[0055] Figure 6 The figure shows a schematic flowchart of encrypting the second desensitized data stream in the embodiments of this specification;

[0056] Figure 7 The figure shows a schematic structural diagram of a real-time big data stream privacy protection device based on Kafka in the embodiments of this specification;

[0057] Figure 8 The figure shows a schematic structural diagram of a computer device in the embodiments of this specification.

[0058]

Explanation of the reference numerals

[0059] 701, data stream receiving unit;

[0060] 702, pre-desensitization unit;

[0061] 703, post-desensitization unit;

[0062] 704, data distribution unit;

[0063] 802, computer device;

[0064] 804, processing device;

[0065] 806, storage resource;

[0066] 808, driving mechanism;

[0067] 810, input / output module;

[0068] 812, input device;

[0069] 814, output device;

[0070] 816, presentation device;

[0071] 818, graphical user interface;

[0072] 820, network interface;

[0073] 822, communication link;

[0074] 824, communication bus. Detailed implementation manners

[0075] The following will clearly and completely describe the technical solutions in the embodiments of this specification with reference to the accompanying drawings in the embodiments of this specification. Obviously, the described embodiments are only a part of the embodiments of this specification, rather than all of them. Based on the embodiments in this specification, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of the embodiments of this specification.

[0076] It should be noted that the terms "first", "second", etc. in the specification, claims, and the above-mentioned accompanying drawings of the embodiments of this specification are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the embodiments of this specification described here can be implemented in an order other than those illustrated or described here. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, device, product, or equipment that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or equipment.

[0077] It should be noted that in the technical solutions of the embodiments of this specification, the acquisition, storage, use, processing, etc. of data all comply with the relevant regulations of national laws and regulations.

[0078] It should be noted that in the embodiments of this specification, some industry-existing solutions such as certain software, components, models, etc. may be mentioned. They should be regarded as exemplary, and their purpose is only to illustrate the feasibility in the implementation of the technical solutions of this application, but it does not mean that the applicant has already or necessarily used this solution.

[0079] Aiming at the problems existing in the prior art, the embodiments of this specification provide a real-time big data stream privacy protection method based on Kafka. Through pre-sensitive information identification and post-desensitization processing, efficient and low-latency privacy protection is achieved, ensuring the security of data during transmission, processing, and storage, while meeting the performance requirements of large-scale data stream processing. Figure 1 The following shows a schematic flowchart of a real-time big data stream privacy protection method based on Kafka in the embodiments of this specification. The process of desensitizing the data stream is described in this figure. The order of steps listed in the embodiments is only one way among the execution orders of numerous steps, and does not represent the only execution order. When the actual system or device product is executed, it can be executed in the order of the method shown in the embodiments or the accompanying drawings, or in parallel. Specifically, as Figure 1 shown, the method may include:

[0080] Step 101: Receive a real-time data stream;

[0081] Step 102: Perform pre-sensitive information identification on the real-time data stream. The pre-sensitive information identification is used to identify static sensitive information, and perform hierarchical desensitization processing on the static sensitive information to obtain a first desensitized data stream.

[0082] Step 103: Perform post-desensitization processing on the first desensitized data stream. The post-desensitization processing is used to identify dynamic sensitive information in the first desensitized data stream according to the processing service information of at least one data user, and perform desensitization processing on the dynamic sensitive information according to the data user respectively to obtain a second desensitized data stream provided to each data user.

[0083] Step 104: Send the second desensitized data stream to the corresponding data user for data processing.

[0084] Using the embodiments of this specification, the pre-sensitive information identification and post-desensitization processing can desensitize the data stream comprehensively, and realize dynamic desensitization of the data stream according to the services processed by different data users. Therefore, while ensuring that the data user can normally process the desensitized data stream, it is possible to avoid providing too much plaintext data to the data user and ensure the privacy and security of the data.

[0085] In the embodiments of this specification, first, a real-time data stream is received, and preprocessing is performed on the data stream to ensure the consistency and quality of the data format.

[0086] Specifically, the real-time data stream can be sent through various data sources such as API interfaces, sensors, and log files, and efficiently written into the data through Kafka Producer. Then, various heterogeneous formats (such as JSON, Avro, CSV) can be converted into a unified internal format to reduce the complexity of subsequent processing. Use Schema Registry for data format verification. SchemaRegistry supports version control of Schema, enabling the data format to remain compatible during evolution. The data user can be a data warehouse (such as HBase, Hive), a real-time analysis platform (such as Flink, Elasticsearch), or an alarm system, etc., and the embodiments of this specification do not make any restrictions.

[0087] In some other embodiments of this specification, in order to ensure that the data stream is not tampered with during transmission and processing and to ensure the authenticity and integrity of the data. The embodiments of this specification also generate a unique hash value for each data stream and append it to the data stream. After receiving the second desensitized data stream, the data user recalculates the hash value and compares it. If the comparison is consistent, it means that the data stream has not been tampered with.

[0088] For large data streams, the data stream is also divided into multiple segments, and then each segment is signed. After receiving the second desensitized data stream, the data user calculates the signature of each segment and verifies the data integrity by verifying the signature.

[0089] The Offset mechanism of Kafka can also be used to monitor the order and integrity of messages, avoiding message loss or repeated consumption, which is not limited in the embodiments of this specification.

[0090] After receiving the data stream in the embodiments of this specification, pre-sensitive information identification is first performed on the data stream. Specifically, as Figure 2 shown, the pre-sensitive information identification of the real-time data stream further includes:

[0091] Step 201: Use a static sensitive identification model to identify each field in the real-time data stream, and determine the second sensitive field and the corresponding sensitive level;

[0092] Step 202: Use the field value corresponding to the second sensitive field in the real-time data stream as the static sensitive information;

[0093] Step 203: Perform hierarchical desensitization processing on the static sensitive information corresponding to the sensitive level.

[0094] In the embodiments of this specification, sensitive data is pre-classified, such as high sensitive level, medium sensitive level, low sensitive level, etc. And encryption algorithms corresponding to each sensitive level are predefined.

[0095] The static sensitive identification model in the embodiments of this specification can be a rule engine, dictionary matching, machine learning model, etc. The rule engine can quickly identify sensitive fields (such as ID numbers and mobile phone numbers) in structured data based on preset regular expressions and keyword matching. Dictionary matching can use a sensitive information dictionary in a specific domain (such as financial domain terms) for quick comparison. Machine learning models can include support vector machines (SVMs), decision trees, deep learning models, etc., which can identify more complex and variable sensitive data, especially when the data format is not fixed and contains a large amount of unstructured text. These models can continuously optimize the model performance through incremental learning and online learning.

[0096] After the static sensitive identification model identifies the second sensitive field, the field value corresponding to the second sensitive field is the static sensitive information. The desensitization of the static sensitive information has nothing to do with business processing. After desensitizing the static sensitive information, the data user can still process the business normally.

[0097] After determining the static sensitive information, as Figure 3 shown, the hierarchical desensitization processing of the static sensitive information corresponding to the sensitive level further includes:

[0098] Step 301: Determine the encryption algorithm corresponding to the sensitivity level;

[0099] Step 302: Use the encryption algorithm to encrypt the static sensitive information to obtain the first desensitized data stream including the encrypted static sensitive information.

[0100] In the embodiments of this specification, the staff can pre-associate multiple encryption algorithms with multiple sensitivity levels based on experience. When determining the sensitivity level corresponding to the second sensitive field, the encryption algorithm corresponding to this sensitivity level is determined according to the association relationship, and then the static sensitive information is encrypted using the encryption algorithm.

[0101] After the pre-sensitive recognition desensitizes the static sensitive information to obtain the first desensitized data stream, the post-desensitization process performs dynamic desensitization on the first desensitized data stream to adapt to different data users while ensuring the privacy security of more data.

[0102] The post-desensitization process is considered based on the following situations:

[0103] The dynamic encryption module is considered based on the following situations:

[0104] The encryption method of the pre-module is static, that is, it has determined which fields to encrypt before the data enters Kafka.

[0105] The privacy requirements of different business parties are different (some downstream parties only need partial desensitization).

[0106] The data levels are different (some data may need to be further hidden).

[0107] Specifically, as Figure 4 shown, before performing the post-desensitization process on the first desensitized data stream, the method further includes:

[0108] Step 401: Receive the processing service information provided by the data user;

[0109] Step 402: Generate a dynamic desensitization policy corresponding to the data user according to the processing service information;

[0110] Performing the post-desensitization process on the first desensitized data stream further includes:

[0111] Step 403: Extract at least one field and the corresponding field value in the first desensitized data stream that have not been desensitized, and use the extracted fields as the fields to be desensitized and analyzed;

[0112] Step 404: Execute the dynamic desensitization policy on the to-be-desensitized analysis field, and determine whether the to-be-desensitized analysis field is a first sensitive field. If so, use the field value of this field as the dynamic sensitive information;

[0113] Step 405: Perform desensitization processing on the dynamic sensitive information.

[0114] In the embodiments of this specification, the processing business information represents the business types processed by the data user. For example, for consumption data, the consumption amount, consumption category, or consumption area, etc. can be analyzed. The embodiments of this specification generate a corresponding dynamic desensitization policy for the data user according to the processing business information.

[0115] When performing dynamic desensitization, first extract at least one non-desensitized field in the first desensitized data stream that has been identified and desensitized by the pre-sensitive information, and the corresponding field value. Dynamic desensitization is to use the dynamic desensitization policy corresponding to the processing business information of the data user to identify which of the non-desensitized fields in the first desensitized data stream are fields that need to be dynamically desensitized and which are fields that do not need to be dynamically desensitized.

[0116] According to an embodiment of this specification, further including for executing the dynamic desensitization policy on the to-be-desensitized analysis field and determining whether the to-be-desensitized analysis field is a first sensitive field:

[0117] Identify the semantics of the to-be-desensitized analysis field, and match the semantics with the processing business information;

[0118] If the matching is not successful, determine whether the to-be-desensitized analysis field contains sensitive information;

[0119] If it contains sensitive information, the to-be-desensitized analysis field is the first sensitive field.

[0120] In the embodiments of this specification, if the semantics of the to-be-desensitized analysis field do not match the processing business information, it means that there is no need to provide the field value in the plaintext form of this to-be-analyzed desensitized field to this data user. Therefore, the field value corresponding to this to-be-analyzed desensitized field may need to be desensitized. However, since not all fields record the privacy data of users, the embodiments of this specification also determine whether the fields that may need to be desensitized contain sensitive information. If so, it means that the field value of this field needs to be desensitized. If not, since the field that needs to be desensitized does not contain sensitive information, there is no need to desensitize the field value of this field.

[0121] Through the above method, it is achieved that the desensitized content in the second desensitized data does not affect the normal processing of the business, and at the same time, the risk of privacy data leakage can be reduced.

[0122] It should be noted that the processing service information of different data users can be different. Therefore, the corresponding dynamic desensitization policies for different data users can also be different. As a result, after the same first desensitized data stream is desensitized by the dynamic desensitization policies of different data users, the desensitized fields and the fields in plaintext form in the obtained second desensitized data stream can also be different, thus realizing personalized desensitization for data users processing different services without modifying the processing logic for identifying pre-sensitive information, reducing the development workload. If a new data user is added, only a dynamic desensitization policy needs to be generated according to the processing service information of the newly added data user, flexibly adapting to the desensitization requirements of different data users.

[0123] According to an embodiment of the present specification, in order to prevent a malicious eavesdropper from reverse calculating the desensitized field values in the second desensitized data stream after intercepting the second desensitized data stream sent to the data user, the embodiment of the present specification introduces a hierarchical desensitization method. Specifically, as Figure 5 shown, when receiving the processing service information provided by the data user, the method further includes:

[0124] Step 501: Analyze the network security level of the data user;

[0125] Generating the dynamic desensitization policy corresponding to the data user according to the processing service information further includes:

[0126] Step 502: Generate the dynamic desensitization policy corresponding to the data user according to the processing service information and the network security level;

[0127] Performing desensitization processing on the dynamic sensitive information further includes:

[0128] Step 503: Perform hierarchical desensitization processing on the dynamic sensitive information corresponding to the network security level.

[0129] In the embodiment of the present specification, if the network security level is low, it means that the data stream is easily eavesdropped during the distribution process. On the contrary, if the network security level is high, it means that the data stream is not easily eavesdropped during the distribution process. Therefore, in the embodiment of the present specification, each network security level is associated with an encryption algorithm in advance. The lower the network security level, the higher the encryption strength of the encryption algorithm is required. On the contrary, the higher the network security level, the lower the encryption strength of the encryption algorithm can be appropriately. So as to determine the encryption algorithm corresponding to the network security level according to the association relationship, and use the determined encryption algorithm to encrypt the identified sensitive information to avoid data leakage caused by a low network security level.

[0130] Feasibly, the network security level of the data user can be analyzed in combination with the network type, protocol, encryption method, and the security API of the device or system. The network security level of the data user can also be evaluated through penetration testing, network vulnerability scanning, etc., and the embodiments of this specification do not make limitations.

[0131] In addition, if the encryption strength of the encryption algorithm corresponding to the network security level is higher than the encryption strength of the encryption algorithm corresponding to the sensitivity level of the static sensitive information in the pre - sensitive information identification process, it indicates that there is a risk of leakage of the pre - desensitized static sensitive information at the current network security level. For example, an attacker may obtain the plaintext information through a limited number of attempts. Therefore, in the embodiments of this specification, if the encryption strength of the encryption algorithm corresponding to the network security level is higher than the encryption strength of the encryption algorithm corresponding to the sensitivity level of the static sensitive information in the pre - sensitive information identification process, the encrypted static sensitive information is re - encrypted according to the encryption algorithm corresponding to the network security level (or an encryption algorithm whose encryption strength matches the encryption strength of the encryption algorithm corresponding to the network security level). For example, if the sensitivity level of the static sensitive information is low and a lightweight processing scheme (such as masking) is used to desensitize the static sensitive information, but in the post - desensitization process, it is found that the encryption algorithm corresponding to the network security level is strong encryption (such as AES encryption), so the desensitized static sensitive information is re - encrypted using strong encryption (such as AES encryption). Optionally, the content of the mask in the static sensitive information can be filled with missing values and then strongly encrypted.

[0132] To further improve privacy security, according to an embodiment of this specification, as Figure 6 shown, the method further includes:

[0133] Step 601: Receive a first key provided by the data user;

[0134] After obtaining the second desensitized data stream provided to each data user, the method further includes:

[0135] Step 602: Encrypt the corresponding second desensitized data stream using the first key to obtain an encrypted data stream;

[0136] Sending the second desensitized data stream to the corresponding data user for data processing further includes:

[0137] Step 603: Send the encrypted data stream to the corresponding data user so that the data user can decrypt the encrypted data stream using a second key corresponding to the first key and then perform the data processing.

[0138] In the embodiments of this specification, the first key and the second key may be a public-private key pair, where the first key is the public key and the second key is the private key. The data user provides the public key to Kafka, and then Kafka encrypts the second desensitized data stream using the public key and sends the encrypted data to the data user. The data user then decrypts the encrypted second desensitized data stream using the private key. Therefore, even if the second desensitized data stream is maliciously intercepted during the distribution process, the interceptor cannot decrypt the intercepted data stream because it cannot obtain the private key, which greatly improves the security of the data stream.

[0139] In some other embodiments of this specification, the fields that need to be encrypted twice can also be determined according to the dynamic desensitization policy.

[0140] Specifically, the method further includes: identifying the semantics of the field to be desensitized and analyzed, and matching the semantics with the processing service information;

[0141] If the match is successful, it is determined whether the field to be desensitized and analyzed contains sensitive information;

[0142] If it contains sensitive information, the field to be desensitized and analyzed is the field that needs to be encrypted twice.

[0143] Then, the value of the field that needs to be encrypted twice is used as the data to be encrypted twice, and the data to be encrypted twice is encrypted using the first key to obtain the ciphertext of the data to be encrypted twice. The data stream including the ciphertext of the data to be encrypted twice, the desensitized field value of the first sensitive field, and the desensitized static sensitive information is sent to the data user. The above field values in the data stream are all ciphertexts. Only when the data user decrypts the ciphertext of the data to be encrypted twice in the received data stream using the correct second key can it obtain the plaintext information and perform business processing normally, thereby improving the security of the data stream, reducing the computational amount of double encryption / decryption, and improving the real-time performance of the data stream.

[0144] Based on the same inventive concept, the embodiments of this specification also provide a real-time big data stream privacy protection device based on Kafka, as Figure 7 shown, including:

[0145] A data stream receiving unit 701, configured to receive a real-time data stream;

[0146] A pre-desensitization unit 702, configured to perform pre-sensitive information identification on the real-time data stream. The pre-sensitive information identification is used to identify static sensitive information and perform hierarchical desensitization processing on the static sensitive information to obtain a first desensitized data stream;

[0147] A post-desensitization unit 703 is configured to perform post-desensitization processing on the first desensitized data stream. The post-desensitization processing is used to identify dynamic sensitive information in the first desensitized data stream according to the processing service information of at least one data user, and perform desensitization processing on the dynamic sensitive information respectively according to the data user to obtain a second desensitized data stream provided to each data user.

[0148] A data distribution unit 704 is configured to send the second desensitized data stream to the corresponding data user for data processing.

[0149] The beneficial effects obtained by the above device are the same as those obtained by the above method, and are not described in detail in the embodiments of this specification.

[0150] As Figure 8 shown in the structural schematic diagram of the computer device in the embodiments of this specification, the method in the present invention can be applied to the computer device in this embodiment. The computer device 802 may include one or more processing devices 804, such as one or more central processing units (CPUs), and each processing unit may implement one or more hardware threads.

[0151] The computer device 802 may further include any storage resource 806, which is used to store any type of information such as code, settings, data, etc.

[0152] Non-limiting, for example, the storage resource 806 may include any one or more combinations of the following: any type of RAM, any type of ROM, flash memory devices, hard disks, optical discs, etc.

[0153] More generally, any storage resource may use any technology to store information.

[0154] Furthermore, any storage resource may provide volatile or non-volatile retention of information.

[0155] Furthermore, any storage resource may represent a fixed or removable component of the computer device 802.

[0156] In one case, when the processing device 804 executes the associated instructions stored in any storage resource or combination of storage resources, the computer device 802 may perform any operation of the associated instructions. The computer device 802 further includes one or more drive mechanisms 808 for interacting with any storage resource, such as a hard disk drive mechanism, an optical disc drive mechanism, etc.

[0157] The computer device 802 may also include an input / output module 810 (I / O) for receiving various inputs (via the input device 812) and for providing various outputs (via the output device 814). A specific output mechanism may include a presentation device 816 and an associated graphical user interface (GUI) 818. In other embodiments, the input / output module 810 (I / O), the input device 812, and the output device 814 may not be included, and it may only be a computer device in the network. The computer device 802 may also include one or more network interfaces 820 for exchanging data with other devices via one or more communication links 822. One or more communication buses 824 couple the components described above together.

[0158] The communication link 822 may be implemented in any manner, for example, through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 822 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc. governed by any protocol or combination of protocols.

[0159] The embodiments of this specification also provide a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the above method is implemented.

[0160] The embodiments of this specification also provide a computer-readable instruction, and when the processor executes the instruction, the program therein causes the processor to execute the above method.

[0161] It should be understood that in various embodiments of the embodiments of this specification, the magnitudes of the sequence numbers of the above processes do not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of this specification.

[0162] It should also be understood that in the embodiments of this specification, the term "and / or" is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent three cases: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in the embodiments of this specification generally represents an "or" relationship between the associated objects before and after.

[0163] Those of ordinary skill in the art will realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in this specification can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to their functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this specification's embodiments.

[0164] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated here.

[0165] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there can be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed couplings or direct couplings or communication connections to each other can be indirect couplings or communication connections through some interfaces, devices, or units, and can also be electrical, mechanical, or other forms of connection.

[0166] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this specification's embodiments.

[0167] In addition, in each embodiment of this specification, the various functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0168] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this specification, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this specification. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0169] In the embodiments of this specification, specific embodiments are used to elaborate on the principles and implementation manners of the embodiments of this specification. The descriptions of the above embodiments are only used to help understand the methods and their core ideas of the embodiments of this specification; at the same time, for those of ordinary skill in the art, according to the ideas of the embodiments of this specification, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the embodiments of this specification.

Claims

1. A real-time big data stream privacy protection method based on Kafka, characterized in that, The method includes: Receiving a real-time data stream; Performing pre-sensitive information identification on the real-time data stream, where the pre-sensitive information identification is used to identify static sensitive information, and performing hierarchical desensitization processing on the static sensitive information to obtain a first desensitized data stream; Performing post-desensitization processing on the first desensitized data stream, where the post-desensitization processing is used to identify dynamic sensitive information in the first desensitized data stream according to the processing service information of at least one data user, and performing desensitization processing on the dynamic sensitive information according to each data user to obtain a second desensitized data stream provided to each data user; Sending the second desensitized data stream to the corresponding data user for data processing.

2. The method according to claim 1, characterized in that Before performing post-desensitization processing on the first desensitized data stream, the method further includes: Receiving the processing service information provided by the data user; Generating a corresponding dynamic desensitization policy for the data user according to the processing service information; Performing post-desensitization processing on the first desensitized data stream further includes: Extracting at least one un-desensitized field and the corresponding field value in the first desensitized data stream, and using the extracted field as the field to be desensitized and analyzed; Executing the dynamic desensitization policy on the field to be desensitized and analyzed, and determining whether the field to be desensitized and analyzed is a first sensitive field. If so, using the field value of the field as the dynamic sensitive information; Performing desensitization processing on the dynamic sensitive information.

3. The method according to claim 2, wherein Executing the dynamic desensitization policy on the field to be desensitized and analyzed, and determining whether the field to be desensitized and analyzed is a first sensitive field further includes: Identifying the semantics of the field to be desensitized and analyzed, and matching the semantics with the processing service information; If the matching is not successful, determining whether the field to be desensitized and analyzed contains sensitive information; If it contains sensitive information, the field to be desensitized and analyzed is the first sensitive field.

4. The method according to claim 2, wherein When receiving the processing service information provided by the data user, the method further includes: Analyzing the network security level of the data user; Generating a corresponding dynamic desensitization policy for the data user according to the processing service information further includes: Generating a corresponding dynamic desensitization policy for the data user according to the processing service information and the network security level; Performing desensitization processing on the dynamic sensitive information further includes: Performing hierarchical desensitization processing on the dynamic sensitive information corresponding to the network security level.

5. The method according to claim 1, wherein The method further includes: Receiving a first key provided by the data user; After obtaining the second desensitized data stream provided to each data user, the method further includes: Encrypting the corresponding second desensitized data stream with the first key to obtain an encrypted data stream; Sending the second desensitized data stream to the corresponding data user for data processing further includes: Sending the encrypted data stream to the corresponding data user, so that the data user can decrypt the encrypted data stream with a second key corresponding to the first key and then perform the data processing.

6. The method according to claim 1, characterized in that Performing pre-sensitive information identification on the real-time data stream further includes: Identify each field in the real-time data stream using a static sensitive identification model to determine the second sensitive field and the corresponding sensitivity level; Use the field value corresponding to the second sensitive field in the real-time data stream as the static sensitive information; Perform hierarchical desensitization processing on the static sensitive information corresponding to the sensitivity level; 7. The method according to claim 6, wherein Performing hierarchical desensitization processing on the static sensitive information corresponding to the sensitivity level further includes: Determine the encryption algorithm corresponding to the sensitivity level; Use the encryption algorithm to encrypt the static sensitive information to obtain the first desensitized data stream including the encrypted static sensitive information.

8. A real-time big data stream privacy protection device based on Kafka, characterized in that The device includes: A data stream receiving unit for receiving a real-time data stream; A pre-desensitization unit for performing pre-sensitive information identification on the real-time data stream. The pre-sensitive information identification is used to identify static sensitive information, perform hierarchical desensitization processing on the static sensitive information, and obtain a first desensitized data stream; A post-desensitization unit for performing post-desensitization processing on the first desensitized data stream. The post-desensitization processing is used to identify dynamic sensitive information in the first desensitized data stream according to the processing service information of at least one data user, and perform desensitization processing on the dynamic sensitive information according to the data user respectively to obtain a second desensitized data stream provided to each data user; A data distribution unit for sending the second desensitized data stream to the corresponding data user for data processing.

9. A computer device, comprising a memory, a processor, and a computer program stored on the memory, characterized in that, When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images