A big data parallel desensitization method and system

Abstract

Multiple embodiments of this specification relate to the field of information technology, and specifically to a method and system for parallel desensitization of big data. The method includes the following steps: reading a big data file, calling a sharding service to extract a number of data units from the big data according to the initial sharding size, assigning numbers and including them in a list to be desensitized; calling a number of desensitizing services in parallel to read data units from the list to be desensitized and desensitize them; when desensitization is completed, calling the desensitizing service to feed back the operating data to the sharding service, the operating data including the number, desensitized field table and desensitization time; calling the sharding service to dynamically adjust the sharding size according to all operating data to generate subsequent data units; after all data units are desensitized, calling the merge service to combine all data units according to the number to obtain the desensitized big data.

Description

Technical Field

[0001] Multiple embodiments of this specification relate to the field of information technology, and specifically to a big data parallel desensitization method and system. Background Art

[0002] In the era of rapid development of informatization and digitalization, information technology has penetrated into every corner of social life. Enterprises and organizations have accumulated massive amounts of data in their daily operations, including sensitive information such as personal privacy and trade secrets. With the frequent occurrence of data leakage incidents, the importance of protecting these sensitive data has become increasingly prominent. Therefore, data desensitization, as one of the important means to ensure data security, has become an indispensable part. Data desensitization refers to the process of processing raw data so that it loses its direct association with a specific individual or entity, thereby protecting sensitive information from being identified or leaked. By removing or changing key information in sensitive fields, even if the data is illegally obtained, the attacker cannot easily restore the true identity or other important information. The data after desensitization can be anonymized to a certain extent, which not only ensures the security of the data, but also does not hinder its use in statistical analysis, research and development and other activities, thereby increasing the value of data resources. With the introduction of relevant policies, the legislation on personal information protection has been increasingly improved, and higher requirements have been placed on data processors. In this context, whether to comply with legal regulations or for the needs of their own risk management, data owners must pay attention to and implement effective data desensitization measures. However, the current desensitization technology applied to big data files has the problem of low efficiency. Summary of the invention

[0003] Multiple embodiments of this specification describe a big data parallel desensitization method and system.

[0004] In a first aspect, the embodiments of this specification provide a method for parallel desensitization of big data, including the steps of:

[0005] Read the big data file, call the sharding service to cut a number of data units from the big data according to the initial shard size, assign numbers and include them in the list to be desensitized;

[0006] Calling several desensitization services in parallel to read data units from the to-be-desensitized list and perform desensitization;

[0007] When desensitization is completed, the desensitization service is called to feed back the operation data to the slicing service, and the operation data includes the number, the desensitization field table and the desensitization time;

[0008] Calling the sharding service to dynamically adjust the sharding size according to all running data to generate subsequent data units;

[0009] After all data units are desensitized, the merge service is called to combine all data units according to the number to obtain the desensitized big data.

[0010] In a second aspect, the embodiments of this specification provide a big data parallel desensitization system, including:

[0011] The reading module reads the big data file, calls the sharding service to cut a number of data units from the big data according to the initial sharding size, assigns numbers and puts them into the list to be desensitized;

[0012] A desensitization module, calling several desensitization services in parallel to read data units from the to-be-desensitized list and perform desensitization;

[0013] A feedback module, when desensitization is completed, calls the desensitization service to feed back the operation data to the slicing service, the operation data including the number, the desensitization field table and the desensitization time;

[0014] An adjustment module, calling the sharding service to dynamically adjust the sharding size according to all running data to generate subsequent data units;

[0015] The merging module, after all data units are desensitized, calls the merging service according to the number to combine all data units to obtain the desensitized big data.

[0016] In a third aspect, an embodiment of this specification provides an electronic device, including a processor and a memory;

[0017] The processor is connected to the memory;

[0018] The memory is used to store executable program code;

[0019] The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute the method described in any one of the above aspects.

[0020] In a fourth aspect, an embodiment of the present specification provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method described in any of the above aspects when executed by a processor.

[0021] In a fifth aspect, an embodiment of this specification provides a computer program product, including a computer program, which implements the method described in any of the above aspects when executed by a processor.

[0022] The beneficial effects brought by the technical solutions provided by some embodiments of this specification include at least:

[0023] In multiple embodiments of the present specification, a large data parallel desensitization method and system are provided, which can use dynamic data unit size division when desensitizing large data files in parallel, divide the size of the data unit according to the desensitization situation, and reflect the desensitized content in the control of the desensitization process with higher efficiency. By reflecting the desensitized content, the efficiency of desensitization can be improved. In the process of cutting data units, the desensitization operation is performed at the same time, and there is no need to wait until all data units are cut. At the same time, the cutting of subsequent data units can be adjusted with the help of the desensitization operation of the desensitized data units, so that the data units include as much repeated desensitized content as possible, thereby improving the efficiency of desensitization.

[0024] Other features and advantages of the various embodiments of the present specification will be further disclosed in the following detailed description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of this specification, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 Schematic diagram of the desensitization method provided in the embodiments of this specification.

[0027] Figure 2 A schematic diagram of the desensitization method provided in the embodiments of this specification.

[0028] Figure 3 A schematic diagram of adjusting the slice size provided in an embodiment of this specification.

[0029] Figure 4 A schematic diagram of the flow of a method for extracting data units provided in an embodiment of this specification.

[0030] Figure 5 A schematic flow chart of a method for generating subsequent data units provided in an embodiment of this specification.

[0031] Figure 6 A schematic flow chart of another method for generating subsequent data units provided in an embodiment of this specification.

[0032] Figure 7 A schematic diagram of the deployment architecture of the desensitization method provided in the embodiments of this specification.

[0033] Figure 8 A schematic diagram of the desensitization interaction interface provided in an embodiment of this specification.

[0034] Fig. 9 A schematic diagram of a desensitization system provided in an embodiment of this specification.

[0035] Fig.10 A schematic diagram of an electronic device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0036] The following is an explanation and description of the technical solutions of the embodiments of this specification in conjunction with the drawings of the embodiments of this specification, but the following embodiments are only preferred embodiments of this specification, not all. Based on the embodiments in the implementation mode, other embodiments obtained by those skilled in the art without creative work are all within the scope of protection of this specification.

[0037] The terms "first", "second", "third", etc. in the description and claims of this specification and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0038] In the following description, terms such as "inside", "outside", "up", "down", "left", "right", etc. that indicate directions or positional relationships are only used to facilitate the description of the embodiments and simplify the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as a limitation of this specification.

[0039] The data involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection of relevant data complies with the relevant laws, regulations and standards of relevant countries and regions.

[0040] Before describing the technical solution in this specification, an introduction is given to the application scenarios and related technologies of the technical solution.

[0041] With the development of informatization, information systems have been collecting a large amount of information non-stop. For example, the operators of shopping websites collect users' browsing history, purchase history, preference settings and other data. These data are not only used to improve user experience and personalized recommendations, but also help merchants conduct market analysis and decision-making; insurance companies collect detailed information such as the insured's health status, family background, and financial situation to assess risk levels, determine premiums, and design insurance products; banks collect customers' account transaction records, credit scores, asset allocation and other financial information in order to provide more accurate financial services, such as loan approval and personal financial management advice. This information contains a large amount of sensitive information of users, such as personal identity identification information (PII), medical health records, financial details, etc. Once leaked, it will not only infringe on personal privacy, but may also lead to serious consequences such as fraud and identity theft.

[0042] Data desensitization helps protect information security. By removing or changing key information in sensitive fields, it is difficult for attackers to restore the real identity or other important information even if the data is illegally obtained. Data desensitization can also promote data sharing and utilization. Properly processed data can be anonymized to a certain extent, which does not affect its use for statistical analysis, machine learning training and other purposes, and ensures the security of the data. Finally, good desensitization measures can enhance the public's trust in enterprises and institutions, making people more willing to share their data, thereby promoting business innovation and service improvement. The desensitization process usually includes the following steps: first, identify the type of sensitive data that needs to be protected, which requires a deep understanding of business needs and legal and regulatory requirements; then select appropriate desensitization algorithms and technical means, such as replacement, generalization, and noise addition to mask the original data; then perform desensitization operations on the data while maintaining the availability and accuracy of the data; finally, verify the desensitization effect to ensure that the processed data cannot be easily re-associated with specific individuals or entities.

[0043] When processing big data files 11, current desensitization technology usually processes the big data files 11 by slicing them. For example, it divides them into 5MB sub-files or divides them into 5-line or 10-line sub-files. The simplest way to desensitize is to replace sensitive data with a preset replacement character, such as the following: Figure 1As shown in Figure 1. On the one hand, the currently used desensitization methods are difficult to efficiently process large-scale data sets due to their high computational complexity. On the other hand, when it comes to multidimensional data or time series data, how to ensure the consistency and coherence of the desensitized data is also a difficult problem. In addition, some desensitization methods may introduce biases, affecting the reliability of subsequent analysis results. The existence of these problems leads to slow data processing, high resource consumption, and potential data quality degradation. In order to address the above problems, researchers are exploring more efficient desensitization technologies and tools, aiming to find methods that can quickly process massive data while maintaining good data quality and privacy protection levels. For example, a distributed computing framework is used to accelerate the desensitization process, or an artificial intelligence algorithm is combined to automatically adjust the desensitization parameters to optimize performance. At the same time, some scholars have proposed the possibility of using encryption technology to perform secure computing without exposing the original data, which will open up new paths for the development of desensitization technology in the future. In short, in the face of the growing amount of big data and complex privacy protection needs, it is crucial to continue to research and develop adaptive and efficient desensitization technologies.

[0044] This manual first provides a large data parallel desensitization method. Figure 2 , including the steps of:

[0045] Step S101) Read the big data file 11, call the slicing service 21 to cut a number of data units 12 from the big data according to the initial slicing size, assign numbers and put them into the list to be desensitized;

[0046] Step S102) calling several desensitization services 22 in parallel to read the data unit 12 from the to-be-desensitized list and perform desensitization;

[0047] Step S103) After desensitization is completed, the desensitization service 22 is called to feed back the operation data to the slicing service 21, wherein the operation data includes the number, the desensitization field table and the desensitization time;

[0048] Step S104) calling the slicing service 21 to dynamically adjust the slicing size according to all the running data to generate subsequent data units 12;

[0049] Step S105) After all data units 12 are desensitized, a merging service is called to combine all data units 12 according to the numbers to obtain desensitized big data.

[0050] The big data file 11 can be cut into data units 12 by dividing them according to size, number of rows, number of columns, etc. Exemplarily, 1MB or 5MB of content is cut from the big data file 11 as a data unit 12, and the big data file 11 is divided. In another exemplary embodiment, 5 lines are read from the big data file 11 as a data unit 12. The lines are divided by carriage returns, that is, "\r\n". In another exemplary embodiment, the characters "." are used for division, and the encoding of the characters "." is "3002". The division based on carriage returns may interrupt words, that is, it may interrupt private data and cause desensitization omissions, while the division based on the characters "." will not interrupt words. The encoding of the character "." may be different in different encodings. After the big data file 11 is identified and encoded, the character "." can be identified. The data unit 12 can be obtained from the big data file 11 by segmentation. In this embodiment, the segmentation of the big data file 11 is not completed in one go, but while the data units 12 are continuously segmented, the data units 12 that have been segmented are immediately read by multiple desensitizing services 22 and desensitized in parallel.

[0051] Specifically, the method of calling several desensitization services 22 in parallel to read the data unit 12 from the to-be-desensitized list and perform desensitization includes:

[0052] Registering a number of desensitization services 22 on one or more devices;

[0053] The to-be-desensitized list responds to the reading of the desensitization service 22 in a first-in-first-out manner;

[0054] The desensitizing service 22 matches the sensitive content contained in the data unit 12 according to a pre-configured sensitive matching expression, and replaces the sensitive content with a pre-set replacement character.

[0055] The desensitization service 22 matches the sensitive content in the data unit 12 according to the preset sensitive matching expression, and replaces the matched content. The desensitization service 22 first stores a desensitization table for indicating how to perform desensitization. The desensitization table records several desensitization rules, each of which includes a sequence number, a sensitive matching expression, and a replacement expression. The sensitive matching expression is a regular expression, and the replacement expression records what content will replace the criticized content with. For example, please refer to Table 1, which is the desensitization table used in this embodiment.

[0056] Table 1 Desensitization table

[0057]

[0058] The desensitization rule with sequence number 1 is used for desensitizing mobile phone numbers in China. It retains the country code (if any) and the first three digits of the phone number, and replaces the rest with "X". The desensitization rule with sequence number 2 is for 18-digit ID card numbers. This rule hides the date of birth and the sequence code part, only displays the first six digits of the address code and the last digit of the check code, and replaces the rest with asterisks (*). The desensitization rule with sequence number 3 is applicable to most standard bank card numbers. It will disclose the first and last four digits of the card number, and the middle part is all represented by asterisks (*). The desensitization rule with sequence number 4 completely hides the username part of the email address, retaining only the domain name and top-level domain to ensure that the email address cannot be easily identified. The desensitization rule with sequence number 5 replaces the last eight-bit segment with "XXX", thereby obscuring the specific host identification, but still retaining the network part information.

[0059] The desensitization service 22 also records the time used for desensitization, i.e., desensitization time, during the desensitization process of the data unit 12. At the same time, a desensitization field table is prepared for the sensitive content that is successfully matched.

[0060] Exemplarily, data unit 12 is "Mr. Zhang Weiqiang is a software engineer living in Beijing with more than ten years of programming experience. You can contact him by sending an email to zhangweiqiang@example.com to discuss potential cooperation opportunities. His mobile phone number is 13800000000, so you can communicate project details at any time. The landline number is 010-66666666, which is suitable for formal communication during office hours. If you have any questions about project cooperation, please contact Mr. Zhang through the above methods." Then, through the matching of the desensitization table, the desensitized field table obtained includes "Zhang Weiqiang", "zhangweiqiang@example.com", "13800000000", and "010-666666666". The corresponding desensitized content is "XXX", "XXXXXXXXXX@XXXXXXX.com", "1XXXXXXXXXX", and "XXX-XXXXXXXX". The desensitization module feeds back the number, desensitized field table, and desensitization time to the sharding service 21. The sharding service 21 adjusts the shard size according to the received operating data. Please refer to the attached. Figure 3, the ways in which the sharding service 21 adjusts the shard size include: keeping the shard size unchanged, increasing the shard size, and decreasing the shard size. After the sharding service 21 adjusts the shard size for a period of time, it will adjust the shard size again according to the received operating data. As a recommended method, the sharding service 21 reads the received operating data in a certain period to adjust the shard size, and the effective duration of the adjustment of the shard size is the duration of one cycle. Specifically, at the beginning of a cycle, the sharding service 21 first reads all the operating data received in the previous cycle, and after obtaining the result of the adjustment of the shard size, it performs the operation of cutting the big data of this cycle to obtain the data unit 12 according to the adjusted shard size. When the desensitization of all data units 12 is completed, the desensitized data units 12 are combined according to the numbers to obtain the desensitized big data file 11, that is, the desensitized file 13.

[0061] On the other hand, please see the attached Figure 4 The method of calling the slicing service 21 to extract a plurality of data units 12 from the big data according to the initial slicing size, assigning numbers and including them in the list to be desensitized includes:

[0062] Step S201) Divide the big data into a number of sub-packets according to a preset length, assign a packet number, and read the sub-packets into the memory in sequence;

[0063] Step S202) cutting a number of data units 12 from the sub-packet currently in the memory according to the initial fragment size, and assigning sequence numbers to the data units 12;

[0064] Step S203) Generate a number for the data unit 12 according to the packet number of the sub-packet and the sequence number of the data unit 12;

[0065] Step S204) The data unit 12 is included in the to-be-desensitized list according to the number.

[0066] In order to improve the efficiency of cutting data units 12 from big data, the big data is read into the memory and cut. Therefore, the big data file 11 is first cut once to form multiple sub-packets, and the sub-packets have packet numbers. The sub-packets are read into the memory in sequence according to the packet numbers, and the sub-packets are cut to obtain data units 12.

[0067] Since the size of the data unit 12 changes with the change of the fragment size, the size of the last data unit 12 obtained after the sub-packet segmentation may not meet the fragment size. You can either read the next sub-packet immediately or read the next sub-packet in advance, and read from the next sub-packet to make the size of the data unit 12 meet the fragment size. You can also operate in a way that the size of the last data unit 12 of the sub-packet does not meet the fragment size.

[0068] On the other hand, the method for the slicing service 21 to dynamically adjust the slicing size according to all the running data to generate the subsequent data unit 12 includes:

[0069] The slicing service 21 groups all the operation data according to a preset time period;

[0070] Count the intra-group similarity of the desensitized field table contained in the running data in each group;

[0071] When the similarity within the group in the previous cycle is higher than a preset first threshold, the slice size is increased, and when the similarity within the group is lower than a preset second threshold, the slice size is adjusted to the initial slice size.

[0072] The desensitized field table can be regarded as a vector, and the desensitized field table is converted into a vector. The intra-group similarity of the desensitized field table can be directly represented by the vector distance of the desensitized field table in the group. The desensitized field table can be converted into a vector by sorting the desensitized fields in alphabetical order of the pinyin of the first word to form a vector, or it can be implemented by using the text vectorization technology disclosed in the art. For example, when using the bag-of-words model, it is not necessary to sort the sensitive content. Pre-trained language models such as BERT, RoBERTa, DistilBERT, etc. can also be used for text vectorization. Similarly, the desensitized field table can also be vectorized using Doc2Vec.

[0073] On the other hand, in another embodiment, the method for the slicing service 21 to dynamically adjust the slicing size according to all running data to generate subsequent data units 12 includes:

[0074] Step S301) The slicing service 21 groups all the operation data according to a preset time period;

[0075] Step S302) Divide the previous cycle into multiple subgroups according to the initial grouping size;

[0076] Step S303) Calculate the average similarity of the desensitized field table in the subgroup, which is recorded as the intra-group similarity;

[0077] Step S304) Calculate the average of the intra-group similarities of all sub-groups, which is recorded as the extra-group similarity;

[0078] Step S305) adjusting the group size until a group size that maximizes the out-of-group similarity is obtained, which is recorded as the optimal group size;

[0079] Step S306) Adjust the fragment size according to the optimal packet size.

[0080] Exemplarily, in step S301, the preset duration period can be 1 to 3 minutes or other durations. When a cycle starts, all the desensitized field tables in the previous cycle are read first. The previous cycle is divided into multiple subgroups again according to time, and each subgroup includes multiple desensitized field tables. Then the similarity between the desensitized field tables of the subgroups is calculated in pairs, and finally the mean of all similarities is calculated, that is, the similarity within the group. Then the mean of all similarities within the group is calculated and recorded as the similarity outside the group. When the size of the subgroup division makes the similarity outside the group the largest, the group size is recorded as the optimal grouping. Adjust the fragment size according to the optimal grouping, that is, adjust the fragment size to match the optimal grouping. Exemplarily, 12 operating data were received in the previous cycle, that is, a total of 12 desensitized field tables. When divided according to time, every 4 operating data are divided into 3 subgroups. There will be 4 operating data in each subgroup, that is, there will be 4 desensitized field tables. Calculating the similarity between the four desensitized field tables in pairs will require six calculations to obtain six similarities. The average of these six similarities is the intra-group similarity. The three subgroups have a total of three intra-group similarities. The sum of these three intra-group similarities and then dividing by 3 will give the extra-group similarity.

[0081] The 12 operation data in the previous cycle are divided into subgroups again, and illustratively, divided into 4 subgroups, each of which includes 3 operation data. After the same calculation, the out-of-group similarity when divided into 4 subgroups is obtained.

[0082] The 12 operating data in the previous cycle are again divided into subgroups, illustratively, into 6 subgroups, that is, each subgroup includes 2 operating data, and finally the out-of-group similarity when divided into 6 subgroups can be obtained.

[0083] Adjusting the fragment size according to the optimal packet size specifically includes:

[0084] When the similarity outside the group corresponding to the optimal group size is higher than the preset threshold, the fragment size is adjusted to be equal to the optimal group size. When the similarity outside the group corresponding to the optimal group size is not higher than the preset threshold, the fragment size is adjusted to the initial fragment size.

[0085] Exemplarily, when the out-of-group similarity when divided into 6 subgroups is the highest and higher than the preset threshold, the shard size of the current cycle is adjusted to twice the size of the previous cycle. When the out-of-group similarity when divided into 6 subgroups is the highest, but not higher than the preset threshold, the shard size of the current cycle is adjusted to the initial shard size. When the desensitized field similarity of the running data within a subgroup is high, it means that the same sensitive content continues to appear, and the length of the relatively close desensitized field set has a certain periodicity, which is due to the principle of locality of the text content. Specifically, the law that the same or similar content will reappear in a period of time in the future is usually called the "Principle of Locality". By predicting and storing data patterns that may recur, the efficiency of desensitizing sensitive data can be improved. Exemplarily, when a content mentions a mobile phone number, the mobile phone number has a greater probability of appearing again in a certain range of subsequent content. In this embodiment, this range is characterized by external similarity to find the best range, which can contain more repeated content, so that more repeated sensitive content can be replaced in one desensitization replacement operation, thereby improving the desensitization efficiency.

[0086] On the other hand, in another embodiment, the method for the slicing service 21 to dynamically adjust the slicing size according to all running data to generate subsequent data units 12 includes:

[0087] Step S401) The slicing service 21 groups all the operation data according to a preset time period;

[0088] Step S402) Divide the previous cycle into multiple subgroups according to the initial group size;

[0089] Step S403) Calculate the average similarity of the desensitized field table in the subgroup, which is recorded as the intra-group similarity;

[0090] Step S404) Calculate the average of the intra-group similarities of all sub-groups, which is recorded as the extra-group similarity;

[0091] Step S405) Calculate the average similarity of the desensitization time corresponding to all the running data in the subgroup, and record it as the time proximity within the group;

[0092] Step S406) Calculate the average of the intra-group time proximity of all sub-groups, and record it as the extra-group time proximity;

[0093] Step S407) adjusting the group size until a group size that maximizes the weighted sum of the out-group similarity and the out-group time proximity is obtained, which is recorded as the optimal group size;

[0094] Step S408) adjusting the fragment size according to the optimal packet size.

[0095] Correspondingly, after calculating the external similarity, the desensitization time of the operating data contained in each subgroup is calculated pairwise for similarity, and finally the average of all similarities is calculated to obtain the intra-group time proximity. The average of the average time proximity of all subgroups is recorded as the out-of-group time proximity. The desensitization time reflects the size of the desensitized field table and the time of replacement from the perspective of time. When the desensitization time is close, it means that the total number of desensitized fields is close. By combining the weighted mean of the external similarity and the external time proximity to divide the optimal shard size, the efficiency of desensitization can be further improved.

[0096] The method provided in this application is applied to Figure 7 The system architecture shown, Figure 7 is a schematic diagram of the system architecture in the embodiment of the present application, such as Figure 7 As shown, the system architecture includes a server 20 and a terminal device 30, and an interactive interface 31 is provided and deployed on the terminal device 30, wherein the interactive interface 31 can be run on the terminal device 30 in the form of a browser, or in the form of an independent application (application, APP), etc. The specific presentation form of the client is not limited here, and the interactive interface 31 is as shown in the attached Figure 8 As shown. The server 20 involved in this application can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal device 30 can be a smart phone, a tablet computer, a laptop, a PDA, a personal computer, a smart speaker, a smart TV, a smart watch, a car device, a wearable device, etc., but is not limited to this. The terminal device 30 and the server 20 can be directly or indirectly connected via wired or wireless communication, and this application does not limit this. The number of servers 20 and terminal devices 30 is also not limited. The solution provided in this application can be completed independently by the terminal device 30, or by the server 20, or by the terminal device 30 and the server 20 in cooperation, and this application does not make specific limitations.

[0097] On the other hand, this specification provides a large data parallel desensitization system, please refer to the attached Fig. 9 ,include:

[0098] The reading module 100 reads the big data file 11, calls the slicing service 21 to cut a number of data units 12 from the big data according to the initial slicing size, assigns numbers and puts them into the list to be desensitized;

[0099] The desensitization module 200 calls several desensitization services 22 in parallel to read the data unit 12 from the to-be-desensitized list and perform desensitization;

[0100] Feedback module 300, when desensitization is completed, calls the desensitization service 22 to feed back the operation data to the slicing service 21, the operation data including the number, desensitization field table and desensitization time;

[0101] An adjustment module 400, calling the slicing service 21 to dynamically adjust the slicing size according to all running data to generate subsequent data units 12;

[0102] The merging module 500, after all the data units 12 are desensitized, calls the merging service according to the numbering to combine all the data units 12 to obtain the desensitized big data.

[0103] See also Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of this specification is shown.

[0104] like Fig.10 As shown, the electronic device 1100 may include: at least one processor 1101, at least one network interface 1104, a user interface 1103, a memory 1105 and at least one communication bus 1102. Among them, the communication bus 1102 can be used to realize the connection and communication of the above-mentioned components. Among them, the user interface 1103 may include a button, and the optional user interface may also include a standard wired interface and a wireless interface. Among them, the network interface 1104 may include but is not limited to a Bluetooth module, an NFC module, a Wi-Fi module, etc. Among them, the processor 1101 may include one or more processing cores. The processor 1101 uses various interfaces and lines to connect various parts in the entire electronic device 1100, and executes various functions and processes data of the routing device 1100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 1105, and calling data stored in the memory 1105. Optionally, the processor 1101 can be implemented in at least one hardware form of DSP, FPGA, and PLA. The processor 1101 may integrate one or a combination of CPU, GPU and modem, etc. Among them, the CPU mainly processes the operating system, user interface and application programs, etc.; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communication.

[0105] It is understandable that the above-mentioned modem may not be integrated into the processor 1101, but may be implemented by a separate chip.

[0106] Among them, the memory 1105 may include RAM or ROM. Optionally, the memory 1105 includes a non-transitory computer-readable medium. The memory 1105 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 1105 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 1105 may also be at least one storage device located away from the aforementioned processor 1101. The memory 1105 as a computer storage medium may include an operating system, a network communication module, a user interface module and an application. The processor 1101 may be used to call the application stored in the memory 1105 and execute the methods in the above-mentioned multiple embodiments.

[0107] The embodiments of this specification also provide a computer-readable storage medium, which stores instructions, and when the instructions are executed on a computer or a processor, the computer or the processor executes the multiple steps in the above embodiments. If the components of the above electronic device are implemented in the form of software functional units and sold or used as independent products, they can be stored in the computer-readable storage medium.

[0108] The embodiments of this specification also provide a computer program product, including a computer program, which implements multiple steps in the above embodiments when executed by a processor.

[0109] In the absence of conflict, the technical features in this embodiment and implementation scheme can be combined arbitrarily.

[0110] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes a plurality of computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of this specification is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes multiple available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state drive (SSD)).

[0111] When implemented by hardware or firmware, the aforementioned method flow is programmed into the hardware circuit to obtain the corresponding hardware circuit structure and realize the corresponding function. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit, and its logic function is determined by the user programming the device. The designer programs by himself to "integrate" a digital system on a PLD, without asking a chip manufacturer to design and make a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when writing program development, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL), and HDL is not just one, but many. Those skilled in the art should also be aware that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit to easily obtain the hardware circuit that implements the logic method flow.

[0112] The embodiments described above are merely preferred embodiments of this specification and are not intended to limit the scope of this specification. Without departing from the design spirit of this specification, various modifications and improvements made to the technical solutions of this specification by ordinary technicians in this field should fall within the scope of protection determined by the claims of this specification.

Claims

1. A big data parallel desensitization method, characterized in that: Includes steps: Read the big data file, call the sharding service to cut a number of data units from the big data according to the initial shard size, assign numbers and include them in the list to be desensitized; Calling several desensitization services in parallel to read data units from the to-be-desensitized list and perform desensitization; When desensitization is completed, the desensitization service is called to feed back the operation data to the slicing service, and the operation data includes the number, the desensitization field table and the desensitization time; Calling the sharding service to dynamically adjust the sharding size according to all running data to generate subsequent data units; After all data units are desensitized, the merging service is called to combine all data units according to the number to obtain the desensitized big data; The method for the slicing service to dynamically adjust the slicing size according to all running data to generate subsequent data units includes: The slicing service groups all running data according to a preset time period; Count the intra-group similarity of the desensitized field table contained in the running data in each group; When the similarity within the group in the previous cycle is higher than a preset first threshold, the slice size is increased, and when the similarity within the group is lower than a preset second threshold, the slice size is adjusted to the initial slice size.

2. A big data parallel desensitization method according to claim 1, characterized in that: The method of calling the sharding service to extract a number of data units from the big data according to the initial sharding size, assigning numbers and including them in the list to be desensitized includes: Divide the big data into a number of sub-packets according to a preset length, assign packet numbers, and read the sub-packets into memory in sequence; Cutting a number of data units from the sub-packet currently in the memory according to the initial fragment size, and assigning sequence numbers to the data units; Generate a number for the data unit according to the packet number of the sub-packet and the sequence number of the data unit; The data units are included in the list to be desensitized according to their numbers.

3. A big data parallel desensitization method according to claim 1, characterized in that: The method of calling several desensitization services in parallel to read data units from the to-be-desensitized list and perform desensitization includes: Register several desensitization services on one or more devices; The to-be-desensitized list responds to the reading of the desensitization service in a first-in-first-out manner; The desensitization service matches the sensitive content contained in the data unit according to a pre-configured sensitive matching expression, and replaces the sensitive content with a pre-set replacement character.

4. A big data parallel desensitization method, characterized in that: Includes steps: Read the big data file, call the sharding service to cut a number of data units from the big data according to the initial shard size, assign numbers and include them in the list to be desensitized; Calling several desensitization services in parallel to read data units from the to-be-desensitized list and perform desensitization; When desensitization is completed, the desensitization service is called to feed back the operation data to the slicing service, and the operation data includes the number, the desensitization field table and the desensitization time; Calling the sharding service to dynamically adjust the sharding size according to all running data to generate subsequent data units; After all data units are desensitized, the merging service is called to combine all data units according to the number to obtain the desensitized big data; The method for the slicing service to dynamically adjust the slicing size according to all running data to generate subsequent data units includes: The slicing service groups all running data according to a preset time period; Divide the previous period into multiple subgroups according to the initial group size; Calculate the average similarity of the desensitized field table within the subgroup, and record it as the intra-group similarity; Calculate the average of the intra-group similarities of all subgroups and record it as the extra-group similarity; Adjust the group size until the group size that maximizes the out-of-group similarity is obtained, which is recorded as the optimal group size; The fragment size is adjusted according to the optimal packet size.

5. A big data parallel desensitization method, characterized in that: Includes steps: Read the big data file, call the sharding service to cut a number of data units from the big data according to the initial shard size, assign numbers and include them in the list to be desensitized; Calling several desensitization services in parallel to read data units from the to-be-desensitized list and perform desensitization; When desensitization is completed, the desensitization service is called to feed back the operation data to the slicing service, and the operation data includes the number, the desensitization field table and the desensitization time; Calling the sharding service to dynamically adjust the sharding size according to all running data to generate subsequent data units; After all data units are desensitized, the merging service is called to combine all data units according to the number to obtain the desensitized big data; The method for the slicing service to dynamically adjust the slicing size according to all running data to generate subsequent data units includes: The slicing service groups all running data according to a preset time period; Divide the previous period into multiple subgroups according to the initial group size; Calculate the average similarity of the desensitized field table within the subgroup, and record it as the intra-group similarity; Calculate the average of the intra-group similarities of all subgroups and record it as the extra-group similarity; Calculate the average similarity of the desensitization time corresponding to all running data in the subgroup, and record it as the time proximity within the group; Calculate the average of the intra-group time proximity of all subgroups and record it as the extra-group time proximity; Adjust the group size until a group size that maximizes the weighted sum of the out-group similarity and the out-group time proximity is obtained, which is recorded as the optimal group size; The fragment size is adjusted according to the optimal packet size.

6. A big data parallel desensitization system, characterized in that: include: The reading module reads the big data file, calls the sharding service to cut a number of data units from the big data according to the initial sharding size, assigns numbers and puts them into the list to be desensitized; A desensitization module, calling several desensitization services in parallel to read data units from the to-be-desensitized list and perform desensitization; A feedback module, when desensitization is completed, calls the desensitization service to feed back the operation data to the slicing service, the operation data including the number, the desensitization field table and the desensitization time; An adjustment module, calling the sharding service to dynamically adjust the sharding size according to all running data to generate subsequent data units; The merging module, after all data units are desensitized, calls the merging service according to the number to combine all data units to obtain the desensitized big data; The method for the slicing service to dynamically adjust the slicing size according to all running data to generate subsequent data units includes: The slicing service groups all running data according to a preset time period; Count the intra-group similarity of the desensitized field table contained in the running data in each group; When the similarity within the group in the previous cycle is higher than a preset first threshold, the slice size is increased, and when the similarity within the group is lower than a preset second threshold, the slice size is adjusted to the initial slice size.

7. An electronic device, characterized in that: including a processor and a memory; The processor is connected to the memory; The memory is used to store executable program code; The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images