A data cleaning method for intelligent big data platform

Abstract

The invention discloses a data cleaning method for an intelligent big data platform, relates to the technical field of the Internet and big data, and solves the problems of diverse formats, complex structures and slow real-time processing speed of information data; the data cleaning method comprises obtaining massive information data fed back by a mobile terminal; dividing the massive information data into a plurality of sub-data blocks according to the terminal type and the data collection time; classifying and grading the information data in each sub-data block according to the event business type; format checking and data cleaning the information data after data classification according to the format standards of each business data; extracting key information of the information data and verifying the relevance of the information data; merging all sub-data blocks and information data with relevance; sorting the merged information data according to the urgency of the event; completing the identity verification of the mobile terminal through a chaotic check code, and automatically correcting the content of the information data through a content cleaning model.

Description

Technical Field

[0001] The present invention relates to the field of Internet and big data technology, and in particular to a data cleaning method for an intelligent big data platform. Background Art

[0002] With the continuous upgrading of information collection technology and the continuous expansion of big data application scenarios, data has become an important asset in modern society, but data quality issues have also become one of the important factors restricting big data applications. Data quality issues include data accuracy, completeness, consistency, uniqueness, reliability and other aspects. Data cleaning, as one of the preprocessing and basic work of big data, mainly deals with data quality issues to improve data quality and credibility.

[0003] As a comprehensive data management and analysis platform, the data cleaning method of the intelligent big data platform needs to take into account the particularity and complexity of the smart city big data field.

[0004] Patent No. CN2020101547162 discloses a multi-source heterogeneous ecological environment big data processing system and method based on a data lake, which includes: an ecological environment data acquisition layer, an ecological environment data cleaning layer, an ecological environment data storage layer, an ecological environment data processing layer, and an ecological environment data management layer. The ecological environment data acquisition layer is used to collect the original data of the ecological environment; the ecological environment data cleaning layer is used to preprocess and standardize the data acquired by the ecological environment data acquisition layer; the ecological environment data storage layer is used to classify and hierarchically store the data transmitted by the ecological environment data cleaning layer; the ecological environment data processing layer is used to integrate the processing of stream and batch ecological environment data; the ecological environment data management layer is used to monitor the collection, cleaning, storage and processing process of ecological environment data. The above invention can improve the access and analysis efficiency of environmental data and significantly reduce storage costs.

[0005] Patent No. CN2022100703463 discloses a method and system for constructing an industrial park knowledge graph that integrates multi-source heterogeneous data. The method integrates digital economy knowledge and digital economy industrial park data from different sources into structured knowledge centered on the digital economy industrial park through data cleaning; annotates entity relationships in structured knowledge according to annotation strategies; constructs and trains a digital economy industrial park knowledge fusion model based on a convolutional neural network to perform preliminary extraction of entity relationships in structured knowledge; retrieves repeated entity relationship triplets and extracts entity relationships, thereby constructing entity-relationship-entity triplets of the digital economy industrial park knowledge graph and performing knowledge fusion. The above invention realizes the rapid construction of a large-scale digital economy industrial park knowledge graph, helps to improve the efficiency of information analysis and processing, facilitates the search for implicit influences and conduction effects between elements of the digital economy industrial park, and forms large-scale industrial analysis results.

[0006] Although the above patents all provide methods for cleaning data in various business aspects of smart cities, when dealing with massive data from multiple sources, they are unable to timely guarantee the security of massive information data with diverse sources, complex attributes and uneven quality, and the information data cleaning process is complicated and the processing speed is too slow. Summary of the invention

[0007] The purpose of the present invention is to provide a data cleaning method for an intelligent big data platform, which can complete the identity verification of mobile terminals through a special data cleaning process using a chaotic check code, and automatically correct the content of information data using a content cleaning model. At the same time, the massive information data is divided twice according to the data collection time and the type of mobile terminal to form a number of sub-data blocks; so as to improve the processing speed of massive information data, ensure the quality of information data, and simplify the cleaning process of massive information data.

[0008] The present invention utilizes the following technical solutions:

[0009] A data cleaning method for an intelligent big data platform comprises the following steps:

[0010] S1: The big data platform uses the data collection module to obtain massive information data fed back by mobile terminals;

[0011] S2: The data segmentation module divides the collected massive information data into several sub-data blocks according to the terminal type and data collection time;

[0012] S3: According to the event business type, the data classification module classifies and grades the information data in each sub-data block;

[0013] S4: According to the data format standards of each business, the data verification module will perform format verification and data cleaning on the information data that has completed data classification;

[0014] S5: The data association module extracts key information from the verified and cleaned information data, and verifies the association of the information data based on the key information to adjust the information data;

[0015] S6: The data aggregation module merges all sub-data blocks and related information data;

[0016] S7: According to the urgency of the event, the information sorting module sorts the merged information data.

[0017] Preferably, in step S1, the workflow of the big data platform using the data acquisition module to obtain the mobile terminal information data is as follows:

[0018] A: The mobile terminal sends a POST request and / or a PUT request to the big data platform to establish a verification channel;

[0019] Mobile terminals include but are not limited to mobile phones, web pages, tablet computers and laptops;

[0020] B: The big data platform agrees to the POST request and / or PUT request, establishes a connection with the verification channel, and sends the chaos verification code to the mobile terminal respectively:

[0021]

[0022] Wherein, C represents the chaotic check code, i represents the random number sequence, x represents the number of random digits, tent() represents the discrete chaotic function, rand() represents the random generation function, j represents the mapping number sequence, n represents the total mapping number, and Logs() represents the mapping modulation function;

[0023] C: The mobile terminal receives the chaotic check code to complete the identity verification and establishes a parallel transmission channel with the data acquisition module:

[0024]

[0025] Among them, GrT() represents the parallel transmission channel group, k represents the chaotic check code number, s represents the number of chaotic check code bits, C k represents the k-th chaotic check code, Offer() represents the channel creation function, Track() represents the audio channel function, Source() represents the video channel function, Text() represents the text channel function, and Image() represents the image channel function;

[0026] D: The big data platform divides the mobile terminals into several batches according to the completion time of identity verification, and sorts the mobile terminals in the same batch according to the construction time of the parallel transmission channel;

[0027] E: The mobile terminal sends the information data to the parallel transmission channel according to the batch and sorting results;

[0028] Information data includes but is not limited to basic identity information, matter classification information, basic directory information, implementation list information, application registration information, application materials information, intermediary service information, online evaluation information, consultation information, complaint information, case acceptance information, material directory information, handling process information, handling progress information, special procedure information, completion information, basic supervision information, supervision process information, supervision result information, electronic seal information, electronic certificate information, payment information, logistics information, user experience monitoring information, government service evaluation information, and service situation analysis information;

[0029] F: The data acquisition module uses parallel transmission channels and distributed acceleration algorithms to receive information data from the mobile terminal and transmit it to the data segmentation module for processing.

[0030] Preferably, in step S2, the process of the data segmentation module dividing the massive information data into several sub-data blocks is as follows:

[0031] First, the received massive information data is simulated and divided according to the type of mobile terminal to construct simulated data blocks, and the capacity and number of simulated data blocks are determined:

[0032]

[0033] in, represents the simulated data block of the zth mobile terminal, vr represents the virtual identifier, impart() represents the data segmentation function, FD represents the massive information data, ⊙ represents the Hadamard outer product, MT z represents the zth type of mobile terminal;

[0034] Then, according to the integrity of the information data in the simulation data block, the capacity and quantity of the simulation data block are adjusted;

[0035] Subsequently, the information data in each simulation data block is labeled according to the data collection time, and the labeled information data is clustered according to a preset time range;

[0036]

[0037] Among them, Km α represents the αth clustering data block that meets the time range, α represents the total number of time ranges that meet the time range, Indicates the data collection time of extracting all information data in the simulation data block of the zth mobile terminal; the time range is year, month, day and hour;

[0038] Finally, the clustered information data is separated to complete the internal segmentation of the simulated data block, and several sub-data blocks are formed and assigned identification numbers, and the capacity and number of each sub-data block are calculated at the same time:

[0039]

[0040] in, represents the βth sub-data block, represents the Hadamard inner product, and rl represents the identification number.

[0041] Preferably, in step S3, the process of the data classification module classifying the internal information data of each sub-data block is as follows:

[0042] S31: Acquire characteristic attributes of internal information data of the sub-data block;

[0043] Characteristic attributes include, but are not limited to, dependency, extensibility, transferability, storage, concentration, sharing, predictability, validity, invalidity, and / or processing;

[0044] S32: Classify information data with the same or similar characteristic attributes according to event business types;

[0045] Event business types include but are not limited to identity authentication, event management, service application, service feedback, service interaction, event acceptance, event handling, event completion, electronic supervision, basic support and data services;

[0046] S33: classifying the classified information data according to the sensitive dimensions of the information data;

[0047] Sensitive dimensions include, but are not limited to, personal privacy information, business secrets, national security information, healthcare information, education information, and social media information;

[0048] S34: Perform data verification on the classified information data.

[0049] Preferably, in step S4, the data verification module includes a format verification unit and a data cleaning unit; the format verification unit uses a format error correction algorithm to verify and correct the format of information data at all levels according to various business data format standards; business data format standards include but are not limited to naming rules, naming conventions, data types, date formats, metadata standards, data dictionaries, interface standards, integration specifications, permission specifications and access policies; the data cleaning unit uses a preset content cleaning model to perform information content error correction on the information data that has completed format verification and correction.

[0050] Preferably, the sensitivity dimension level includes 5 levels, from high to low, 5: top secret, 4: confidential, 3: secret, 2: internal disclosure and 1: external disclosure, and the higher the level, the larger the number assigned;

[0051] The content cleaning model first encrypts the information data after format correction according to the sensitive dimension level to obtain the encrypted feature values ​​of the information data at each level:

[0052]

[0053] Where P represents the encrypted feature value, y represents the sensitive dimension level, and y≥3; γ represents the identification number sequence number, and δ represents the total number of digits of the identification number;

[0054] Then, the obtained encrypted feature values ​​at all levels and information data with sensitivity dimension levels of 1 to 2 are input into the feature extraction layer, and feature extraction of different depths is performed according to the sensitivity dimension level to obtain a reduced dimension feature matrix; the feature extraction layer includes 4 3X3 convolutional layers, 2 3X3 maximum pooling layers and 3 normalization layers; the number of feature extractions of encrypted feature values ​​at different sensitivity dimension levels is the square of the sensitivity dimension level, and the feature extraction layer is trained using a multidimensional scaling algorithm;

[0055] Then, the obtained reduced dimension feature matrix is ​​loaded into the learning and training layer for multiple iterative training to obtain the feature weight matrix of the information data; the learning and training layer includes 2 1X1 convolutional layers, 1 3X3 upsampling layer, 1 3X3 downsampling layer, 3 InceptionV3 blocks, 4 residual blocks, 3 average pooling layers and RELU activation function; for the eigenvalues ​​of different levels in the reduced dimension feature matrix, the connection methods of the 3 InceptionV3 blocks and 4 residual blocks of the learning and training layer are as follows: if the sensitive dimension level belongs to 1 to 2, the 4 residual blocks are connected in series and then input into 3 parallel InceptionV3 blocks; if the sensitive dimension level belongs to 3 to 4, the 3 InceptionV3 blocks are respectively connected in parallel with the 3 residual blocks and then input into the 4th residual block; if the sensitive dimension level is 5, the 3 residual blocks respectively use 1 InceptionV3 block as the second branch and are connected in series with the 4th residual block; thereby reducing the sensitive dimension of the feature weight matrix;

[0056] Finally, the acquired feature weight matrix is ​​sequentially input into two average pooling layers, two fully connected layers and the SoftMax activation function to realize the abnormal judgment of the information data, and then different exception processing algorithms are dispatched to correct the abnormal content of the information data; the exception processing algorithms include but are not limited to data deduplication algorithm, data filling algorithm, data conversion algorithm and data simplification algorithm.

[0057] Preferably, in step S5, the process of the data association module performing association verification and adjustment on the information data of each sub-data block is as follows:

[0058] S51: extract key information of the information data in the sub-data block; the key information includes event address, event time and event subject;

[0059] S52: De-duplication and relevance verification of information data according to the acquired key information and the preset time range;

[0060] S53: Construct the information data that has completed the relevance verification into a relevance information map, and combine the identification number and key information to form a unique identification of each sub-data block.

[0061] Preferably, in step S6, the process of the data aggregation module merging each sub-data block and the internal information data is as follows:

[0062] S61: sorting all sub-data blocks according to the unique identifier of each sub-data block, and parsing the unique identifier to obtain all event information of each sub-data block;

[0063] S62: Calculate the correlation between each event information in different sub-data blocks;

[0064] S63: Virtually linking different sub-data blocks whose correlation exceeds a set aggregation threshold;

[0065] S64: If all event information in the current sub-data block is virtually linked by the additional sub-data block, the information data of the current sub-data block is segmented to complete data aggregation with other information data; otherwise, go to step S65;

[0066] S65: If the individual event information in the current sub-data block is not virtually linked to the additional sub-data block, the virtually linked part of the current sub-data block is split, and the non-virtual linked part is reassigned with an identification number, thereby completing data aggregation with other information data;

[0067] S66: Merge the information data that has completed data aggregation to form a complete event information chain.

[0068] Preferably, in step S7, the data sorting module includes an event characterization unit and an event sorting unit; the event characterization unit uses an expert prior model to determine and refine the event nature of the complete event information chain; the event nature includes natural disasters, accident disasters, public health events and social security events; the event sorting unit uses a parallel cardinality algorithm to sort the characterized complete event information chain according to the urgency of the event to construct an event processing coding table; the event urgency is divided into three levels: first-level emergency, second-level emergency and third-level emergency according to the degree of impact of the emergency on society.

[0069] Preferably, the data sorting module also includes an event dispatching unit; the event dispatching unit distributes the event information in the event processing coding table according to similar historical events, the event responsible departments at different time nodes and the event processing progress, and creates an event processing schedule to complete event processing tracking.

[0070] The present invention completes the identity verification of the mobile terminal through the chaotic check code, automatically corrects the content of the information data through the content cleaning model, and divides the massive information data into secondary parts according to the data collection time and the type of the mobile terminal to form a plurality of sub-data blocks; the processing speed of the massive information data is improved, the quality of the information data is guaranteed, and the cleaning process of the massive information data is simplified. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0072] Figure 1 This is a principle block diagram of the data cleaning method for the intelligent big data platform. DETAILED DESCRIPTION

[0073] The present invention is described in detail below with reference to the accompanying drawings and embodiments:

[0074] like Figure 1 As shown, the data cleaning method of the intelligent big data platform described in the present invention includes the following steps:

[0075] S1: The big data platform uses the data collection module to obtain massive information data fed back by mobile terminals;

[0076] In the present invention, the workflow of the big data platform using the data acquisition module to obtain mobile terminal information data is as follows:

[0077] A: The mobile terminal sends a POST request and / or a PUT request to the big data platform to establish a verification channel;

[0078] Mobile terminals include but are not limited to mobile phones, web pages, tablet computers and laptops;

[0079] B: The big data platform agrees to the POST request and / or PUT request, establishes a connection with the verification channel, and sends the chaos verification code to the mobile terminal respectively:

[0080]

[0081] Wherein, C represents the chaotic check code, i represents the random number sequence, x represents the number of random digits, tent() represents the discrete chaotic function, rand() represents the random generation function, j represents the mapping number sequence, n represents the total mapping number, and Logs() represents the mapping modulation function;

[0082] C: The mobile terminal receives the chaotic check code to complete the identity verification and establishes a parallel transmission channel with the data acquisition module:

[0083]

[0084] Among them, GrT() represents the parallel transmission channel group, k represents the chaotic check code number, s represents the number of chaotic check code bits, C k represents the k-th chaotic check code, Offer() represents the channel creation function, Track() represents the audio channel function, Source() represents the video channel function, Text() represents the text channel function, and Image() represents the image channel function;

[0085] D: The big data platform divides the mobile terminals into several batches according to the completion time of identity verification, and sorts the mobile terminals in the same batch according to the construction time of the parallel transmission channel;

[0086] E: The mobile terminal sends the information data to the parallel transmission channel according to the batch and sorting results;

[0087] Information data includes but is not limited to basic identity information, matter classification information, basic directory information, implementation list information, application registration information, application materials information, intermediary service information, online evaluation information, consultation information, complaint information, case acceptance information, material directory information, handling process information, handling progress information, special procedure information, completion information, basic supervision information, supervision process information, supervision result information, electronic seal information, electronic certificate information, payment information, logistics information, user experience monitoring information, government service evaluation information, and service situation analysis information;

[0088] F: The data acquisition module uses parallel transmission channels and distributed acceleration algorithms to receive information data from mobile terminals and transmit it to the data segmentation module for processing;

[0089] S2: The data segmentation module divides the collected massive information data into several sub-data blocks according to the terminal type and data collection time;

[0090] In the present invention, the process of the data block module dividing the massive information data into several sub-data blocks is as follows:

[0091] First, the received massive information data is simulated and divided according to the type of mobile terminal to construct simulated data blocks, and the capacity and number of simulated data blocks are determined:

[0092]

[0093] in, represents the simulated data block of the zth mobile terminal, vr represents the virtual identifier, impart() represents the data segmentation function, FD represents the massive information data, ⊙ represents the Hadamard outer product, MT z represents the zth type of mobile terminal;

[0094] Then, according to the integrity of the information data in the simulation data block, the capacity and quantity of the simulation data block are adjusted;

[0095] In the present invention, integrity refers to whether all necessary components of a set of data exist and can be used or accessed normally.

[0096] Subsequently, the information data in each simulation data block is labeled according to the data collection time, and the labeled information data is clustered according to a preset time range;

[0097]

[0098] Among them, Km α represents the αth clustering data block that meets the time range, α represents the total number of time ranges that meet the time range, Indicates the data collection time of extracting all information data in the simulation data block of the zth mobile terminal; the time range is year, month, day and hour;

[0099] Finally, the clustered information data is separated to complete the internal segmentation of the simulated data block, and several sub-data blocks are formed and assigned identification numbers, and the capacity and number of each sub-data block are calculated at the same time:

[0100]

[0101] in, represents the βth sub-data block, represents the Hadamard inner product, rl represents the identification number;

[0102] S3: According to the event business type, the data classification module classifies and grades the information data in each sub-data block;

[0103] In the present invention, the process of the data classification module classifying the internal information data of each sub-data block is as follows:

[0104] S31: Acquire characteristic attributes of internal information data of the sub-data block;

[0105] Characteristic attributes include, but are not limited to, dependency, extensibility, transferability, storage, concentration, sharing, predictability, validity, invalidity, and / or processing;

[0106] In this embodiment, dependency: Matter is a concrete, real resource; while information data is an abstract, intangible resource; information data must be attached to a material carrier, and can only be transmitted if the carrier has a certain amount of energy;

[0107] Expandability: Material and energy resources will decrease as long as they are used; however, information data will continue to expand and regenerate during use and will never be exhausted; Transmission: Information data can be transmitted in many ways, such as oral language, body language, handwritten text, printed text, and telecommunications signals;

[0108] Storage: Information data can be stored for use at another time or by others; there are many ways to store information data, such as human brain, computer memory, writing, printing, microfilming, video recording, photography, and audio recording;

[0109] Condensation: Information concentration is the process of people summarizing and synthesizing a large amount of information. For example, summaries, reports, proposals, news reports, experience, knowledge, etc. are all extracted after collecting a large amount of information data. Microfilm and CD-ROM are modern technologies that condense and store information.

[0110] Sharability: Information data is different from material resources; it can be transferred and shared by everyone; the more scientific and socially standardized the information data is, the more shareable it is;

[0111] Predictive; that is, to deduce the future information data form through current information data; information data has an advanced reflection of reality and reflects the development trend of things; this is the value of information data for "judgment" and even "decision-making";

[0112] Validity and invalidity; information is valid if it meets the needs of the recipient, otherwise it is invalid; it is valid if it is needed at this time, but invalid if it is not needed at that time; it is valid for this person, but may be invalid for others;

[0113] Processability: If information data is analyzed and processed, new information data will often be generated, which will increase the value of the information data;

[0114] S32: Classify information data with the same or similar characteristic attributes according to event business types;

[0115] Event business types include but are not limited to identity authentication, event management, service application, service feedback, service interaction, event acceptance, event handling, event completion, electronic supervision, basic support and data services;

[0116] S33: classifying the classified information data according to the sensitive dimensions of the information data;

[0117] Sensitive dimensions include, but are not limited to, personal privacy information, business secrets, national security information, healthcare information, education information, and social media information;

[0118] In this embodiment, personal information refers to various information recorded electronically or otherwise that can identify a specific natural person alone or in combination with other information or reflect the activities of a specific natural person, including name, ID number, communication contact information, address, account password, property status, whereabouts, etc.;

[0119] Trade secrets, also known as business secrets, refer to technical information and business information such as design data, procedures, product formulas, manufacturing processes, manufacturing methods, management know-how, customer lists, supply information, production and marketing strategies, bid prices and bid contents in bidding, which are not known to the public, can bring economic benefits to the right holder, are practical and have been kept confidential by the right holder; among them, not known to the public means that the information cannot be directly obtained from public channels; can bring economic benefits to the right holder, and are practical means that the information has definite applicability and can bring real or potential economic benefits or competitive advantages to the right holder;

[0120] S34: Checking the classified information data;

[0121] S4: According to the data format standards of each business, the data verification module will perform format verification and data cleaning on the information data that has completed data classification;

[0122] In the present invention, the data verification module includes a format verification unit and a data cleaning unit; the format verification unit uses a format error correction algorithm to verify and correct the format of information data at all levels according to various business data format standards; the business data format standards include but are not limited to naming rules, data types, date formats, metadata standards, data dictionaries, interface standards, integration specifications, permission specifications and access policies; the data cleaning unit uses a preset content cleaning model to perform information content error correction on the information data that has completed format verification and correction;

[0123] S5: The data association module extracts key information from the verified and cleaned information data, and verifies the association of the information data based on the key information to adjust the information data;

[0124] In the present invention, the process of the data association module performing association verification and adjustment on the information data of each sub-data block is as follows:

[0125] S51: extract key information of the information data in the sub-data block; the key information includes event address, event time and event subject;

[0126] S52: De-duplication and relevance verification of information data according to the acquired key information and the preset time range;

[0127] S53: constructing the information data that has completed the relevance verification into a relevance information map, and combining the identification number and the key information to form a unique identification of each sub-data block;

[0128] S6: The data aggregation module merges all sub-data blocks and related information data;

[0129] In the present invention, the process of the data aggregation module merging each sub-data block and the internal information data is as follows:

[0130] S61: sorting all sub-data blocks according to the unique identifier of each sub-data block, and parsing the unique identifier to obtain all event information of each sub-data block;

[0131] S62: Calculate the correlation between each event information in different sub-data blocks;

[0132] S63: Virtually linking different sub-data blocks whose correlation exceeds a set aggregation threshold;

[0133] S64: If all event information in the current sub-data block is virtually linked by the additional sub-data block, the information data of the current sub-data block is segmented to complete data aggregation with other information data; otherwise, go to step S65;

[0134] S65: If the individual event information in the current sub-data block is not virtually linked to the additional sub-data block, the virtually linked part of the current sub-data block is split, and the non-virtual linked part is reassigned with an identification number, thereby completing data aggregation with other information data;

[0135] S66: merging the information data after data aggregation to form a complete event information chain;

[0136] S7: According to the urgency of the event, the information sorting module sorts the merged information data;

[0137] In the present invention, the data sorting module includes an event characterization unit, an event sorting unit and an event dispatching unit; the event characterization unit uses an expert prior model to determine and refine the event nature of the complete event information chain; the event nature includes natural disasters, accident disasters, public health events and social security events; the event sorting unit uses a parallel cardinality algorithm to sort the characterized complete event information chain according to the event urgency to construct an event processing code table; the event urgency is divided into three levels of first-level emergency events, second-level emergency events and third-level emergency events according to the impact of the emergency event on the society; the event dispatching unit distributes the event information in the event processing code table according to similar historical events, the event responsible departments at different time nodes and the event processing progress, and creates an event processing schedule to complete event processing tracking;

[0138] In this embodiment, the data cleaning unit uses a preset content cleaning model to perform information content error correction on the information data that has completed format verification and correction. The process is as follows:

[0139] The sensitivity dimension level includes 5 levels, from high to low: 5: top secret, 4: confidential, 3: secret, 2: internal disclosure and 1: external disclosure. The higher the level, the larger the number assigned;

[0140] The content cleaning model first encrypts the information data after format correction according to the sensitive dimension level to obtain the encrypted feature values ​​of the information data at each level:

[0141]

[0142] Where P represents the encrypted feature value, y represents the sensitive dimension level, and y≥3; γ represents the identification number sequence number, and δ represents the total number of digits of the identification number;

[0143] Then, the obtained encrypted feature values ​​at all levels and information data with sensitivity dimension levels of 1 to 2 are input into the feature extraction layer, and feature extraction of different depths is performed according to the sensitivity dimension level to obtain a reduced dimension feature matrix; the feature extraction layer includes 4 3X3 convolutional layers, 2 3X3 maximum pooling layers and 3 normalization layers; the number of feature extractions of encrypted feature values ​​at different sensitivity dimension levels is the square of the sensitivity dimension level, and the feature extraction layer is trained using a multidimensional scaling algorithm;

[0144] Then, the obtained reduced dimension feature matrix is ​​loaded into the learning and training layer for multiple iterative training to obtain the feature weight matrix of the information data; the learning and training layer includes 2 1X1 convolutional layers, 1 3X3 upsampling layer, 1 3X3 downsampling layer, 3 InceptionV3 blocks, 4 residual blocks, 3 average pooling layers and RELU activation function; for the eigenvalues ​​of different levels in the reduced dimension feature matrix, the connection methods of the 3 InceptionV3 blocks and 4 residual blocks of the learning and training layer are as follows: if the sensitive dimension level belongs to 1 to 2, the 4 residual blocks are connected in series and then input into 3 parallel InceptionV3 blocks; if the sensitive dimension level belongs to 3 to 4, the 3 InceptionV3 blocks are respectively connected in parallel with the 3 residual blocks and then input into the 4th residual block; if the sensitive dimension level is 5, the 3 residual blocks respectively use 1 InceptionV3 block as the second branch and are connected in series with the 4th residual block; thereby reducing the sensitive dimension of the feature weight matrix;

[0145] Finally, the acquired feature weight matrix is ​​sequentially input into two average pooling layers, two fully connected layers and the SoftMax activation function to realize the abnormality judgment of the information data, and then different abnormality processing algorithms are dispatched to correct the abnormal content of the information data; the abnormality processing algorithms include but are not limited to data deduplication algorithm, data filling algorithm, data conversion algorithm and data simplification algorithm;

[0146] In this embodiment, the working principle of the content cleaning model is as follows: the content cleaning model is a method based on feature extraction and learning training, which is used to handle various anomalies and errors in information data; the workflow of the content cleaning model is mainly divided into three steps, namely feature extraction, learning training and anomaly handling;

[0147] In the feature extraction stage, the content cleaning model first encrypts the information data that has completed format correction according to the sensitivity dimension level to obtain the encrypted feature values ​​of the information data at each level. The number of feature extractions of the encrypted feature values ​​of different sensitivity dimension levels is the square of the sensitivity dimension level, and the feature extraction layer is trained using a multidimensional scaling algorithm; the feature extraction layer includes 4 3×3 convolutional layers, 2 3×3 maximum pooling layers, and 3 normalization layers; the sensitivity dimension level includes 5 levels, from high to low, top secret, confidential, secret, internally disclosed, and externally disclosed, and the higher the level, the larger the number assigned;

[0148] In the learning and training stage, the content cleaning model inputs the encrypted feature values ​​of each level into the feature extraction layer, and performs feature extraction of different depths according to different sensitive dimension levels to obtain a reduced dimension feature matrix. Then, the reduced dimension feature matrix is ​​loaded into the learning and training layer for multiple iterative training to obtain the feature weight matrix of the information data. The learning and training layer includes 2 1×1 convolutional layers, 1 3×3 upsampling layer, 1 3×3 downsampling layer, 3 InceptionV3 blocks, 4 residual blocks, 3 average pooling layers and RELU activation function; the feature values ​​of different sensitive dimension levels are trained through different connection methods to reduce the sensitive dimension of the feature weight matrix; if the sensitive dimension level belongs to 1 to 2, the 4 residual blocks are connected in series and then input into 3 parallel InceptionV3 blocks; if the sensitive dimension level belongs to 3 to 4, the 3 InceptionV3 blocks are connected in parallel with the 3 residual blocks and then input into the 4th residual block; if the sensitive dimension level is 5, the 3 residual blocks use 1 InceptionV3 block as the second branch and are connected in series with the 4th residual block;

[0149] In the exception handling stage, the content cleaning model will input the acquired feature weight matrix into two average pooling layers, two fully connected layers and the SoftMax activation function in turn to realize the abnormal judgment of information data. If there is an abnormality in the data, the model will schedule different exception handling algorithms to correct the data, such as data deduplication algorithm, data filling algorithm, data conversion algorithm and data simplification algorithm; after multiple training and processing, the model can effectively clean the information data that meets the expected format and specifications.

[0150] In this embodiment, the distributed acceleration algorithm, POST request, PUT request, format error correction algorithm, expert prior model and parallel cardinality algorithm are all commonly used technical means in this field and will not be described in detail here.

[0151] Example:

[0152] First, the mobile terminal sends a POST request and / or a PUT request to the big data platform to establish a verification channel; the mobile terminal includes but is not limited to mobile phones, web pages, tablet computers and laptops; the big data platform agrees to the POST request and / or the PUT request, and establishes a connection with the verification channel, and sends a chaotic verification code to the mobile terminal at the same time. The mobile terminal receives the chaotic verification code to complete the identity verification and establishes a parallel transmission channel with the data acquisition module. The big data platform divides the mobile terminals into several batches according to the completion time of the identity verification, and sorts the mobile terminals in the same batch according to the construction time of the parallel transmission channel. The mobile terminal sends the information data to the parallel transmission channel according to the batch and sorting results. Channel; information data includes but is not limited to basic identity information, matter classification information, basic directory information, implementation list information, application registration information, application materials information, intermediary service information, online evaluation information, consultation information, complaint information, case acceptance information, material directory information, handling process information, handling progress information, special procedure information, completion information, basic supervision information, supervision process information, supervision result information, electronic seal information, electronic certificate information, payment information, logistics information, user experience monitoring information, government service evaluation information and service situation analysis information; the data acquisition module uses parallel transmission channels and distributed acceleration algorithms to receive information data from mobile terminals and transmit them to the data segmentation module for processing;

[0153] Subsequently, the received massive information data is simulated and divided according to the type of mobile terminal to construct simulated data blocks, and the capacity and number of the simulated data blocks are determined. Then, according to the integrity of the information data in the simulated data blocks, the capacity and number of the simulated data blocks are adjusted. Then, according to the data collection time, the information data in each simulated data block is marked, and the marked information data is clustered according to a preset time range. Then, the clustered information data is separated to complete the internal segmentation of the simulated data block, and a number of sub-data blocks are formed and assigned identification numbers, and the capacity and number of each sub-data block are measured at the same time;

[0154] Next, the characteristic attributes of the internal information data of the sub-data block are obtained, including but not limited to dependency, extensibility, transferability, storage, concentration, sharing, predictability, validity, invalidity and / or processability, etc.; the information data with the same or similar characteristic attributes are classified according to the event business type, and the event business type includes but is not limited to identity authentication, event management, service application, service feedback, service interaction, event acceptance, event handling, event completion, electronic supervision, basic support and data services; and the classified information data is graded according to the sensitive dimensions of the information data, and the sensitive dimensions are but not limited to personal privacy information, commercial secrets, national security information, medical care information, education information and social media information; and the graded information data is then subject to data verification;

[0155] Subsequently, the format verification unit of the data verification module uses a format correction algorithm to verify and correct the format of information data at all levels according to the format standards of each business data; the business data format standards include but are not limited to naming rules, naming conventions, data types, date formats, metadata standards, data dictionaries, interface standards, integration specifications, authority specifications and access policies; the data cleaning unit uses a preset content cleaning model to correct the information content of the information data that has completed format verification and correction; then extract the key information of the information data in the sub-data block, including the event address, event time and event subject; then, according to the obtained key information and the preset time range, the information data is de-duplicated, normalized and verified for relevance, and the preset time range can be year, month, day, hour, or even specific to minutes and seconds; then the information data that has completed relevance verification is constructed into a related information map, and the unique identification of each sub-data block is formed in combination with the identification number and key information;

[0156] Then, all sub-data blocks are sorted according to their unique identifiers, and the unique identifiers are parsed to obtain all event information of each sub-data block, and the correlation between event information in different sub-data blocks is calculated. At the same time, different sub-data blocks with correlation exceeding the set aggregation threshold are virtually linked. The aggregation threshold can be set to 0.75-0.99; thus, the following situations are generated:

[0157] If all event information in the current sub-data block is virtually linked by the additional sub-data block, the information data of the current sub-data block is segmented, thereby completing data aggregation; if individual event information in the current sub-data block is not virtually linked by the additional sub-data block, the current sub-data block will segment the virtually linked part and re-assign identification numbers to the non-virtual linked part, thereby completing data aggregation; at the same time, the information data that has completed data aggregation will be merged to form a complete event information chain;

[0158] Finally, the event characterization unit of the data sorting module uses the expert prior model to determine and refine the event nature of the complete event information chain; the event nature includes natural disasters, accidental disasters, public health events and social security events; the event sorting unit uses the parallel cardinality algorithm to sort the characterized complete event information chain according to the urgency of the event to construct an event processing coding table; the event urgency is divided into three levels: first-level emergency events, second-level emergency events and third-level emergency events according to the degree of impact of the emergency on the society; the event dispatching unit distributes the event information in the event processing coding table according to similar historical events, the event responsible departments at different time nodes and the event processing progress, and creates an event processing schedule to complete event processing tracking.

Claims

1. A data cleaning method for an intelligent big data platform, characterized in that: The following steps are involved: S1: The big data platform uses the data collection module to obtain massive information data fed back by mobile terminals; S2: The data segmentation module divides the collected massive information data into several sub-data blocks according to the terminal type and data collection time; S3: According to the event business type, the data classification module classifies and grades the information data in each sub-data block; S4: According to the data format standards of each business, the data verification module will perform format verification and data cleaning on the information data that has completed data classification; S5: The data association module extracts key information from the verified and cleaned information data, and verifies the association of the information data based on the key information to adjust the information data; S6: The data aggregation module merges all sub-data blocks and related information data; S7: According to the urgency of the event, the information sorting module sorts the merged information data; In step S1, the workflow of the big data platform using the data acquisition module to obtain mobile terminal information data is as follows: A: The mobile terminal sends a POST request and / or a PUT request to the big data platform to establish a verification channel; Mobile terminals include but are not limited to mobile phones, web pages, tablet computers and laptops; B: The big data platform agrees to the POST request and / or PUT request, establishes a connection with the verification channel, and sends the chaos verification code to the mobile terminal respectively: Wherein, C represents the chaotic check code, i represents the random number sequence, x represents the number of random digits, tent() represents the discrete chaotic function, rand() represents the random generation function, j represents the mapping number sequence, n represents the total mapping number, and Logs() represents the mapping modulation function; C: The mobile terminal receives the chaotic check code to complete the identity verification and establishes a parallel transmission channel with the data acquisition module: Among them, GrT() represents the parallel transmission channel group, k represents the chaotic check code number, s represents the number of chaotic check code bits, C k represents the k-th chaotic check code, Offer() represents the channel creation function, Track() represents the audio channel function, Source() represents the video channel function, Text() represents the text channel function, and Image() represents the image channel function; D: The big data platform divides the mobile terminals into several batches according to the completion time of identity verification, and sorts the mobile terminals in the same batch according to the construction time of the parallel transmission channel; E: The mobile terminal sends the information data to the parallel transmission channel according to the batch and sorting results; Information data includes but is not limited to basic identity information, matter classification information, basic directory information, implementation list information, application registration information, application materials information, intermediary service information, online evaluation information, consultation information, complaint information, case acceptance information, material directory information, handling process information, handling progress information, special procedure information, completion information, basic supervision information, supervision process information, supervision result information, electronic seal information, electronic certificate information, payment information, logistics information, user experience monitoring information, government service evaluation information, and service situation analysis information; F: The data acquisition module uses parallel transmission channels and distributed acceleration algorithms to receive information data from mobile terminals and transmit it to the data segmentation module for processing; In step S2, the process of the data segmentation module dividing the massive information data into several sub-data blocks is as follows: First, the received massive information data is simulated and divided according to the type of mobile terminal to construct simulated data blocks, and the capacity and number of simulated data blocks are determined: in, represents the simulated data block of the zth mobile terminal, vr represents the virtual identifier, impart() represents the data segmentation function, FD represents the massive information data, ⊙ represents the Hadamard outer product, MT z represents the zth type of mobile terminal; Then, according to the integrity of the information data in the simulation data block, the capacity and quantity of the simulation data block are adjusted; Subsequently, the information data in each simulation data block is labeled according to the data collection time, and the labeled information data is clustered according to a preset time range; Among them, Km α represents the αth clustering data block that meets the time range, α represents the total number of time ranges that meet the time range, Indicates the data collection time of extracting all information data in the simulation data block of the zth mobile terminal; the time range is year, month, day and hour; Finally, the clustered information data is separated to complete the internal segmentation of the simulated data block, and several sub-data blocks are formed and assigned identification numbers, and the capacity and number of each sub-data block are calculated at the same time: in, represents the βth sub-data block, represents the Hadamard inner product, rl represents the identification number; In step S3, the process of the data classification module classifying the internal information data of each sub-data block is as follows: S31: Acquire characteristic attributes of internal information data of the sub-data block; Characteristic attributes include, but are not limited to, dependency, extensibility, transferability, storage, concentration, sharing, predictability, validity, invalidity, and / or processing; S32: Classify information data with the same or similar characteristic attributes according to event business types; Event business types include but are not limited to identity authentication, event management, service application, service feedback, service interaction, event acceptance, event handling, event completion, electronic supervision, basic support and data services; S33: classifying the classified information data according to the sensitive dimensions of the information data; Sensitive dimensions include, but are not limited to, personal privacy information, business secrets, national security information, healthcare information, education information, and social media information; S34: Checking the classified information data; In step S4, the data verification module includes a format verification unit and a data cleaning unit; the format verification unit uses a format error correction algorithm to verify and correct the format of information data at all levels according to various business data format standards; the business data format standards include but are not limited to naming rules, naming conventions, data types, date formats, metadata standards, data dictionaries, interface standards, integration specifications, permission specifications and access policies; the data cleaning unit uses a preset content cleaning model to perform information content error correction on the information data that has completed format verification and correction; The sensitivity dimension level includes 5 levels, from high to low, 5: top secret, 4: confidential, 3: secret, 2: internal disclosure and 1: external disclosure, and the higher the level, the larger the number assigned; The content cleaning model first encrypts the information data after format correction according to the sensitive dimension level to obtain the encrypted feature values ​​of the information data at each level: Where P represents the encrypted feature value, y represents the sensitive dimension level, and y≥3; γ represents the identification number sequence number, and δ represents the total number of digits of the identification number; Then, the obtained encrypted feature values ​​at all levels and information data with sensitivity dimension levels of 1 to 2 are input into the feature extraction layer, and feature extraction of different depths is performed according to the sensitivity dimension level to obtain a reduced dimension feature matrix; the feature extraction layer includes 4 3X3 convolutional layers, 2 3X3 maximum pooling layers and 3 normalization layers; the number of feature extractions of encrypted feature values ​​at different sensitivity dimension levels is the square of the sensitivity dimension level, and the feature extraction layer is trained using a multidimensional scaling algorithm; Then, the obtained reduced dimension feature matrix is ​​loaded into the learning and training layer for multiple iterative training to obtain the feature weight matrix of the information data; the learning and training layer includes 2 1X1 convolutional layers, 1 3X3 upsampling layer, 1 3X3 downsampling layer, 3 InceptionV3 blocks, 4 residual blocks, 3 average pooling layers and RELU activation function; for the eigenvalues ​​of different levels in the reduced dimension feature matrix, the connection methods of the 3 InceptionV3 blocks and 4 residual blocks of the learning and training layer are as follows: if the sensitive dimension level belongs to 1 to 2, the 4 residual blocks are connected in series and then input into 3 parallel InceptionV3 blocks; if the sensitive dimension level belongs to 3 to 4, the 3 InceptionV3 blocks are respectively connected in parallel with the 3 residual blocks and then input into the 4th residual block; if the sensitive dimension level is 5, the 3 residual blocks respectively use 1 InceptionV3 block as the second branch and are connected in series with the 4th residual block; thereby reducing the sensitive dimension of the feature weight matrix; Finally, the acquired feature weight matrix is ​​sequentially input into two average pooling layers, two fully connected layers and the SoftMax activation function to realize the abnormality judgment of the information data, and then different abnormality processing algorithms are dispatched to correct the abnormal content of the information data; the abnormality processing algorithms include but are not limited to data deduplication algorithm, data filling algorithm, data conversion algorithm and data simplification algorithm; In step S7, the data sorting module includes an event characterization unit and an event sorting unit; the event characterization unit uses an expert prior model to determine and refine the event nature of the complete event information chain; the event nature includes natural disasters, accident disasters, public health events and social security events; the event sorting unit uses a parallel cardinality algorithm to sort the characterized complete event information chain according to the urgency of the event to construct an event processing coding table; the event urgency is divided into three levels: first-level emergency, second-level emergency and third-level emergency according to the degree of impact of the emergency on society.

2. The data cleaning method of the intelligent big data platform according to claim 1 is characterized by: In step S5, the data association module verifies and adjusts the association of the information data of each sub-data block as follows: S51: extract key information of the information data in the sub-data block; the key information includes event address, event time and event subject; S52: De-duplication and relevance verification of information data according to the acquired key information and the preset time range; S53: Construct the information data that has completed the relevance verification into a relevance information map, and combine the identification number and key information to form a unique identification of each sub-data block.

3. The data cleaning method of the intelligent big data platform according to claim 1 is characterized by: In step S6, the process of the data aggregation module merging each sub-data block and the internal information data is as follows: S61: sorting all sub-data blocks according to the unique identifier of each sub-data block, and parsing the unique identifier to obtain all event information of each sub-data block; S62: Calculate the correlation between each event information in different sub-data blocks; S63: Virtually linking different sub-data blocks whose correlation exceeds a set aggregation threshold; S64: If all event information in the current sub-data block is virtually linked by the additional sub-data block, the information data of the current sub-data block is segmented, thereby completing data aggregation with other information data; Otherwise, go to step S65; S65: If the individual event information in the current sub-data block is not virtually linked to the additional sub-data block, the virtually linked part of the current sub-data block is split, and the non-virtual linked part is reassigned with an identification number, thereby completing data aggregation with other information data; S66: Merge the information data that has completed data aggregation to form a complete event information chain.

4. The data cleaning method of the intelligent big data platform according to claim 1 is characterized by: The data sorting module also includes an event dispatching unit; the event dispatching unit distributes the event information in the event processing coding table according to similar historical events, the event responsible departments at different time nodes and the event processing progress, and creates an event processing schedule to complete event processing tracking.

Images