Data security processing method and device and related product
By generating key-value pairs and adding noise perturbation to the cached data pool, the problem of slow big data processing speed is solved, and real-time and security of data processing are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- FUZHOU QIYUAN INFORMATION TECHNOLOGY CO LTD
- Filing Date
- 2023-03-14
- Publication Date
- 2026-07-03
AI Technical Summary
Big data processing is slow due to the large volume of data, making it impossible to guarantee the real-time performance and security of data requirements.
Data is formatted by generating key-value pairs, and noise perturbation is added to the cached data pool to generate pseudo-real data, thus enabling fast data processing.
It provides a rapid data processing solution, ensuring the real-time nature and security of data supply.
Smart Images
Figure CN116415295B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of privacy computing technology, and in particular to a data security processing method, apparatus and related products. Background Technology
[0002] The rapid development of big data has gradually revealed the value of data, but the resulting data security issues have also attracted widespread attention. If data is maliciously attacked and stolen, it can cause significant losses and impacts on both users and data managers.
[0003] Therefore, to prevent data theft, data security measures are necessary during storage and use. However, current data security measures suffer from slow processing speeds due to the large volume of data, making it impossible to guarantee real-time data availability. Summary of the Invention
[0004] In view of the above problems, this application provides a data security processing method, apparatus and related products.
[0005] The embodiments of this application disclose the following technical solutions:
[0006] A key-value pair-based data processing method, comprising:
[0007] Obtain the data to be processed;
[0008] The data to be processed is formatted to generate key-value pairs, wherein the key in the key-value pair is the feature attribute name of the data to be processed, and the value in the key-value pair is the assignment of the feature attribute name;
[0009] Identify key-value pairs with the same key, and write all key-value pairs with the same key into the same cached data pool created in real time;
[0010] Locally within the cached data pool, noise perturbations are added to all key-value pairs to generate pseudo-real data.
[0011] Optionally, the method further includes: dividing the data to be processed based on determined data cutting points to obtain a subset of the data to be processed;
[0012] The step of formatting the data to be processed to generate key-value pairs includes: formatting each subset of the data to be processed corresponding to the data to be processed to generate key-value pairs.
[0013] Optionally, the step of segmenting the data to be processed based on determined data segmentation points to obtain a subset of the data to be processed includes:
[0014] The data to be processed is subjected to a quality assessment to obtain a quality assessment value;
[0015] Based on the quality assessment values, data split points are determined.
[0016] Optionally, the step of performing a quality assessment on the data to be processed to obtain a quality assessment value includes:
[0017] Calculate the information entropy of the data to be processed;
[0018] The quality assessment value is obtained based on the information entropy.
[0019] Optionally, calculating the information entropy of the data to be processed includes:
[0020] The probability of the feature attribute name appearing in the data to be processed is evaluated.
[0021] The information entropy of the data to be processed is calculated based on the assessed probability.
[0022] Optionally, obtaining the quality assessment value based on the information entropy includes:
[0023] Map the data to be processed into a real vector;
[0024] The information entropy is injected into the real vector to obtain a quality assessment value.
[0025] Optionally, the method further includes:
[0026] Obtain the target dataset comprising several data blocks;
[0027] Each data block is labeled with attributes to obtain the attribute feature vector corresponding to each data block;
[0028] Based on the dimensions of the attribute feature vectors, construct the nodes of the decision tree;
[0029] The similarity between attribute feature vectors is obtained by calculating the similarity between attribute feature vectors of different data blocks;
[0030] Based on the similarity between the attribute feature vectors, the several data blocks are serialized and recombined to obtain several data subsets;
[0031] Noise is added to the plurality of data subsets and matched with the nodes of the decision tree to generate a decision tree, and the data to be processed is obtained based on the decision tree.
[0032] Optionally, the method further includes: dividing the target data into blocks to obtain several data blocks, and determining the data to be processed in units of the data blocks.
[0033] Optionally, the method further includes: performing parallel attribute annotation processing on several data blocks corresponding to the target data to obtain the feature attribute names and corresponding values of the data to be processed.
[0034] Optionally, the method further includes: invoking a set replacement data sampling mechanism to extract sample data from the target data, forming a dataset to be processed into blocks based on the extracted sample data, so that the block processing is specifically performed on the dataset to be processed into blocks.
[0035] Optionally, the step of slicing the target data into several data blocks and determining the data to be processed in units of the data blocks includes: slicing the target data horizontally or vertically into several data blocks and determining the data to be processed in units of the data blocks.
[0036] A key-value pair-based data processing apparatus, comprising:
[0037] The data acquisition unit is used to acquire data to be processed.
[0038] A formatting unit is used to format the data to be processed to generate key-value pairs, wherein the key in the key-value pair is the feature attribute name of the data to be processed, and the value in the key-value pair is the assignment of the feature attribute name;
[0039] A cache unit is used to identify key-value pairs with the same key so that all key-value pairs with the same key are written into the same cache data pool created in real time.
[0040] A perturbation unit is used to locally add noise perturbation to all key-value pairs in the cached data pool to generate pseudo-real data.
[0041] Optionally, the formatting unit is further configured to perform segmentation processing on the data to be processed based on determined data segmentation points to obtain a subset of data to be processed, so as to format each subset of data to be processed corresponding to the data to be processed to generate key-value pairs.
[0042] Optionally, the formatting unit is specifically used to perform quality assessment on the data to be processed to obtain a quality assessment value; and to determine data segmentation points based on the quality assessment value.
[0043] Optionally, the formatting unit is specifically used to calculate the information entropy of the data to be processed; and to obtain a quality assessment value based on the information entropy.
[0044] Optionally, the formatting unit is specifically used to evaluate the probability of the feature attribute name appearing in the data to be processed; and to calculate the information entropy of the data to be processed based on the evaluated probability.
[0045] Optionally, the formatting unit is specifically used to map the data to be processed into a real vector; and to inject the information entropy into the real vector to obtain a quality assessment value.
[0046] Optionally, the apparatus further includes: a slicing unit, used to slice the target data into several data blocks, and to determine the data to be processed in units of the data blocks.
[0047] Optionally, the apparatus further includes: a labeling unit, used to perform parallel attribute labeling processing on several data blocks corresponding to the target data, to obtain the feature attribute names and corresponding values of the data to be processed.
[0048] Optionally, the apparatus further includes: an extraction unit, configured to invoke a set replacement data sampling mechanism to extract sample data from the target data, to form a dataset to be segmented based on the extracted sample data, and to specifically target the dataset to be segmented during the segmentation process.
[0049] Optionally, the slicing unit is specifically used to slice the target data horizontally or vertically to obtain several data blocks, and the data to be processed is determined in units of the data blocks.
[0050] An electronic device includes a memory and a processor, wherein the memory stores an executable program, and the processor performs the following steps when running the executable program:
[0051] Obtain the data to be processed;
[0052] The data to be processed is formatted to generate key-value pairs, wherein the key in the key-value pair is the feature attribute name of the data to be processed, and the value in the key-value pair is the assignment of the feature attribute name;
[0053] Identify key-value pairs with the same key, and write all key-value pairs with the same key into the same cached data pool created in real time;
[0054] Locally within the cached data pool, noise perturbations are added to all key-value pairs to generate pseudo-real data.
[0055] A computer storage medium storing a computer-executable program, which, when executed, performs the following steps:
[0056] Obtain the data to be processed;
[0057] The data to be processed is formatted to generate key-value pairs, wherein the key in the key-value pair is the feature attribute name of the data to be processed, and the value in the key-value pair is the assignment of the feature attribute name;
[0058] Identify key-value pairs with the same key, and write all key-value pairs with the same key into the same cached data pool created in real time;
[0059] Locally within the cached data pool, noise perturbations are added to all key-value pairs to generate pseudo-real data.
[0060] A computer program product, wherein the computer storage medium stores computer-executable instructions, which, when executed, perform the following steps:
[0061] Obtain the data to be processed;
[0062] The data to be processed is formatted to generate key-value pairs, wherein the key in the key-value pair is the feature attribute name of the data to be processed, and the value in the key-value pair is the assignment of the feature attribute name;
[0063] Identify key-value pairs with the same key, and write all key-value pairs with the same key into the same cached data pool created in real time;
[0064] Locally within the cached data pool, noise perturbations are added to all key-value pairs to generate pseudo-real data.
[0065] The solution provided in this application involves: acquiring data to be processed; formatting the data to be processed to generate key-value pairs, wherein the key in the key-value pair is the name of a feature attribute of the data to be processed, and the value in the key-value pair is the assignment of the feature attribute name; identifying key-value pairs with the same key, and writing all key-value pairs with the same key into the same cached data pool created in real time; and locally in the cached data pool, adding noise perturbation to all key-value pairs to generate pseudo-real data, thereby providing a fast data processing solution and ensuring the real-time nature and security of data supply. Attached Figure Description
[0066] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0067] Figure 1This is a flowchart illustrating a key-value pair-based data processing method according to this application.
[0068] Figure 2 This is a schematic diagram of the structure of a key-value pair-based data processing device according to this application.
[0069] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application.
[0070] Figure 4 This is a schematic diagram of the hardware structure of the electronic device in the embodiments of this application. Detailed Implementation
[0071] Implementing any technical solution of the embodiments of this application does not necessarily require achieving all of the above advantages at the same time.
[0072] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0073] Figure 1 This is a flowchart illustrating a key-value pair-based data processing method according to this application. Figure 1 As shown, it includes:
[0074] S101. Obtain the data to be processed;
[0075] S102. Format the data to be processed to generate key-value pairs, wherein the key in the key-value pair is the feature attribute name of the data to be processed, and the value in the key-value pair is the assignment of the feature attribute name;
[0076] S103. Determine key-value pairs with the same key, and write all key-value pairs with the same key into the same cache data pool created in real time;
[0077] S104. Locally within the cached data pool, add noise perturbations to all key-value pairs to generate pseudo-real data.
[0078] Optionally, the method further includes: dividing the data to be processed based on determined data cutting points to obtain a subset of the data to be processed;
[0079] The step of formatting the data to be processed to generate key-value pairs includes: formatting each subset of the data to be processed corresponding to the data to be processed to generate key-value pairs.
[0080] Optionally, the step of segmenting the data to be processed based on determined data segmentation points to obtain a subset of the data to be processed includes:
[0081] The data to be processed is subjected to a quality assessment to obtain a quality assessment value;
[0082] Based on the quality assessment values, data split points are determined.
[0083] Optionally, the step of performing a quality assessment on the data to be processed to obtain a quality assessment value includes:
[0084] Calculate the information entropy of the data to be processed;
[0085] The quality assessment value is obtained based on the information entropy.
[0086] Optionally, in one embodiment, the method further includes:
[0087] Obtain the target dataset comprising several data blocks;
[0088] Each data block is labeled with attributes to obtain the attribute feature vector corresponding to each data block;
[0089] Based on the dimensions of the attribute feature vectors, construct the nodes of the decision tree;
[0090] The similarity between attribute feature vectors is obtained by calculating the similarity between attribute feature vectors of different data blocks;
[0091] Based on the similarity between the attribute feature vectors, the several data blocks are serialized and recombined to obtain several data subsets;
[0092] Noise is added to the plurality of data subsets and matched with the nodes of the decision tree to generate a decision tree. Based on the decision tree, the data to be processed is obtained, thereby improving data efficiency. Optionally, the method further includes:
[0093] Access the target data source to obtain target data from the target data source, and combine all target data obtained from the same target data source into a target dataset;
[0094] The target dataset is divided into several data blocks.
[0095] Optionally, the step of performing attribute annotation processing on each data block to obtain the attribute feature vector corresponding to each data block includes: performing parallel attribute annotation processing on several data blocks through a parallel annotation processing task to obtain the attribute feature vector corresponding to each data block.
[0096] Optionally, the step of performing parallel attribute annotation processing on several data blocks through a parallel annotation processing task to obtain the attribute feature vector corresponding to each data block includes: obtaining a scheduling command issued by the control node in the distributed processing cluster to start the parallel annotation processing task; and creating a parallel annotation processing thread according to the parallel annotation processing task to allocate the several data blocks one by one to the annotation processing thread for parallel attribute annotation processing.
[0097] Optionally, the step of performing data segmentation processing on the target dataset to obtain several data blocks includes: performing data segmentation processing on the target dataset based on the number of annotation processing threads to obtain several data blocks, such that the number of data blocks is equal to the number of annotation processing threads.
[0098] Optionally, the step of performing data segmentation processing on the target dataset to obtain several data blocks includes: performing data segmentation processing on the target dataset based on the number of annotation processing threads and the data processing volume of a single annotation processing thread to obtain several data blocks, such that the data volume of a single data block is equivalent to the data volume of a single annotation processing thread.
[0099] Optionally, the step of performing parallel attribute annotation processing on several data blocks through a parallel annotation processing task to obtain the attribute feature vector corresponding to each data block includes: the parallel annotation processing task loading a pre-set set of data attribute features, performing parallel attribute annotation processing on several data blocks corresponding to the target dataset, and obtaining the attribute feature vector corresponding to each data block.
[0100] Optionally, the parallel annotation processing task loads a pre-defined set of data attribute features and performs parallel attribute annotation processing on several data blocks corresponding to the target dataset to obtain the attribute feature vector corresponding to each data block. This includes: the parallel annotation processing task loads a pre-defined set of data attribute features and performs parallel attribute annotation processing on several data blocks corresponding to the target dataset according to regular expression matching to obtain the attribute feature vector corresponding to each data block.
[0101] Optionally, the parallel attribute annotation process is performed on several data blocks corresponding to the target dataset to obtain the attribute feature vector corresponding to each data block, including:
[0102] Parallel attribute annotation processing is performed on several data blocks corresponding to the target dataset, and attribute annotation values are assigned to each data block.
[0103] Based on the labeled values, the attribute feature vector corresponding to each data block is obtained.
[0104] Optionally, the step of performing data slicing processing on the target dataset to obtain several data blocks may further include: extracting sample data from the target dataset to form a dataset to be sliced based on the extracted sample data, so that the data slicing processing is specifically performed on the dataset to be sliced.
[0105] Optionally, the method further includes: invoking a set data sampling mechanism with replacement to extract sample data from the target dataset, forming a data set to be segmented based on the extracted sample data, so that the data segmentation process is specifically performed on the data set to be segmented.
[0106] Optionally, calculating the information entropy of the data to be processed includes:
[0107] The probability of the feature attribute name appearing in the data to be processed is evaluated.
[0108] The information entropy of the data to be processed is calculated based on the assessed probability.
[0109] Optionally, obtaining the quality assessment value based on the information entropy includes:
[0110] Map the data to be processed into a real vector;
[0111] The information entropy is injected into the real vector to obtain a quality assessment value.
[0112] Optionally, the method further includes: dividing the target data into blocks to obtain several data blocks, and determining the data to be processed in units of the data blocks.
[0113] Optionally, the method further includes: performing parallel attribute annotation processing on several data blocks corresponding to the target data to obtain the feature attribute names and corresponding values of the data to be processed. For example, obtaining the scheduling command issued by the control node in the distributed processing cluster; and according to the scheduling command, allocating the several data blocks one by one to an annotation node for parallel attribute annotation processing.
[0114] Optionally, the method further includes: invoking a pre-defined sampling mechanism with replacement to extract sample data from the target data, forming a dataset to be segmented based on the extracted sample data, so that during the segmentation process, the data is specifically targeted at the dataset to be segmented, which also effectively adds random noise to ensure data security; in addition, it also allows for the existence of identical data content between data blocks and allows for the duplication of data content between different data blocks, which is equivalent to adding random noise and improving data security.
[0115] Optionally, the step of slicing the target data into several data blocks and determining the data to be processed in units of the data blocks includes: slicing the target data horizontally or vertically into several data blocks and determining the data to be processed in units of the data blocks.
[0116] For example, when performing horizontal slicing, based on the number of labeled nodes, the target data is sliced into several data blocks, such that the number of data blocks is equal to the number of labeled nodes.
[0117] For example, when performing vertical segmentation, based on the number of labeled nodes and the amount of data processed by a single labeled node, data segmentation is performed on the target data to obtain several data blocks, so that the amount of data in a single data block is equal to the amount of data in a single labeled node, thereby enabling good parallelism during parallel processing and greatly reducing running time.
[0118] Furthermore, during parallel annotation, based on a pre-defined set of data attribute features, parallel attribute annotation processing can be performed on several data blocks corresponding to the target data to obtain the attribute feature vector corresponding to each target data.
[0119] For example, the method of performing parallel attribute annotation processing on several data blocks corresponding to the target data based on a pre-defined set of data attribute features to obtain an attribute feature vector corresponding to each target data includes: performing parallel attribute annotation processing on several data blocks corresponding to the target data based on a pre-defined set of data attribute features according to regular expression matching to obtain an attribute feature vector corresponding to each target data.
[0120] Based on a pre-defined set of data attribute features, parallel attribute annotation processing is performed on several data blocks corresponding to the target data to obtain an attribute feature vector for each target data, including:
[0121] Based on a pre-defined set of data attribute features, parallel attribute annotation processing is performed on several data blocks corresponding to the target data, and attribute annotation values are assigned to each data block.
[0122] Based on the labeled values, the attribute feature vector corresponding to each target data is obtained.
[0123] The method may further include: performing a hash operation on the attribute feature vector corresponding to the target data to obtain a hash feature vector, which is then cached in the data warehouse.
[0124] Figure 2 This is a schematic diagram of the structure of a key-value pair-based data processing device according to this application. Figure 2 As shown, it includes:
[0125] Data acquisition unit 201 is used to acquire data to be processed;
[0126] The formatting unit 202 is used to format the data to be processed to generate key-value pairs, wherein the key in the key-value pair is the feature attribute name of the data to be processed, and the value in the key-value pair is the assignment of the feature attribute name;
[0127] Cache unit 203 is used to determine key-value pairs with the same key so as to write all key-value pairs with the same key into the same cache data pool created in real time;
[0128] The perturbation unit 204 is used to add noise perturbation to all key-value pairs locally in the cached data pool to generate pseudo-real data.
[0129] Optionally, the formatting unit is further configured to perform segmentation processing on the data to be processed based on determined data segmentation points to obtain a subset of data to be processed, so as to format each subset of data to be processed corresponding to the data to be processed to generate key-value pairs.
[0130] Optionally, the formatting unit is specifically used to perform quality assessment on the data to be processed to obtain a quality assessment value; and to determine data segmentation points based on the quality assessment value.
[0131] Optionally, the formatting unit is specifically used to calculate the information entropy of the data to be processed; and to obtain a quality assessment value based on the information entropy.
[0132] Optionally, the formatting unit is specifically used to evaluate the probability of the feature attribute name appearing in the data to be processed; and to calculate the information entropy of the data to be processed based on the evaluated probability.
[0133] Optionally, the formatting unit is specifically used to map the data to be processed into a real vector; and to inject the information entropy into the real vector to obtain a quality assessment value.
[0134] Optionally, the apparatus further includes: a slicing unit, used to slice the target data into several data blocks, and to determine the data to be processed in units of the data blocks.
[0135] Optionally, the apparatus further includes: a labeling unit, used to perform parallel attribute labeling processing on several data blocks corresponding to the target data, to obtain the feature attribute names and corresponding values of the data to be processed.
[0136] Optionally, the apparatus further includes: an extraction unit, configured to invoke a set replacement data sampling mechanism to extract sample data from the target data, to form a dataset to be segmented based on the extracted sample data, and to specifically target the dataset to be segmented during the segmentation process.
[0137] Optionally, the slicing unit is specifically used to slice the target data horizontally or vertically to obtain several data blocks, and the data to be processed is determined in units of the data blocks.
[0138] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Figure 3 As shown, it includes a memory and a processor. The memory stores an executable program, and when the processor runs the executable program, it performs the following steps:
[0139] Obtain the data to be processed;
[0140] The data to be processed is formatted to generate key-value pairs, wherein the key in the key-value pair is the feature attribute name of the data to be processed, and the value in the key-value pair is the assignment of the feature attribute name;
[0141] Identify key-value pairs with the same key, and write all key-value pairs with the same key into the same cached data pool created in real time;
[0142] Locally within the cached data pool, noise perturbations are added to all key-value pairs to generate pseudo-real data.
[0143] A computer storage medium storing a computer-executable program, which, when executed, performs the following steps:
[0144] Obtain the data to be processed;
[0145] The data to be processed is formatted to generate key-value pairs, wherein the key in the key-value pair is the feature attribute name of the data to be processed, and the value in the key-value pair is the assignment of the feature attribute name;
[0146] Identify key-value pairs with the same key, and write all key-value pairs with the same key into the same cached data pool created in real time;
[0147] Locally within the cached data pool, noise perturbations are added to all key-value pairs to generate pseudo-real data.
[0148] Figure 4 This is a schematic diagram of the hardware structure of the electronic device in the embodiments of this application; as shown Figure 4As shown, the hardware structure of this electronic device may include: Electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 402 or a computer program loaded from storage unit 406 into random access memory (RAM) 403. The RAM 403 may also store various programs and data required for the operation of device 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.
[0149] Multiple components in electronic device 400 are connected to I / O interface 405, including: input unit 406, output unit 407, storage unit 408, and communication unit 409. Input unit 406 can be any type of device capable of inputting information to electronic device 400. Input unit 406 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device. Output unit 407 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 404 may include, but is not limited to, disks and optical discs. Communication unit 409 allows electronic device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth™ devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.
[0150] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the various means and processes described above. For example, in some embodiments, the above steps can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 40*. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 400 via ROM 402 and / or communication unit 409. In some embodiments, the computing unit 401 can be configured to perform the above steps by any other suitable means (e.g., by means of firmware).
[0151] The electronic devices in this application embodiments exist in various forms, including but not limited to:
[0152] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include: smartphones (e.g., iPhones), multimedia phones, feature phones, and low-end phones, etc.
[0153] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as the iPad.
[0154] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players (such as iPods), handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.
[0155] (4) Server: A device that provides computing services. The components of a server include a processor 410, hard disk, memory, system bus, etc. Servers are similar to general computer architectures, but because they need to provide highly reliable services, they have higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.
[0156] (5) Other electronic devices with data interaction functions.
[0157] A computer program product, wherein the computer storage medium stores computer-executable instructions, which, when executed, perform the following steps:
[0158] Obtain the data to be processed;
[0159] The data to be processed is formatted to generate key-value pairs, wherein the key in the key-value pair is the feature attribute name of the data to be processed, and the value in the key-value pair is the assignment of the feature attribute name;
[0160] Identify key-value pairs with the same key, and write all key-value pairs with the same key into the same cached data pool created in real time;
[0161] Locally within the cached data pool, noise perturbations are added to all key-value pairs to generate pseudo-real data.
[0162] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the apparatus embodiments, the description is relatively simple, and relevant parts can be referred to the description of the apparatus embodiments. The device and system embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components indicated as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0163] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A data processing method based on key-value pairs, characterized in that, include: The process involves acquiring data to be processed, performing a quality assessment on the data to be processed to obtain a quality assessment value, determining data segmentation points based on the quality assessment value, and segmenting the data to be processed based on the determined data segmentation points to obtain a subset of the data to be processed. The step of performing a quality assessment on the data to be processed to obtain a quality assessment value includes: calculating the information entropy of the data to be processed; and obtaining the quality assessment value based on the information entropy. The data to be processed is formatted to generate key-value pairs, wherein the key in the key-value pair is the feature attribute name of the data to be processed, and the value in the key-value pair is the assignment of the feature attribute name; Identify key-value pairs with the same key, and write all key-value pairs with the same key into the same cached data pool created in real time; Locally within the cached data pool, noise perturbations are added to all key-value pairs to generate pseudo-real data.
2. The method according to claim 1, characterized in that, The calculation of the information entropy of the data to be processed includes: evaluating the probability of the feature attribute name appearing in the data to be processed; The information entropy of the data to be processed is calculated based on the assessed probability.
3. The method according to claim 1, characterized in that, The step of obtaining the quality assessment value based on the information entropy includes: mapping the data to be processed into a real vector; The information entropy is injected into the real vector to obtain a quality assessment value.
4. The method according to any one of claims 1-3, characterized in that, The method further includes: obtaining several data blocks included in the target dataset; Each data block is labeled with attributes to obtain the attribute feature vector corresponding to each data block; Based on the dimensions of the attribute feature vectors, construct the nodes of the decision tree; The similarity between attribute feature vectors is obtained by calculating the similarity between attribute feature vectors of different data blocks; Based on the similarity between the attribute feature vectors, the several data blocks are serialized and recombined to obtain several data subsets; Noise is added to the plurality of data subsets and matched with the nodes of the decision tree to generate a decision tree, and the data to be processed is obtained based on the decision tree.
5. A key-value pair-based data processing device, characterized in that, include: A data acquisition unit is configured to acquire data to be processed, perform quality assessment on the data to be processed to obtain a quality assessment value, determine data cutting points based on the quality assessment value, and perform data cutting processing on the data to be processed based on the determined data cutting points to obtain a subset of data to be processed; wherein, the step of performing quality assessment on the data to be processed to obtain a quality assessment value includes: calculating the information entropy of the data to be processed; and obtaining the quality assessment value based on the information entropy; A formatting unit is used to format the data to be processed to generate key-value pairs, wherein the key in the key-value pair is the feature attribute name of the data to be processed, and the value in the key-value pair is the assignment of the feature attribute name; A cache unit is used to identify key-value pairs with the same key so that all key-value pairs with the same key are written into the same cache data pool created in real time. A perturbation unit is used to locally add noise perturbation to all key-value pairs in the cached data pool to generate pseudo-real data.
6. An electronic device comprising a memory and a processor, wherein the memory stores an executable program, and the processor, when running the executable program, performs the following steps: acquiring data to be processed; performing a quality assessment on the data to be processed to obtain a quality assessment value; determining data segmentation points based on the quality assessment value; and segmenting the data to be processed based on the determined data segmentation points to obtain a subset of data to be processed; wherein, The step of performing a quality assessment on the data to be processed to obtain a quality assessment value includes: calculating the information entropy of the data to be processed; and obtaining the quality assessment value based on the information entropy. The data to be processed is formatted to generate key-value pairs, wherein the key in the key-value pair is the feature attribute name of the data to be processed, and the value in the key-value pair is the assignment of the feature attribute name; Identify key-value pairs with the same key, and write all key-value pairs with the same key into the same cached data pool created in real time; Locally within the cached data pool, noise perturbations are added to all key-value pairs to generate pseudo-real data.
7. A computer storage medium storing a computer-executable program, wherein when the computer-executable program is executed, the following steps are performed: acquiring data to be processed; performing a quality assessment on the data to be processed to obtain a quality assessment value; determining data segmentation points based on the quality assessment value; and segmenting the data to be processed based on the determined data segmentation points to obtain a subset of data to be processed; wherein... The step of performing a quality assessment on the data to be processed to obtain a quality assessment value includes: calculating the information entropy of the data to be processed; and obtaining the quality assessment value based on the information entropy. The data to be processed is formatted to generate key-value pairs, wherein the key in the key-value pair is the feature attribute name of the data to be processed, and the value in the key-value pair is the assignment of the feature attribute name; Identify key-value pairs with the same key, and write all key-value pairs with the same key into the same cached data pool created in real time; Locally within the cached data pool, noise perturbations are added to all key-value pairs to generate pseudo-real data.