Data purity calculation method and device
By formatting the data and generating histograms, the problem of not considering continuity and discreteness in data quality assessment is solved, and a more accurate data purity calculation is achieved.
Patent Information
- Application Number
- CN202310250221.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-14
- Publication Date
- 2025-09-23
- Estimated Expiration
- 2043-03-14
AI Technical Summary
Existing technologies fail to effectively consider the continuity and discreteness of data when evaluating data quality, resulting in low evaluation accuracy.
By formatting the data to be processed, generating continuous and discrete data sets, calculating the histograms of continuous and discrete attribute values, and calculating the purity of the data based on these histograms, the continuity and discreteness of the data are taken into consideration to improve the accuracy of the evaluation.
Improves the accuracy of data quality assessment by generating continuity and discreteness histograms to accurately calculate the purity of data.
Smart Images
Figure CN116415133B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of privacy computing technology, and in particular to a method and device for calculating data purity. Background Art
[0002] The rapid development of big data is gradually increasing the value of data. Data quality assessment is necessary during both storage and use. However, current data quality assessments fail to consider both the continuity and discreteness of the data, resulting in low accuracy. Summary of the Invention
[0003] Based on the above problems, the embodiments of the present application provide a method and device for calculating data purity.
[0004] The embodiments of this application disclose the following technical solutions:
[0005] A method for calculating data purity, comprising:
[0006] Format the data to be processed to obtain continuous and discrete data sets;
[0007] Determining a continuity attribute value of each continuous data block in the continuous data set and a discrete attribute value of each discrete data block in the discrete data set;
[0008] sorting all the continuity attribute values to generate a continuity histogram;
[0009] Generate a discrete histogram according to the discrete attribute values in the order of scanning;
[0010] The purity of the data to be processed is calculated according to the continuity histogram and the discreteness histogram.
[0011] Optionally, the method further comprises: performing attribute labeling processing on the data to be processed to obtain corresponding attribute feature descriptions;
[0012] According to the attribute feature description, the continuity and discreteness of the feature of the data to be processed are evaluated respectively to obtain a continuity evaluation value and a discreteness evaluation value;
[0013] Based on the continuity evaluation value and the discreteness evaluation value, the data to be processed is divided into a continuous data set and a discrete data set.
[0014] Optionally, the method further comprises: performing block processing on the data to be processed to obtain a plurality of data blocks;
[0015] The attribute labeling processing is performed on the data to be processed to obtain the corresponding attribute feature description, including: performing attribute labeling processing on a data block basis to obtain an attribute feature vector corresponding to each data block; and concatenating the attribute feature vectors corresponding to all data blocks to obtain the attribute feature description corresponding to the data to be processed.
[0016] Optionally, based on the attribute feature description, the characteristic continuity and discreteness of the data to be processed are evaluated separately to obtain a continuity evaluation value and a discreteness evaluation value, including: calculating the attention value between different attribute feature vectors in the attribute feature description; based on the attention value, the characteristic continuity and discreteness of the data to be processed are evaluated separately to obtain a continuity evaluation value and a discreteness evaluation value.
[0017] Optionally, the data to be processed is divided into a continuous data set and a discrete data set based on the continuity evaluation value and the discreteness evaluation value, including: filtering out data blocks whose step length between continuity evaluation values is less than a set continuity value threshold from the several data blocks to form the continuous data set; filtering out data blocks whose step length between continuity evaluation values is greater than or equal to the set continuity value threshold from the several data blocks to form the discrete data set.
[0018] Optionally, the method further includes: extracting sample data from the data set to be processed, and using the extracted sample data as the data to be processed.
[0019] Optionally, the method further includes: calling a set data sampling mechanism with replacement to extract sample data from the data set to be processed.
[0020] Optionally, the attribute labeling processing of the data to be processed to obtain a corresponding attribute feature description includes: obtaining a scheduling command issued by a control node in a distributed processing cluster; and calling a labeling node to perform attribute labeling processing on the data to be processed according to the scheduling command to obtain a corresponding attribute feature description.
[0021] Optionally, the block processing of the data to be processed to obtain a plurality of data blocks includes: based on the number of the marked nodes, performing data slicing processing on the data to be processed to obtain a plurality of data blocks, so that the number of the data blocks is equal to the number of the marked nodes.
[0022] Optionally, the data to be processed is divided into blocks to obtain several data blocks, including: based on the number of the annotation nodes and the data processing volume of a single annotation node, data segmentation processing is performed on the data to be processed to obtain several data blocks, so that the data volume of a single data block is equal to the data volume of a single annotation node.
[0023] Optionally, performing attribute labeling on the data to be processed to obtain a corresponding attribute feature description includes: performing attribute labeling on the data to be processed based on a preset data attribute feature set to obtain a corresponding attribute feature description.
[0024] Optionally, the attribute labeling processing is performed on the data to be processed based on a preset data attribute feature set to obtain a corresponding attribute feature description, including: according to regular matching, based on a preset data attribute feature set, the attribute labeling processing is performed on the data to be processed to obtain a corresponding attribute feature description.
[0025] Optionally, the attribute labeling processing is performed on the data to be processed based on a preset data attribute feature set to obtain a corresponding attribute feature description, including:
[0026] Based on a pre-set data attribute feature set, parallel attribute labeling processing is performed on the data to be processed, and attribute labeling values are assigned;
[0027] According to the associated annotation value, the attribute feature description corresponding to the data to be processed is obtained.
[0028] Optionally, performing parallel attribute labeling processing on the data to be processed and assigning attribute labeling values includes: performing parallel attribute labeling processing on a plurality of data blocks included in the data to be processed and assigning an attribute labeling value to each data block.
[0029] A data purity calculation device, comprising:
[0030] The formatting unit is used to format the data to be processed to obtain continuous data sets and discrete data sets;
[0031] an attribute value determining unit, configured to determine a continuity attribute value of each continuous data block in the continuous data set and a discrete attribute value of each discrete data block in the discrete data set;
[0032] a first histogram generating unit, configured to sort all the continuity attribute values to generate a continuity histogram;
[0033] a second histogram generating unit, configured to generate a discrete histogram according to the discrete attribute values in the order in which they are scanned;
[0034] The purity calculation unit is used to calculate the purity of the data to be processed according to the continuity histogram and the discrete histogram.
[0035] Optionally, the device further includes:
[0036] The labeling unit is used to label the data to be processed and obtain the corresponding attribute feature description;
[0037] An evaluation value calculation unit, configured to evaluate the continuity and discreteness of the characteristics of the data to be processed according to the attribute feature description, and obtain a continuity evaluation value and a discreteness evaluation value;
[0038] A data division unit is used to divide the data to be processed into a continuous data set and a discrete data set based on the continuity evaluation value and the discreteness evaluation value.
[0039] Optionally, the device further comprises: a block division unit, configured to perform block processing on the data to be processed to obtain a plurality of data blocks;
[0040] The labeling unit is specifically used to: perform attribute labeling processing on a data block basis to obtain an attribute feature vector corresponding to each data block; and perform splicing processing on the attribute feature vectors corresponding to all data blocks to obtain an attribute feature description corresponding to the data to be processed.
[0041] Optionally, the evaluation value calculation unit is specifically used to calculate the attention value between different attribute feature vectors in the attribute feature description; according to the attention value, the feature continuity and discreteness of the data to be processed are evaluated respectively to obtain a continuity evaluation value and a discreteness evaluation value.
[0042] Optionally, the data partitioning unit is used to filter out data blocks whose step length between continuity evaluation values is less than a set continuity value threshold from the several data blocks to form the continuous data set; and to filter out data blocks whose step length between continuity evaluation values is greater than or equal to the set continuity value threshold from the several data blocks to form the discrete data set.
[0043] Optionally, the device further includes a data extraction unit, configured to extract sample data from the data set to be processed, so as to use the extracted sample data as the data to be processed.
[0044] Optionally, the data extraction unit is further configured to call a set data sampling mechanism with replacement to extract sample data from the data set to be processed.
[0045] Optionally, the labeling unit is specifically configured to obtain a scheduling command issued by a control node in a distributed processing cluster; and according to the scheduling command, call a labeling node to perform attribute labeling on the data to be processed to obtain a corresponding attribute feature description.
[0046] Optionally, the blocking unit is specifically configured to perform data blocking processing on the data to be processed based on the number of the labeled nodes to obtain a plurality of data blocks, such that the number of the data blocks is equal to the number of the labeled nodes.
[0047] Optionally, the blocking unit is specifically used to perform data blocking processing on the data to be processed based on the number of the annotation nodes and the data processing volume of a single annotation node, to obtain a plurality of data blocks, so that the data volume of a single data block is equal to the data volume of a single annotation node.
[0048] Optionally, the labeling unit is specifically configured to perform attribute labeling processing on the data to be processed based on a preset data attribute feature set to obtain a corresponding attribute feature description.
[0049] Optionally, the labeling unit is specifically configured to perform attribute labeling processing on the data to be processed based on regular matching and a preset data attribute feature set to obtain corresponding attribute feature descriptions.
[0050] Optionally, the labeling unit is specifically configured to perform parallel attribute labeling processing on the data to be processed based on a preset data attribute feature set, and assign attribute labeling values; and obtain attribute feature descriptions corresponding to the data to be processed according to the corresponding labeling values.
[0051] Optionally, the labeling unit is specifically configured to perform parallel attribute labeling processing on a plurality of data blocks included in the data to be processed, and assign an attribute labeling value to each data block.
[0052] An electronic device includes a memory and a processor, wherein the memory stores an executable program, and the processor performs the following steps when running the executable program:
[0053] Format the data to be processed to obtain continuous and discrete data sets;
[0054] Determining a continuity attribute value of each continuous data block in the continuous data set and a discrete attribute value of each discrete data block in the discrete data set;
[0055] sorting all the continuity attribute values to generate a continuity histogram;
[0056] Generate a discrete histogram according to the discrete attribute values in the order of scanning;
[0057] The purity of the data to be processed is calculated according to the continuity histogram and the discreteness histogram.
[0058] A computer storage medium stores a computer executable program, wherein when the computer executable program is executed, the following steps are performed:
[0059] Format the data to be processed to obtain continuous and discrete data sets;
[0060] Determining a continuity attribute value of each continuous data block in the continuous data set and a discrete attribute value of each discrete data block in the discrete data set;
[0061] sorting all the continuity attribute values to generate a continuity histogram;
[0062] Generate a discrete histogram according to the discrete attribute values in the order of scanning;
[0063] The purity of the data to be processed is calculated according to the continuity histogram and the discreteness histogram.
[0064] A computer program product, wherein the computer storage medium stores computer-executable instructions, and when the computer-executable instructions are executed, the following steps are performed:
[0065] Format the data to be processed to obtain continuous and discrete data sets;
[0066] Determining a continuity attribute value of each continuous data block in the continuous data set and a discrete attribute value of each discrete data block in the discrete data set;
[0067] sorting all the continuity attribute values to generate a continuity histogram;
[0068] Generate a discrete histogram according to the discrete attribute values in the order of scanning;
[0069] The purity of the data to be processed is calculated according to the continuity histogram and the discreteness histogram.
[0070] In the solution provided in the embodiment of the present application, the data to be processed is formatted to obtain a continuous data set and a discrete data set; the continuity attribute value of each continuous data block in the continuous data set and the discrete attribute value of each discrete data block in the discrete data set are determined; all the continuity attribute values are sorted to generate a continuity histogram; a discrete histogram is generated according to the discrete attribute values in the order of scanning; the purity of the data to be processed is calculated based on the continuity histogram and the discrete histogram, taking into account the continuity and discreteness of the data, thereby improving the accuracy of quality assessment. BRIEF DESCRIPTION OF THE DRAWINGS
[0071] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0072] Figure 1 This is a flow chart of a method for filtering data noise according to an embodiment of the present application.
[0073] Figure 2 This is a structural schematic diagram of a data noise filtering device according to an embodiment of the present application.
[0074] Figure 3 This is a structural diagram of an electronic device according to an embodiment of the present application.
[0075] Figure 4 Schematic diagram of the hardware structure of the electronic device in the embodiment of the present application. DETAILED DESCRIPTION
[0076] The implementation of any technical solution in the embodiments of the present application does not necessarily require achieving all of the above advantages at the same time.
[0077] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.
[0078] Figure 1 This is a flow chart of a method for calculating data purity according to an embodiment of the present application. Figure 1 As shown, it includes:
[0079] S101, formatting the data to be processed to obtain a continuous data set and a discrete data set;
[0080] S102, determining a continuity attribute value of each continuous data block in the continuous data set and a discrete attribute value of each discrete data block in the discrete data set;
[0081] S103, sorting all the continuity attribute values to generate a continuity histogram;
[0082] S104, generating a discrete histogram according to the discrete attribute values in the order of scanning;
[0083] S104: Calculate the purity of the data to be processed according to the continuity histogram and the discreteness histogram.
[0084] In step S104, the purity of the data to be processed is calculated based on the distribution density of the continuity histogram and the discrete histogram. For example, a weighted average of the distribution density of the continuity histogram and the discrete histogram is taken to obtain the purity of the data to be processed.
[0085] Optionally, the method further comprises: performing attribute labeling processing on the data to be processed to obtain corresponding attribute feature descriptions;
[0086] According to the attribute feature description, the continuity and discreteness of the feature of the data to be processed are evaluated respectively to obtain a continuity evaluation value and a discreteness evaluation value;
[0087] Based on the continuity evaluation value and the discreteness evaluation value, the data to be processed is divided into a continuous data set and a discrete data set.
[0088] Optionally, the method further comprises: performing block processing on the data to be processed to obtain a plurality of data blocks;
[0089] The attribute labeling processing is performed on the data to be processed to obtain the corresponding attribute feature description, including: performing attribute labeling processing on a data block basis to obtain an attribute feature vector corresponding to each data block; and concatenating the attribute feature vectors corresponding to all data blocks to obtain the attribute feature description corresponding to the data to be processed.
[0090] Optionally, based on the attribute feature description, the characteristic continuity and discreteness of the data to be processed are evaluated separately to obtain a continuity evaluation value and a discreteness evaluation value, including: calculating the attention value between different attribute feature vectors in the attribute feature description; based on the attention value, the characteristic continuity and discreteness of the data to be processed are evaluated separately to obtain a continuity evaluation value and a discreteness evaluation value.
[0091] Optionally, the data to be processed is divided into a continuous data set and a discrete data set based on the continuity evaluation value and the discreteness evaluation value, including: filtering out data blocks whose step length between continuity evaluation values is less than a set continuity value threshold from the several data blocks to form the continuous data set; filtering out data blocks whose step length between continuity evaluation values is greater than or equal to the set continuity value threshold from the several data blocks to form the discrete data set.
[0092] Optionally, the method further includes: extracting sample data from the data set to be processed, and using the extracted sample data as the data to be processed.
[0093] Optionally, the method further includes: calling a set data sampling mechanism with replacement to extract sample data from the data set to be processed.
[0094] Optionally, the attribute labeling processing of the data to be processed to obtain a corresponding attribute feature description includes: obtaining a scheduling command issued by a control node in a distributed processing cluster; and calling a labeling node to perform attribute labeling processing on the data to be processed according to the scheduling command to obtain a corresponding attribute feature description.
[0095] Optionally, the block processing of the data to be processed to obtain a plurality of data blocks includes: based on the number of the marked nodes, performing data slicing processing on the data to be processed to obtain a plurality of data blocks, so that the number of the data blocks is equal to the number of the marked nodes.
[0096] Optionally, the data to be processed is divided into blocks to obtain several data blocks, including: based on the number of the annotation nodes and the data processing volume of a single annotation node, data segmentation processing is performed on the data to be processed to obtain several data blocks, so that the data volume of a single data block is equal to the data volume of a single annotation node.
[0097] Optionally, performing attribute labeling on the data to be processed to obtain a corresponding attribute feature description includes: performing attribute labeling on the data to be processed based on a preset data attribute feature set to obtain a corresponding attribute feature description.
[0098] Optionally, the attribute labeling processing is performed on the data to be processed based on a preset data attribute feature set to obtain a corresponding attribute feature description, including: according to regular matching, based on a preset data attribute feature set, the attribute labeling processing is performed on the data to be processed to obtain a corresponding attribute feature description.
[0099] Optionally, the attribute labeling processing is performed on the data to be processed based on a preset data attribute feature set to obtain a corresponding attribute feature description, including:
[0100] Based on a pre-set data attribute feature set, parallel attribute labeling processing is performed on the data to be processed, and attribute labeling values are assigned;
[0101] According to the associated annotation value, the attribute feature description corresponding to the data to be processed is obtained.
[0102] Optionally, performing parallel attribute labeling processing on the data to be processed and assigning attribute labeling values includes: performing parallel attribute labeling processing on a plurality of data blocks included in the data to be processed and assigning an attribute labeling value to each data block.
[0103] Figure 2 This is a schematic diagram of the structure of a data purity calculation device according to an embodiment of the present application. Figure 2 As shown, it includes:
[0104] A formatting unit 201 is used to format the data to be processed to obtain a continuous data set and a discrete data set;
[0105] An attribute value determining unit 202 is configured to determine a continuity attribute value of each continuous data block in the continuous data set and a discrete attribute value of each discrete data block in the discrete data set;
[0106] A first histogram generating unit 203 is configured to sort all the continuity attribute values to generate a continuity histogram;
[0107] A second histogram generating unit 204 is configured to generate a discrete histogram according to the discrete attribute values in the order in which they are scanned;
[0108] The purity calculation unit 205 is configured to calculate the purity of the data to be processed according to the continuity histogram and the discrete histogram.
[0109] Optionally, the device further includes:
[0110] The labeling unit is used to label the data to be processed and obtain the corresponding attribute feature description;
[0111] An evaluation value calculation unit, configured to evaluate the continuity and discreteness of the characteristics of the data to be processed according to the attribute feature description, and obtain a continuity evaluation value and a discreteness evaluation value;
[0112] A data division unit is used to divide the data to be processed into a continuous data set and a discrete data set based on the continuity evaluation value and the discreteness evaluation value.
[0113] Optionally, the device further comprises: a block division unit, configured to perform block processing on the data to be processed to obtain a plurality of data blocks;
[0114] The labeling unit is specifically used to: perform attribute labeling processing on a data block basis to obtain an attribute feature vector corresponding to each data block; and perform splicing processing on the attribute feature vectors corresponding to all data blocks to obtain an attribute feature description corresponding to the data to be processed.
[0115] Optionally, the evaluation value calculation unit is specifically used to calculate the attention value between different attribute feature vectors in the attribute feature description; according to the attention value, the feature continuity and discreteness of the data to be processed are evaluated respectively to obtain a continuity evaluation value and a discreteness evaluation value.
[0116] Optionally, the data partitioning unit is used to filter out data blocks whose step length between continuity evaluation values is less than a set continuity value threshold from the several data blocks to form the continuous data set; and to filter out data blocks whose step length between continuity evaluation values is greater than or equal to the set continuity value threshold from the several data blocks to form the discrete data set.
[0117] Optionally, the device further includes a data extraction unit, configured to extract sample data from the data set to be processed, so as to use the extracted sample data as the data to be processed.
[0118] Optionally, the data extraction unit is further configured to call a set data sampling mechanism with replacement to extract sample data from the data set to be processed.
[0119] Optionally, the labeling unit is specifically configured to obtain a scheduling command issued by a control node in a distributed processing cluster; and according to the scheduling command, call a labeling node to perform attribute labeling on the data to be processed to obtain a corresponding attribute feature description.
[0120] Optionally, the blocking unit is specifically configured to perform data blocking processing on the data to be processed based on the number of the labeled nodes to obtain a plurality of data blocks, such that the number of the data blocks is equal to the number of the labeled nodes.
[0121] Optionally, the blocking unit is specifically used to perform data blocking processing on the data to be processed based on the number of the annotation nodes and the data processing volume of a single annotation node, to obtain a plurality of data blocks, so that the data volume of a single data block is equal to the data volume of a single annotation node.
[0122] Optionally, the labeling unit is specifically configured to perform attribute labeling processing on the data to be processed based on a preset data attribute feature set to obtain a corresponding attribute feature description.
[0123] Optionally, the labeling unit is specifically configured to perform attribute labeling processing on the data to be processed based on regular matching and a preset data attribute feature set to obtain corresponding attribute feature descriptions.
[0124] Optionally, the labeling unit is specifically configured to perform parallel attribute labeling processing on the data to be processed based on a preset data attribute feature set, and assign attribute labeling values; and obtain attribute feature descriptions corresponding to the data to be processed according to the corresponding labeling values.
[0125] Optionally, the labeling unit is specifically configured to perform parallel attribute labeling processing on a plurality of data blocks included in the data to be processed, and assign an attribute labeling value to each data block.
[0126] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 3 As shown, it includes a memory and a processor, the memory stores an executable program, and the processor performs the following steps when running the executable program:
[0127] Format the data to be processed to obtain continuous and discrete data sets;
[0128] Determining a continuity attribute value of each continuous data block in the continuous data set and a discrete attribute value of each discrete data block in the discrete data set;
[0129] sorting all the continuity attribute values to generate a continuity histogram;
[0130] Generate a discrete histogram according to the discrete attribute values in the order of scanning;
[0131] The purity of the data to be processed is calculated according to the continuity histogram and the discreteness histogram.
[0132] Figure 4 Schematic diagram of the hardware structure of the electronic device in the embodiment of the present application; Figure 4 As shown, the hardware structure of the electronic device may include: the electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 406 into a random access memory (RAM) 403. Various programs and data required for the operation of the device 400 can also be stored in the RAM 403. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.
[0133] Multiple components within electronic device 400 are connected to I / O interface 405, including an input unit 406, an output unit 407, a storage unit 408, and a communication unit 409. Input unit 406 can be any type of device capable of inputting information into electronic device 400. Input unit 406 can receive input numeric or character information and generate key input signals related to user settings and / or function control of the electronic device. Output unit 407 can be any type of device capable of presenting information and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. Storage unit 404 may include, but is not limited to, a magnetic disk or an optical disk. Communication unit 409 allows electronic device 400 to exchange information / data with other devices via computer networks such as the Internet and / or various telecommunication networks and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.
[0134] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various devices and processes described above. For example, in some embodiments, the above steps may be implemented as a computer software program that is tangibly included in a machine-readable medium, such as a storage unit 40*. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 400 via the ROM 402 and / or the communication unit 409. In some embodiments, the computing unit 401 may be configured to perform the above steps in any other appropriate manner (e.g., by means of firmware).
[0135] The electronic devices of the embodiments of the present application exist in various forms, including but not limited to:
[0136] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and are primarily designed to provide voice and data communications. These terminals include smartphones (e.g., iPhones), multimedia phones, feature phones, and low-end phones.
[0137] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, have computing and processing capabilities, and generally also have mobile Internet access. These terminals include PDAs, MIDs, and UMPCs, such as the iPad.
[0138] (3) Portable entertainment devices: These devices can display and play multimedia content. These devices include audio and video players (such as iPods), handheld game consoles, e-books, smart toys, and portable car navigation devices.
[0139] (4) Server: A device that provides computing services. The server consists of a processor 410, a hard disk, memory, a system bus, etc. The server is similar to a general computer architecture, but because it needs to provide highly reliable services, it has higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.
[0140] (5) Other electronic devices with data interaction functions.
[0141] The present application also provides a computer storage medium, wherein the computer storage medium stores a computer executable program. When the computer executable program is executed, the following steps are performed:
[0142] Format the data to be processed to obtain continuous and discrete data sets;
[0143] Determining a continuity attribute value of each continuous data block in the continuous data set and a discrete attribute value of each discrete data block in the discrete data set;
[0144] sorting all the continuity attribute values to generate a continuity histogram;
[0145] Generate a discrete histogram according to the discrete attribute values in the order of scanning;
[0146] The purity of the data to be processed is calculated according to the continuity histogram and the discreteness histogram.
[0147] The present application also provides a computer program product, wherein the computer storage medium stores computer-executable instructions. When the computer-executable instructions are executed, the following steps are performed:
[0148] Format the data to be processed to obtain continuous and discrete data sets;
[0149] Determining a continuity attribute value of each continuous data block in the continuous data set and a discrete attribute value of each discrete data block in the discrete data set;
[0150] sorting all the continuity attribute values to generate a continuity histogram;
[0151] Generate a discrete histogram according to the discrete attribute values in the order of scanning;
[0152] The purity of the data to be processed is calculated according to the continuity histogram and the discreteness histogram.
[0153] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the apparatus embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the apparatus embodiments. The device and system embodiments described above are merely illustrative, wherein the modules described as separate components may or may not be physically separated, and the components indicated as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. A person of ordinary skill in the art can understand and implement them without expending creative work.
[0154] The above is merely one specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.
Claims
1. A method for calculating data purity, characterized in that: include: Format the data to be processed to obtain continuous and discrete data sets; Determining a continuity attribute value of each continuous data block in the continuous data set and a discrete attribute value of each discrete data block in the discrete data set; sorting all the continuity attribute values to generate a continuity histogram; Generate a discrete histogram according to the discrete attribute values in the order of scanning; Calculating the purity of the data to be processed according to the continuity histogram and the discreteness histogram; The method further includes: performing attribute labeling on the data to be processed to obtain a corresponding attribute feature description, including: obtaining a scheduling command issued by a control node in the distributed processing cluster; calling a labeling node to perform attribute labeling on the data to be processed according to the scheduling command to obtain a corresponding attribute feature description; The method further comprises: performing attribute labeling processing on the data to be processed to obtain corresponding attribute feature descriptions, and performing parallel attribute labeling processing on the data to be processed based on a predetermined data attribute feature set according to regular matching, and assigning attribute labeling values; Obtaining attribute feature descriptions corresponding to the data to be processed according to the associated annotation values; The method further comprises: performing block processing on the data to be processed to obtain a plurality of data blocks; The attribute labeling processing is performed on the data to be processed to obtain the corresponding attribute feature description, including: performing attribute labeling processing on a data block basis to obtain an attribute feature vector corresponding to each data block; and concatenating the attribute feature vectors corresponding to all data blocks to obtain the attribute feature description corresponding to the data to be processed.
2. The method according to claim 1, characterized in that The method further includes: performing attribute labeling processing on the data to be processed to obtain corresponding attribute feature descriptions; According to the attribute feature description, the continuity and discreteness of the feature of the data to be processed are evaluated respectively to obtain a continuity evaluation value and a discreteness evaluation value; Based on the continuity evaluation value and the discreteness evaluation value, the data to be processed is divided into a continuous data set and a discrete data set.
3. The method according to claim 2, characterized in that According to the attribute feature description, the characteristic continuity and discreteness of the data to be processed are evaluated respectively to obtain a continuity evaluation value and a discreteness evaluation value, including: calculating the attention value between different attribute feature vectors in the attribute feature description; according to the attention value, the characteristic continuity and discreteness of the data to be processed are evaluated respectively to obtain a continuity evaluation value and a discreteness evaluation value.
4. The method according to claim 2, characterized in that The method of dividing the data to be processed into a continuous data set and a discrete data set based on the continuity evaluation value and the discreteness evaluation value includes: filtering out data blocks whose step length between continuity evaluation values is less than a set continuity value threshold from the plurality of data blocks to form the continuous data set; and filtering out data blocks whose step length between continuity evaluation values is greater than or equal to the set continuity value threshold from the plurality of data blocks to form the discrete data set.
5. The method according to any one of claims 1 to 4, characterized in that The method further includes: extracting sample data from the data set to be processed, and using the extracted sample data as the data to be processed.
6. The method according to claim 5, characterized in that The method further includes: calling a set data sampling mechanism with replacement to extract sample data from the data set to be processed.
7. The method according to claim 1, characterized in that The block processing of the data to be processed to obtain a plurality of data blocks includes: based on the number of the marked nodes, performing data block processing on the data to be processed to obtain a plurality of data blocks, so that the number of the data blocks is equal to the number of the marked nodes.
8. A data purity calculation device, characterized in that: include: The formatting unit is used to format the data to be processed to obtain continuous data sets and discrete data sets; The attribute value determining unit is used to determine the continuity attribute value of each continuous data block in the continuous data set. and discrete attribute values of each discrete data block in the discrete data set; a first histogram generating unit, configured to sort all the continuity attribute values to generate a continuity histogram; a second histogram generating unit, configured to generate a discrete histogram according to the discrete attribute values in the order in which they are scanned; a purity calculation unit, configured to calculate the purity of the data to be processed based on the continuity histogram and the discrete histogram; The device further includes: performing attribute labeling processing on the data to be processed to obtain a corresponding attribute feature description, including: obtaining a scheduling command issued by a control node in the distributed processing cluster; calling a labeling node to perform attribute labeling processing on the data to be processed according to the scheduling command to obtain a corresponding attribute feature description; The method further comprises: performing attribute labeling processing on the data to be processed to obtain corresponding attribute feature descriptions, and performing parallel attribute labeling processing on the data to be processed based on a predetermined data attribute feature set according to regular matching, and assigning attribute labeling values; Obtaining attribute feature descriptions corresponding to the data to be processed according to the associated annotation values; The device further comprises: performing block processing on the data to be processed to obtain a plurality of data blocks; The attribute labeling processing is performed on the data to be processed to obtain the corresponding attribute feature description, including: performing attribute labeling processing on a data block basis to obtain an attribute feature vector corresponding to each data block; and concatenating the attribute feature vectors corresponding to all data blocks to obtain the attribute feature description corresponding to the data to be processed.
Citation Information
Patent Citations
Data classification method and device and electronic equipment
CN111178380A
Privacy protection method based on Hadoop platform under MapReduce environment
CN111737759A