Method, device and related product for filtering data noise

Abstract

The application discloses a data noise filtering method and device and related products. The method comprises the following steps: performing attribute annotation processing on target data to obtain corresponding attribute feature description; evaluating the feature continuity and discreteness of the target data according to the attribute feature description to obtain continuity evaluation value and discreteness evaluation value; dividing the target data into a continuity data set and a discreteness data set based on the continuity evaluation value and the discreteness evaluation value; determining first noise in the continuity data set according to a set first information entropy; determining second noise in the discreteness data set according to a set second information entropy; filtering the first noise from the continuity data set to obtain continuity net data, and filtering the second noise from the discreteness data set to obtain discreteness net data; and generating a net data set according to the continuity net data and the discreteness net data, so that more fine-grained denoising is realized, and the accuracy of noise processing is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of privacy computing, in particular to a data noise filtering method and device and related products. BACKGROUND

[0002] The rapid development of big data has gradually embodied the value of data. Data noise processing is required during data storage or use. However, when data noise processing is performed, the overall data is processed, resulting in a large processing granularity and low noise processing accuracy. SUMMARY

[0003] To solve the above problems, the embodiments of the present application provide a data noise filtering method, device and related products.

[0004] The embodiments of the present application disclose the following technical solutions:

[0005] A data noise filtering method comprises the following steps:

[0006] Attribute annotation processing is performed on target data to obtain corresponding attribute feature descriptions;

[0007] According to the attribute feature descriptions, the continuity and discreteness of the features of the target data are respectively evaluated to obtain continuity evaluation values and discreteness evaluation values;

[0008] Based on the continuity evaluation values and the discreteness evaluation values, the target data is divided into a continuity data set and a discreteness data set;

[0009] According to a set first information entropy, a first noise in the continuity data set is determined;

[0010] According to a set second information entropy, a second noise in the discreteness data set is determined;

[0011] The first noise is filtered from the continuity data set to obtain continuity clean data, and the second noise is filtered from the discreteness data set to obtain discreteness clean data;

[0012] According to the continuity clean data and the discreteness clean data, a clean data set is generated.

[0013] Optionally, the method further comprises block processing the target data to obtain a plurality of data blocks.

[0014] The attribute annotation processing of the target data obtains a corresponding attribute feature description, which comprises: performing attribute annotation processing on data blocks to obtain an attribute feature vector corresponding to each data block; and performing splicing processing on the attribute feature vectors corresponding to all data blocks to obtain the attribute feature description corresponding to the target data.

[0015] Optionally, the evaluation of the feature continuity and discreteness of the target data according to the attribute feature description obtains a continuity evaluation value and a discreteness evaluation value, which comprises: calculating attention values between different attribute feature vectors in the attribute feature description; and evaluating the feature continuity and discreteness of the target data according to the attention values to obtain the continuity evaluation value and the discreteness evaluation value.

[0016] Optionally, the division of the target data into a continuity data set and a discreteness data set based on the continuity evaluation value and the discreteness evaluation value comprises: screening data blocks with a step length between continuity evaluation values less than a set continuity value threshold from the data blocks to form the continuity data set; and screening data blocks with a step length between continuity evaluation values greater than or equal to the set continuity value threshold from the data blocks to form the discreteness data set.

[0017] Optionally, the determination of first noise in the continuity data set according to a set first information entropy comprises: calculating an information entropy of the continuity data set and comparing it with the set first information entropy to determine the first noise in the continuity data set.

[0018] Optionally, the determination of second noise in the discreteness data set according to a set second information entropy comprises: calculating an information entropy of the discreteness data set and comparing it with the set second information entropy to determine the second noise in the discreteness data set.

[0019] Optionally, the generation of a net data set according to the continuity net data and the discreteness net data comprises: fusing the continuity net data and the discreteness net data based on an attention matrix between the continuity net data and the discreteness net data.

[0020] Optionally, the method further comprises: extracting sample data from the target data set to use the extracted sample data as the target data.

[0021] Optionally, the method further comprises: calling a set replacement data sampling mechanism to extract sample data from the target data set.

[0022] Optionally, the attribute labeling processing on the target data to obtain the corresponding attribute feature description comprises: obtaining a scheduling command issued by a control node in a distributed processing cluster; and calling a labeling node to perform attribute labeling processing on the target data according to the scheduling command to obtain the corresponding attribute feature description.

[0023] Optionally, the block processing on the target data to obtain a plurality of data blocks comprises: performing data block processing on the target data based on the number of labeling nodes to obtain a plurality of data blocks, so that the number of data blocks is equal to the number of labeling nodes.

[0024] Optionally, the block processing on the target data to obtain a plurality of data blocks comprises: performing data block processing on the target data based on the number of labeling nodes and the data processing amount of a single labeling node to obtain a plurality of data blocks, so that the data amount of a single data block is equal to the data amount of a single labeling node.

[0025] Optionally, the attribute labeling processing on the target data to obtain the corresponding attribute feature description comprises: performing attribute labeling processing on the target data based on a pre-set data attribute feature set to obtain the corresponding attribute feature description.

[0026] Optionally, the attribute labeling processing on the target data to obtain the corresponding attribute feature description based on the pre-set data attribute feature set comprises: performing attribute labeling processing on the target data based on the pre-set data attribute feature set according to regular matching to obtain the corresponding attribute feature description.

[0027] Optionally, the attribute labeling processing on the target data to obtain the corresponding attribute feature description based on the pre-set data attribute feature set comprises:

[0028] performing parallel attribute labeling processing on the target data based on the pre-set data attribute feature set and assigning attribute labeling values;

[0029] obtaining the attribute feature description corresponding to the target data according to the attribute labeling values.

[0030] Optionally, the parallel attribute labeling processing on the target data and the assignment of attribute labeling values comprise: performing parallel attribute labeling processing on a plurality of data blocks included in the target data and assigning attribute labeling values to each data block.

[0031] A data noise filtering device comprises:

[0032] a labeling unit configured to perform attribute labeling processing on target data to obtain a corresponding attribute feature description;

[0033] an evaluation unit configured to evaluate the continuity and discreteness of the feature of the target data according to the attribute feature description, to obtain a continuity evaluation value and a discreteness evaluation value;

[0034] a division unit configured to divide the target data into a continuity data set and a discreteness data set based on the continuity evaluation value and the discreteness evaluation value;

[0035] a first noise determination unit configured to determine a first noise in the continuity data set according to a set first information entropy;

[0036] a second noise determination unit configured to determine a second noise in the discreteness data set according to a set second information entropy;

[0037] a first noise filtering unit configured to filter out the first noise from the continuity data set to obtain continuity net data;

[0038] a second noise filtering unit configured to filter out the second noise from the discreteness data set to obtain discreteness net data;

[0039] a net data generation unit configured to generate a net data set according to the continuity net data and the discreteness net data.

[0040] Optionally, the apparatus further comprises a chunking unit configured to perform chunking processing on the target data to obtain a plurality of data chunks.

[0041] The labeling unit is specifically configured to perform attribute labeling processing on the data chunks to obtain an attribute feature vector corresponding to each data chunk, and to perform splicing processing on the attribute feature vectors corresponding to all the data chunks to obtain the attribute feature description corresponding to the target data.

[0042] Optionally, the evaluation unit is specifically configured to calculate an attention value between different attribute feature vectors in the attribute feature description, and to evaluate the continuity and discreteness of the feature of the target data according to the attention value to obtain the continuity evaluation value and the discreteness evaluation value.

[0043] Optionally, the division unit is specifically configured to select, from the plurality of data chunks, data chunks with a step length between the continuity evaluation values being less than a set continuity value threshold to form the continuity data set, and to select, from the plurality of data chunks, data chunks with a step length between the continuity evaluation values being greater than or equal to the set continuity value threshold to form the discreteness data set.

[0044] Optionally, the first noise determination unit is specifically configured to calculate information entropy of the continuity data set, and compare the information entropy with a set first information entropy to determine the first noise in the continuity data set.

[0045] Optionally, the second noise determination unit is specifically configured to calculate information entropy of the discreteness data set, and compare the information entropy with a set second information entropy to determine the second noise in the discreteness data set.

[0046] Optionally, the net data generation unit is specifically configured to fuse the continuity net data and the discreteness net data based on an attention matrix between the continuity net data and the discreteness net data.

[0047] Optionally, the apparatus further comprises an extraction unit configured to extract sample data from a target data set, and use the extracted sample data as the target data.

[0048] Optionally, the extraction unit is further configured to call a set replacement data sampling mechanism to extract sample data from the target data set.

[0049] Optionally, the labeling unit is specifically configured to acquire a scheduling command issued by a control node in a distributed processing cluster, and call a labeling node to perform attribute labeling processing on target data according to the scheduling command to obtain corresponding attribute feature descriptions.

[0050] Optionally, the block unit is specifically configured to perform data block processing on the target data based on the number of labeling nodes to obtain a plurality of data blocks, so that the number of data blocks is equal to the number of labeling nodes.

[0051] Optionally, the block unit is specifically configured to perform data block processing on the target data based on the number of labeling nodes and the data processing amount of a single labeling node to obtain a plurality of data blocks, so that the data amount of a single data block is equal to the data amount of a single labeling node.

[0052] Optionally, the labeling unit is specifically configured to perform attribute labeling processing on target data based on a pre-set data attribute feature set to obtain corresponding attribute feature descriptions.

[0053] Optionally, the labeling unit is specifically configured to perform attribute labeling processing on target data based on a pre-set data attribute feature set according to regular matching to obtain corresponding attribute feature descriptions.

[0054] Optionally, the labeling unit is specifically configured to perform parallel attribute labeling processing on the target data based on a pre-set attribute feature set, and assign attribute labeling values; and obtain attribute feature descriptions corresponding to the target data according to the attribute labeling values.

[0055] Optionally, the labeling unit is specifically configured to perform parallel attribute labeling processing on a plurality of data blocks included in the target data, and assign attribute labeling values to each data block.

[0056] An electronic device includes a memory and a processor, the memory stores an executable program, and the processor executes the following steps when running the executable program:

[0057] Attribute labeling processing is performed on target data to obtain corresponding attribute feature descriptions;

[0058] According to the attribute feature descriptions, the feature continuity and discreteness of the target data are evaluated respectively to obtain continuity evaluation values and discreteness evaluation values;

[0059] Based on the continuity evaluation values and the discreteness evaluation values, the target data is divided into a continuity data set and a discreteness data set;

[0060] According to a set first information entropy, a first noise in the continuity data set is determined;

[0061] According to a set second information entropy, a second noise in the discreteness data set is determined;

[0062] The first noise is filtered out from the continuity data set to obtain continuity net data, and the second noise is filtered out from the discreteness data set to obtain discreteness net data;

[0063] According to the continuity net data and the discreteness net data, a net data set is generated.

[0064] A computer storage medium stores a computer executable program, and the computer executable program performs the following steps when executed:

[0065] Attribute labeling processing is performed on target data to obtain corresponding attribute feature descriptions;

[0066] According to the attribute feature descriptions, the feature continuity and discreteness of the target data are evaluated respectively to obtain continuity evaluation values and discreteness evaluation values;

[0067] Based on the continuity evaluation values and the discreteness evaluation values, the target data is divided into a continuity data set and a discreteness data set;

[0068] determine a first noise in the continuity data set according to a set first information entropy;

[0069] determine a second noise in the discreteness data set according to a set second information entropy;

[0070] filter the first noise from the continuity data set to obtain continuity net data, and filter the second noise from the discreteness data set to obtain discreteness net data;

[0071] generate a net data set according to the continuity net data and the discreteness net data.

[0072] A computer program product, the computer storage medium stores computer executable instructions, the computer executable instructions are executed to perform the following steps:

[0073] perform attribute annotation processing on target data to obtain corresponding attribute feature descriptions;

[0074] evaluate the continuity and discreteness of the target data according to the attribute feature descriptions to obtain continuity evaluation values and discreteness evaluation values;

[0075] divide the target data into a continuity data set and a discreteness data set based on the continuity evaluation values and the discreteness evaluation values;

[0076] determine a first noise in the continuity data set according to a set first information entropy;

[0077] determine a second noise in the discreteness data set according to a set second information entropy;

[0078] filter the first noise from the continuity data set to obtain continuity net data, and filter the second noise from the discreteness data set to obtain discreteness net data;

[0079] generate a net data set according to the continuity net data and the discreteness net data.

[0080] In the scheme provided by the embodiment of the application, attribute annotation processing is performed on target data to obtain corresponding attribute feature descriptions; the continuity and discreteness of features of the target data are respectively evaluated according to the attribute feature descriptions to obtain continuity evaluation values and discreteness evaluation values; the target data is divided into a continuity data set and a discreteness data set based on the continuity evaluation values and the discreteness evaluation values; first noise in the continuity data set is determined according to a set first information entropy; second noise in the discreteness data set is determined according to a set second information entropy; the first noise is filtered from the continuity data set to obtain continuity net data, and the second noise is filtered from the discreteness data set to obtain discreteness net data; and a net data set is generated according to the continuity net data and the discreteness net data, so that more fine-grained implementation of noise removal is realized, and the accuracy of noise processing is improved. BRIEF DESCRIPTION OF DRAWINGS

[0081] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0082] Figure 1 The flowchart of a data noise filtering method according to an embodiment of the present application.

[0083] Figure 2 The structural diagram of a data noise filtering device according to an embodiment of the present application.

[0084] Figure 3 The structural diagram of an electronic device according to an embodiment of the present application.

[0085] Figure 4 The hardware structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0086] Implementing any of the technical solutions of the embodiments of the present application does not necessarily need to achieve all the advantages above.

[0087] In order to make the person skilled in the art better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0088] Figure 1 This is a schematic flowchart illustrating a data noise filtering method according to an embodiment of this application. Figure 1 As shown, it includes:

[0089] S101. Perform attribute annotation processing on the target data to obtain the corresponding attribute feature descriptions;

[0090] S102. Based on the attribute feature description, evaluate the continuity and discreteness of the target data respectively to obtain the continuity evaluation value and the discreteness evaluation value.

[0091] S103. Based on the continuity evaluation value and the discrete evaluation value, the target data is divided into a continuous dataset and a discrete dataset.

[0092] S104. Determine the first noise in the continuous dataset based on the set first information entropy;

[0093] S105. Determine the second noise in the discrete dataset based on the set second information entropy;

[0094] S106. Filter out the first noise from the continuous dataset to obtain continuous net data, and filter out the second noise from the discrete dataset to obtain discrete net data;

[0095] S107. Generate a net data set based on the continuous net data and the discrete net data.

[0096] Optionally, the method further includes: dividing the target data into blocks to obtain several data blocks;

[0097] The step of performing attribute annotation processing on the target data to obtain the corresponding attribute feature description includes: performing attribute annotation processing on a data block basis to obtain the attribute feature vector corresponding to each data block; and concatenating the attribute feature vectors corresponding to all data blocks to obtain the attribute feature description corresponding to the target data.

[0098] Optionally, the step of evaluating the continuity and discreteness of the target data based on the attribute feature description to obtain a continuity evaluation value and a discreteness evaluation value includes: calculating the attention value between different attribute feature vectors in the attribute feature description; and evaluating the continuity and discreteness of the target data based on the attention value to obtain a continuity evaluation value and a discreteness evaluation value.

[0099] Optionally, dividing the target data into a continuous dataset and a discrete dataset based on the continuous evaluation value and the discrete evaluation value includes: selecting data blocks from the plurality of data blocks whose step size between continuous evaluation values ​​is less than a set continuous value threshold to form the continuous dataset; and selecting data blocks from the plurality of data blocks whose step size between continuous evaluation values ​​is greater than or equal to the set continuous value threshold to form the discrete dataset.

[0100] Optionally, determining the first noise in the continuous dataset based on the set first information entropy includes: calculating the information entropy of the continuous dataset and comparing it with the set first information entropy to determine the first noise in the continuous dataset.

[0101] Optionally, determining the second noise in the discrete dataset based on the set second information entropy includes: calculating the information entropy of the discrete dataset and comparing it with the set second information entropy to determine the second noise in the discrete dataset.

[0102] Optionally, generating a net data set based on the continuous net data and the discrete net data includes: fusing the continuous net data and the discrete net data based on the attention matrix between the continuous net data and the discrete net data.

[0103] Optionally, the method further includes: extracting sample data from the target dataset to use the extracted sample data as the target data.

[0104] Optionally, the method further includes: invoking a pre-defined data sampling mechanism with replacement to extract sample data from the target dataset.

[0105] Optionally, the step of performing attribute annotation processing on the target data to obtain the corresponding attribute feature description includes: obtaining a scheduling command issued by the control node in the distributed processing cluster; and, according to the scheduling command, calling the annotation node to perform attribute annotation processing on the target data to obtain the corresponding attribute feature description.

[0106] Optionally, the step of dividing the target data into blocks to obtain several data blocks includes: based on the number of labeled nodes, performing data slicing on the target data to obtain several data blocks, such that the number of data blocks is equal to the number of labeled nodes.

[0107] Optionally, the target data is divided into blocks to obtain several data blocks, including: based on the number of labeled nodes and the data processing volume of a single labeled node, the target data is divided into blocks to obtain several data blocks, such that the data volume of a single data block is equivalent to the data volume of a single labeled node.

[0108] Optionally, the step of performing attribute annotation processing on the target data to obtain the corresponding attribute feature description includes: performing attribute annotation processing on the target data based on a pre-defined set of data attribute features to obtain the corresponding attribute feature description.

[0109] Optionally, the step of performing attribute annotation processing on the target data based on a pre-defined set of data attribute features to obtain the corresponding attribute feature description includes: performing attribute annotation processing on the target data based on a pre-defined set of data attribute features according to regular expression matching to obtain the corresponding attribute feature description.

[0110] Optionally, the step of performing attribute annotation processing on the target data based on a pre-defined set of data attribute features to obtain corresponding attribute feature descriptions includes:

[0111] Based on a pre-defined set of data attribute features, parallel attribute annotation processing is performed on the target data, and attribute annotation values ​​are assigned.

[0112] Based on the corresponding annotation value, the attribute feature description corresponding to the target data is obtained.

[0113] Optionally, the step of performing parallel attribute annotation processing on the target data and assigning attribute annotation values ​​includes: performing parallel attribute annotation processing on several data blocks included in the target data and assigning attribute annotation values ​​to each data block.

[0114] Figure 2 This is a schematic diagram of a data noise filtering device according to an embodiment of this application. Figure 2 As shown, it includes:

[0115] The annotation unit 201 is used to perform attribute annotation processing on the target data to obtain the corresponding attribute feature description;

[0116] Evaluation unit 202 is used to evaluate the continuity and discreteness of the target data according to the attribute feature description, and obtain a continuity evaluation value and a discrete evaluation value.

[0117] The partitioning unit 203 is used to partition the target data into a continuous dataset and a discrete dataset based on the continuity evaluation value and the discrete evaluation value.

[0118] The first noise determination unit 204 is used to determine the first noise in the continuous dataset based on a set first information entropy.

[0119] The second noise determination unit 205 is used to determine the second noise in the discrete dataset based on a set second information entropy.

[0120] The first noise filtering unit 206 is used to filter out the first noise from the continuous dataset to obtain continuous net data;

[0121] The second noise filtering unit 207 is used to filter out the second noise from the discrete dataset to obtain discrete net data;

[0122] The net data generation unit 208 is used to generate a net data set based on the continuous net data and the discrete net data.

[0123] Optionally, the device further includes a slicing unit for dividing the target data into blocks to obtain several data blocks;

[0124] The annotation unit is specifically used to perform attribute annotation processing on a data block basis to obtain the attribute feature vector corresponding to each data block; and to concatenate the attribute feature vectors corresponding to all data blocks to obtain the attribute feature description corresponding to the target data.

[0125] Optionally, the evaluation unit is specifically used to calculate the attention value between different attribute feature vectors in the attribute feature description; based on the attention value, the feature continuity and discreteness of the target data are evaluated respectively to obtain a continuity evaluation value and a discreteness evaluation value.

[0126] Optionally, the partitioning unit is specifically used to select data blocks from the plurality of data blocks whose step size between continuous evaluation values ​​is less than a set continuous value threshold, so as to form the continuous dataset; and to select data blocks from the plurality of data blocks whose step size between continuous evaluation values ​​is greater than or equal to the set continuous value threshold, so as to form the discrete dataset.

[0127] Optionally, the first noise determination unit is specifically used to calculate the information entropy of the continuous dataset and compare it with a set first information entropy to determine the first noise in the continuous dataset.

[0128] Optionally, the second noise determination unit is specifically used to calculate the information entropy of the discrete dataset and compare it with a set second information entropy to determine the second noise in the discrete dataset.

[0129] Optionally, the net data generation unit is specifically used to fuse the continuous net data and the discrete net data based on the attention matrix between the continuous net data and the discrete net data.

[0130] Optionally, the apparatus further includes an extraction unit for extracting sample data from the target dataset, so as to use the extracted sample data as the target data.

[0131] Optionally, the extraction unit is further configured to invoke a pre-defined data sampling mechanism with replacement to extract sample data from the target dataset.

[0132] Optionally, the annotation unit is specifically used to obtain the scheduling command issued by the control node in the distributed processing cluster; according to the scheduling command, the annotation node is invoked to perform attribute annotation processing on the target data to obtain the corresponding attribute feature description.

[0133] Optionally, the segmentation unit is specifically used to perform data segmentation processing on the target data based on the number of labeled nodes, to obtain several data blocks, such that the number of data blocks is equal to the number of labeled nodes.

[0134] Optionally, the segmentation unit is specifically used to perform data segmentation processing on the target data based on the number of labeled nodes and the data processing volume of a single labeled node, to obtain several data blocks, such that the data volume of a single data block is equivalent to the data volume of a single labeled node.

[0135] Optionally, the annotation unit is specifically used to perform attribute annotation processing on the target data based on a pre-set set of data attribute features to obtain the corresponding attribute feature description.

[0136] Optionally, the annotation unit is specifically used to perform attribute annotation processing on the target data based on a pre-defined set of data attribute features according to regular expression matching, so as to obtain the corresponding attribute feature description.

[0137] Optionally, the annotation unit is specifically used to perform parallel attribute annotation processing on the target data based on a pre-set set of data attribute features, and assign attribute annotation values; and to obtain the attribute feature description corresponding to the target data according to the annotation values.

[0138] Optionally, the annotation unit is specifically used to perform parallel attribute annotation processing on several data blocks included in the target data, and to assign attribute annotation values ​​to each data block.

[0139] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Figure 3As shown, it includes a memory and a processor. The memory stores an executable program, and when the processor runs the executable program, it performs the following steps:

[0140] The target data is labeled with attributes to obtain the corresponding attribute feature descriptions;

[0141] Based on the attribute feature description, the continuity and discreteness of the target data are evaluated respectively to obtain continuity evaluation values ​​and discrete evaluation values;

[0142] Based on the continuity evaluation value and the discrete evaluation value, the target data is divided into a continuous dataset and a discrete dataset;

[0143] Based on the set first information entropy, determine the first noise in the continuous dataset;

[0144] Based on the established second information entropy, the second noise in the discrete dataset is determined;

[0145] Continuous net data is obtained by filtering out the first noise from the continuous dataset, and discrete net data is obtained by filtering out the second noise from the discrete dataset.

[0146] A net data set is generated based on the continuous net data and the discrete net data.

[0147] Figure 4 This is a schematic diagram of the hardware structure of the electronic device in the embodiments of this application; as shown Figure 4 As shown, the hardware structure of this electronic device may include: Electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 402 or a computer program loaded from storage unit 406 into random access memory (RAM) 403. The RAM 403 may also store various programs and data required for the operation of device 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0148] Multiple components in electronic device 400 are connected to I / O interface 405, including: input unit 406, output unit 407, storage unit 408, and communication unit 409. Input unit 406 can be any type of device capable of inputting information to electronic device 400. Input unit 406 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device. Output unit 407 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 404 may include, but is not limited to, disks and optical discs. Communication unit 409 allows electronic device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth™ devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0149] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the various means and processes described above. For example, in some embodiments, the above steps can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 40*. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 400 via ROM 402 and / or communication unit 409. In some embodiments, the computing unit 401 can be configured to perform the above steps by any other suitable means (e.g., by means of firmware).

[0150] The electronic devices in this application embodiments exist in various forms, including but not limited to:

[0151] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include: smartphones (e.g., iPhones), multimedia phones, feature phones, and low-end phones, etc.

[0152] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as the iPad.

[0153] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players (such as iPods), handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0154] (4) Server: A device that provides computing services. The components of a server include a processor 410, hard disk, memory, system bus, etc. Servers are similar to general computer architectures, but because they need to provide highly reliable services, they have higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0155] (5) Other electronic devices with data interaction functions.

[0156] This application also provides a computer storage medium storing a computer-executable program, which, when executed, performs the following steps:

[0157] The target data is labeled with attributes to obtain the corresponding attribute feature descriptions;

[0158] Based on the attribute feature description, the continuity and discreteness of the target data are evaluated respectively to obtain continuity evaluation values ​​and discrete evaluation values;

[0159] Based on the continuity evaluation value and the discrete evaluation value, the target data is divided into a continuous dataset and a discrete dataset;

[0160] Based on the set first information entropy, determine the first noise in the continuous dataset;

[0161] Based on the established second information entropy, the second noise in the discrete dataset is determined;

[0162] Continuous net data is obtained by filtering out the first noise from the continuous dataset, and discrete net data is obtained by filtering out the second noise from the discrete dataset.

[0163] A net data set is generated based on the continuous net data and the discrete net data.

[0164] This application also provides a computer program product, wherein the computer storage medium stores computer-executable instructions, and when the computer-executable instructions are executed, the following steps are performed:

[0165] The target data is labeled with attributes to obtain the corresponding attribute feature descriptions;

[0166] Based on the attribute feature description, the continuity and discreteness of the target data are evaluated respectively to obtain continuity evaluation values ​​and discrete evaluation values;

[0167] Based on the continuity evaluation value and the discrete evaluation value, the target data is divided into a continuous dataset and a discrete dataset;

[0168] Based on the set first information entropy, determine the first noise in the continuous dataset;

[0169] Based on the established second information entropy, the second noise in the discrete dataset is determined;

[0170] Continuous net data is obtained by filtering out the first noise from the continuous dataset, and discrete net data is obtained by filtering out the second noise from the discrete dataset.

[0171] A net data set is generated based on the continuous net data and the discrete net data.

[0172] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the apparatus embodiments, the description is relatively simple, and relevant parts can be referred to the description of the apparatus embodiments. The device and system embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components indicated as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0173] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method of filtering data noise, characterized by, The method comprises the following steps: calling a set of back-to-back data sampling mechanisms to sample data from a target data set, and taking the sampled data as target data; performing attribute labeling processing on the target data to obtain corresponding attribute feature descriptions, including: obtaining a scheduling command issued by a control node in a distributed processing cluster; according to the scheduling command, calling a labeling node to perform attribute labeling processing on the target data to obtain corresponding attribute feature descriptions; performing block processing on the target data to obtain a plurality of data blocks, including: based on the number of labeling nodes, performing data block processing on the target data to obtain a plurality of data blocks, so that the number of data blocks is equal to the number of labeling nodes; the attribute labeling processing on the target data to obtain the corresponding attribute feature descriptions further comprises: performing attribute labeling processing on the data blocks to obtain attribute feature vectors corresponding to each data block; and performing splicing processing on the attribute feature vectors corresponding to all data blocks to obtain the attribute feature descriptions corresponding to the target data; According to the attribute feature descriptions, the feature continuity and discreteness of the target data are evaluated respectively to obtain continuity evaluation values and discreteness evaluation values, including: calculating the attention values between different attribute feature vectors in the attribute feature descriptions; according to the attention values, the feature continuity and discreteness of the target data are evaluated respectively to obtain continuity evaluation values and discreteness evaluation values; Based on the continuity evaluation values and the discreteness evaluation values, the target data is divided into a continuity data set and a discreteness data set, including: filtering out data blocks with a step length between continuity evaluation values less than a set continuity threshold from the plurality of data blocks to form the continuity data set; filtering out data blocks with a step length between continuity evaluation values greater than or equal to the set continuity threshold from the plurality of data blocks to form the discreteness data set; According to a set of first information entropy, determine the first noise in the continuity data set; According to a set of second information entropy, determine the second noise in the discreteness data set; Filtering out the first noise from the continuity data set to obtain continuity net data, and filtering out the second noise from the discreteness data set to obtain discreteness net data; According to the continuity net data and the discreteness net data, generate a net data set, including: based on the attention matrix between the continuity net data and the discreteness net data, fuse the continuity net data and the discreteness net data.

2. The method of claim 1, wherein, The method comprises the following steps:

3. The method of claim 1, wherein, According to a set of first information entropy, determine the first noise in the continuity data set, including: calculating the information entropy of the continuity data set, and comparing it with the set of first information entropy to determine the first noise in the continuity data set.

4. A device for filtering out data noise, characterized by The method comprises the following steps: According to a set of second information entropy, determine the second noise in the discreteness data set, including: calculating the information entropy of the discreteness data set, and comparing it with the set of second information entropy to determine the second noise in the discreteness data set. The method comprises the following steps: The extraction unit is configured to extract sample data from a target data set, so as to take the extracted sample data as the target data; The labeling unit is configured to perform attribute labeling processing on the target data to obtain a corresponding attribute feature description, including: obtaining a scheduling command issued by a control node in a distributed processing cluster; and performing attribute labeling processing on the target data by calling a labeling node according to the scheduling command to obtain a corresponding attribute feature description; The target data is processed in blocks to obtain a plurality of data blocks, including: performing data block processing on the target data based on the number of labeling nodes to obtain a plurality of data blocks, so that the number of data blocks is equal to the number of labeling nodes; and performing attribute labeling processing on the target data to obtain a corresponding attribute feature description, further including: performing attribute labeling processing on the data blocks to obtain an attribute feature vector corresponding to each data block; and performing splicing processing on the attribute feature vectors corresponding to all data blocks to obtain an attribute feature description corresponding to the target data; The evaluation unit is configured to evaluate the feature continuity and discreteness of the target data according to the attribute feature description to obtain a continuity evaluation value and a discreteness evaluation value, including: calculating attention values between different attribute feature vectors in the attribute feature description; and evaluating the feature continuity and discreteness of the target data according to the attention values to obtain a continuity evaluation value and a discreteness evaluation value; The division unit is configured to divide the target data into a continuity data set and a discreteness data set based on the continuity evaluation value and the discreteness evaluation value, including: screening data blocks with a step length between continuity evaluation values less than a set continuity threshold from the plurality of data blocks to form the continuity data set; and screening data blocks with a step length between continuity evaluation values greater than or equal to the set continuity threshold from the plurality of data blocks to form the discreteness data set; The first noise determination unit is configured to determine a first noise in the continuity data set according to a set first information entropy; The second noise determination unit is configured to determine a second noise in the discreteness data set according to a set second information entropy The third noise filtering unit is configured to filter out the first noise from the continuity data set to obtain continuity net data; The fourth noise filtering unit is configured to filter out the second noise from the discreteness data set to obtain discreteness net data; The net data generation unit is configured to generate a net data set according to the continuity net data and the discreteness net data, including: fusing the continuity net data and the discreteness net data based on an attention matrix between the continuity net data and the discreteness net data. It includes a memory and a processor, the memory stores an executable program, and the processor executes the executable program to perform the following steps:

5. An electronic device, comprising: A set of data sampling mechanism is called to extract sample data from a target data set, and the extracted sample data is taken as the target data; ​ The target data is subjected to attribute labeling processing to obtain a corresponding attribute feature description, including: obtaining a scheduling command issued by a control node in a distributed processing cluster; and calling a labeling node to perform attribute labeling processing on the target data according to the scheduling command to obtain a corresponding attribute feature description; The target data is subjected to block processing to obtain a plurality of data blocks, including: performing data block processing on the target data based on the number of labeling nodes to obtain a plurality of data blocks, so that the number of data blocks is equal to the number of labeling nodes; and performing attribute labeling processing on the target data in units of data blocks to obtain an attribute feature vector corresponding to each data block, and performing splicing processing on the attribute feature vectors corresponding to all data blocks to obtain an attribute feature description corresponding to the target data; The feature continuity and discreteness of the target data are respectively evaluated according to the attribute feature description to obtain a continuity evaluation value and a discreteness evaluation value, including: calculating attention values between different attribute feature vectors in the attribute feature description; and evaluating the feature continuity and discreteness of the target data according to the attention values to obtain a continuity evaluation value and a discreteness evaluation value; The target data is divided into a continuity data set and a discreteness data set based on the continuity evaluation value and the discreteness evaluation value, including: selecting data blocks with a step length between continuity evaluation values less than a set continuity threshold from the plurality of data blocks to form the continuity data set; and selecting data blocks with a step length between continuity evaluation values greater than or equal to the set continuity threshold from the plurality of data blocks to form the discreteness data set; A first noise in the continuity data set is determined according to a set first information entropy; A second noise in the discreteness data set is determined according to a set second information entropy; The first noise is filtered out from the continuity data set to obtain continuity net data, and the second noise is filtered out from the discreteness data set to obtain discreteness net data; A net data set is generated based on the continuity net data and the discreteness net data, including: performing fusion on the continuity net data and the discreteness net data based on an attention matrix between the continuity net data and the discreteness net data.

6. A computer storage medium, characterized in that The computer storage medium stores a computer executable program, and the computer executable program is executed to perform the following steps: A set replacement data sampling mechanism is called to extract sample data from a target data set, and the extracted sample data is used as target data; The target data is subjected to attribute labeling processing to obtain a corresponding attribute feature description, including: obtaining a scheduling command issued by a control node in a distributed processing cluster; and calling a labeling node to perform attribute labeling processing on the target data according to the scheduling command to obtain a corresponding attribute feature description; The target data is processed in blocks to obtain a plurality of data blocks, including: based on the number of labeling nodes, performing data block processing on the target data to obtain a plurality of data blocks, so that the number of data blocks is equal to the number of labeling nodes; the attribute labeling processing of the target data is performed to obtain the corresponding attribute feature description, and further comprising: performing attribute labeling processing in units of data blocks to obtain the attribute feature vector corresponding to each data block; the attribute feature vectors corresponding to all data blocks are spliced to obtain the attribute feature description corresponding to the target data; According to the attribute feature description, the feature continuity and discreteness of the target data are respectively evaluated to obtain continuity evaluation values and discreteness evaluation values, including: calculating the attention value between different attribute feature vectors in the attribute feature description; according to the attention value, the feature continuity and discreteness of the target data are respectively evaluated to obtain continuity evaluation values and discreteness evaluation values; Based on the continuity evaluation values and the discreteness evaluation values, the target data is divided into a continuity data set and a discreteness data set, including: filtering out data blocks with a step length between continuity evaluation values less than a set continuity threshold from the plurality of data blocks to form the continuity data set; filtering out data blocks with a step length between continuity evaluation values greater than or equal to the set continuity threshold from the plurality of data blocks to form the discreteness data set; According to a set first information entropy, a first noise in the continuity data set is determined; According to a set second information entropy, a second noise in the discreteness data set is determined; The first noise is filtered out from the continuity data set to obtain continuity net data, and the second noise is filtered out from the discreteness data set to obtain discreteness net data; According to the continuity net data and the discreteness net data, a net data set is generated, including: based on an attention matrix between the continuity net data and the discreteness net data, the continuity net data and the discreteness net data are fused.

7. A computer program product, characterised in that, The computer storage medium stores computer executable instructions, and the computer executable instructions are executed to perform the following steps: A set replacement data sampling mechanism is called to extract sample data from a target data set, and the extracted sample data is used as target data; The target data is processed in blocks to obtain a plurality of data blocks, including: based on the number of labeling nodes, performing data block processing on the target data to obtain a plurality of data blocks, so that the number of data blocks is equal to the number of labeling nodes; the attribute labeling processing of the target data is performed to obtain the corresponding attribute feature description, and further comprising: performing attribute labeling processing in units of data blocks to obtain the attribute feature vector corresponding to each data block; the attribute feature vectors corresponding to all data blocks are spliced to obtain the attribute feature description corresponding to the target data; According to the attribute feature description, the feature continuity and discreteness of the target data are respectively evaluated to obtain continuity evaluation values and discreteness evaluation values, including: calculating the attention value between different attribute feature vectors in the attribute feature description; according to the attention value, the feature continuity and discreteness of the target data are respectively evaluated to obtain continuity evaluation values and discreteness evaluation values; Based on the continuity evaluation values and the discreteness evaluation values, the target data is divided into a continuity data set and a discreteness data set, including: filtering out data blocks with a step length between continuity evaluation values less than a set continuity threshold from the plurality of data blocks to form the continuity data set; filtering out data blocks with a step length between continuity evaluation values greater than or equal to the set continuity threshold from the plurality of data blocks to form the discreteness data set; According to a set first information entropy, a first noise in the continuity data set is determined; According to a set second information entropy, a second noise in the discreteness data set is determined; The first noise is filtered out from the continuity data set to obtain continuity net data, and the second noise is filtered out from the discreteness data set to obtain discreteness net data; According to the continuity net data and the discreteness net data, a net data set is generated, including: based on an attention matrix between the continuity net data and the discreteness net data, the continuity net data and the discreteness net data are fused. The target data is processed in blocks to obtain a plurality of data blocks, including: based on the number of labeled nodes, the target data is processed in data blocks to obtain a plurality of data blocks, so that the number of data blocks is equal to the number of labeled nodes; the attribute labeling processing of the target data is performed to obtain the corresponding attribute feature description, and further comprising: performing attribute labeling processing in data blocks to obtain the attribute feature vector corresponding to each data block; the attribute feature vectors corresponding to all data blocks are spliced to obtain the attribute feature description corresponding to the target data; According to the attribute feature description, the feature continuity and discreteness of the target data are evaluated respectively to obtain continuity evaluation value and discreteness evaluation value, including: calculating the attention value between different attribute feature vectors in the attribute feature description; according to the attention value, the feature continuity and discreteness of the target data are evaluated respectively to obtain the continuity evaluation value and the discreteness evaluation value; Based on the continuity evaluation value and the discreteness evaluation value, the target data is divided into continuity data set and discreteness data set, including: from the plurality of data blocks, the data blocks with a step length between the continuity evaluation values less than a set continuous value threshold are selected to form the continuity data set; from the plurality of data blocks, the data blocks with a step length between the continuity evaluation values greater than or equal to the set continuous value threshold are selected to form the discreteness data set; According to the set first information entropy, the first noise in the continuity data set is determined; According to the set second information entropy, the second noise in the discreteness data set is determined; The first noise is filtered out from the continuity data set to obtain the continuity net data, and the second noise is filtered out from the discreteness data set to obtain the discreteness net data; According to the continuity net data and the discreteness net data, a net data set is generated, including: based on the attention matrix between the continuity net data and the discreteness net data, the continuity net data and the discreteness net data are fused.

Images