Abnormal data processing method, device, equipment and storage medium

Abstract

The present invention relates to data processing, and provides a method, device, equipment and storage medium for processing abnormal data. The method can obtain data to be analyzed; identify the abnormal type to which the data to be analyzed belongs; if the abnormal type is a preset type, discretize the data to be analyzed to obtain discrete data; divide the discrete data according to the data frequency of the discrete data to obtain multiple sequence groups; search for the target nearest neighbor group of each sequence group; locate the data inflection point of the data to be analyzed according to each sequence group and the target nearest neighbor group; identify the abnormal data in the data to be analyzed according to the data inflection point; remove the abnormal data from the data to be analyzed to obtain the target data. The present invention can improve the quality of the target data. In addition, the present invention also relates to blockchain technology, and the target data can be stored in the blockchain.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an abnormal data processing method, device, equipment and storage medium. Background Art

[0002] In the medical platform, with the explosion of medical data, dirty data is also increasing. In the process of medical data analysis, dirty data will affect the data analysis results, thereby causing inaccurate data analysis results. Therefore, how to filter dirty data plays an important role in the data analysis process.

[0003] In the current dirty data filtering methods, binning, cluster analysis or regression analysis are usually used to remove dirty data. However, the calculation methods of these methods are complicated, resulting in the inability to quickly remove dirty data. In addition, these methods cannot accurately determine dirty data, resulting in low accuracy of data filtering. Summary of the invention

[0004] In view of the above, it is necessary to provide an abnormal data processing method, device, equipment and storage medium, which can improve the detection accuracy of abnormal data, thereby improving the data quality of target data.

[0005] In one aspect, the present invention provides a method for processing abnormal data, the method comprising:

[0006] Obtain the data to be analyzed;

[0007] Identify the abnormal type to which the data to be analyzed belongs;

[0008] If the abnormality type is a preset type, discretize the data to be analyzed to obtain discrete data;

[0009] Divide the discrete data according to the data frequency of the discrete data to obtain a plurality of sequence groups;

[0010] Search for the target nearest neighbor grouping for each sequence grouping;

[0011] Locating the data inflection point of the data to be analyzed according to each sequence grouping and the target nearest neighbor grouping;

[0012] Identifying abnormal data in the data to be analyzed according to the data inflection point;

[0013] The abnormal data is removed from the data to be analyzed to obtain target data.

[0014] According to a preferred embodiment of the present invention, identifying the abnormal type of the data to be analyzed includes:

[0015] Counting the total amount of the data to be analyzed;

[0016] Determine a first position serial number and a second position serial number in the data to be analyzed based on a first preset value, a second preset value and the total amount of data;

[0017] Sort the data to be analyzed in ascending order to obtain a data sequence and a sequence number of the data to be analyzed in the data sequence;

[0018] Calculating a first quantile based on the data to be analyzed, the sequence number, and the first position number, and calculating a second quantile based on the data to be analyzed, the sequence number, and the second position number;

[0019] Calculate the difference between the first quantile and the second quantile to obtain a data threshold;

[0020] Determine a target interval according to the first quantile, the second quantile and the data threshold;

[0021] If any data in the data to be analyzed is not in the target interval, the abnormal type is determined to be the preset type.

[0022] According to a preferred embodiment of the present invention, the discretization of the data to be analyzed to obtain discrete data includes:

[0023] Locating the data field corresponding to the data to be analyzed according to the data source of the data to be analyzed;

[0024] Acquire a plurality of preset ranges in the data domain;

[0025] The data to be analyzed is discretized into scores corresponding to the multiple preset ranges to obtain the discretization results.

[0026] According to a preferred embodiment of the present invention, dividing the discrete data according to the data frequency of the discrete data to obtain a plurality of sequence groups comprises:

[0027] Sorting the discrete data according to the data frequency to obtain a discrete sequence;

[0028] Generating a plurality of frequency intervals according to the data frequency and a preset frequency distance;

[0029] The discrete sequence is divided according to the multiple frequency intervals and the data frequency to obtain the multiple sequence groups.

[0030] According to a preferred embodiment of the present invention, locating the data inflection point of the data to be analyzed according to each sequence group and the target nearest neighbor group includes:

[0031] Obtaining the maximum frequency and the minimum frequency of each sequence grouping from the data frequency;

[0032] Calculating a ratio of a difference between the maximum frequency and the minimum frequency in the preset frequency distance to obtain a grouping slope of each sequence grouping;

[0033] Calculate the slope difference between each sequence group and the target nearest neighbor group based on the group slope;

[0034] A sequence group corresponding to a slope difference value having the largest value is selected from the slope differences as the data inflection point.

[0035] According to a preferred embodiment of the present invention, identifying abnormal data in the data to be analyzed according to the data inflection point includes:

[0036] Determine the target nearest neighbor group corresponding to the data inflection point as a feature group;

[0037] Extracting, from the discrete sequence, a sequence group whose data frequency is less than the characteristic group as a target group;

[0038] The data to be analyzed corresponding to the target group is determined as the abnormal data.

[0039] According to a preferred embodiment of the present invention, searching for the target nearest neighbor group of each sequence group includes:

[0040] determining a discrete position of each sequence grouping in the discrete sequence;

[0041] Determining a search direction according to an arrangement direction of the plurality of sequence groups in the discrete sequence;

[0042] Traversing the plurality of sequence groups in the discrete sequence in the search direction based on the discrete position to obtain a plurality of search groups;

[0043] The search group having the smallest position distance to the discrete position is determined as the target nearest neighbor group.

[0044] On the other hand, the present invention further provides an abnormal data processing device, the abnormal data processing device comprising:

[0045] An acquisition unit, used for acquiring data to be analyzed;

[0046] An identification unit, used to identify the abnormal type to which the data to be analyzed belongs;

[0047] A processing unit, configured to discretize the data to be analyzed to obtain discrete data if the abnormality type is a preset type;

[0048] A division unit, configured to divide the discrete data according to the data frequency of the discrete data to obtain a plurality of sequence groups;

[0049] A search unit, used for searching a target nearest neighbor group of each sequence group;

[0050] A positioning unit, used for positioning a data inflection point of the data to be analyzed according to each sequence group and the target nearest neighbor group;

[0051] The identification unit is further used to identify abnormal data in the data to be analyzed according to the data inflection point;

[0052] The elimination unit is used to eliminate the abnormal data from the data to be analyzed to obtain target data.

[0053] On the other hand, the present invention further provides an electronic device, comprising:

[0054] a memory storing computer readable instructions; and

[0055] A processor executes the computer-readable instructions stored in the memory to implement the abnormal data processing method.

[0056] On the other hand, the present invention further proposes a computer-readable storage medium, in which computer-readable instructions are stored. The computer-readable instructions are executed by a processor in an electronic device to implement the abnormal data processing method.

[0057] It can be seen from the above technical solutions that the present invention can avoid data analysis and processing on data samples that do not contain abnormal data by detecting the abnormal type of the data to be analyzed, thereby improving the processing efficiency of abnormal data. Furthermore, when the abnormal type is the preset type, the data to be analyzed is discretized to improve the efficiency of determining the abnormal data. The data inflection point can be accurately located through the relationship between each sequence grouping and the target nearest neighbor grouping, which can improve the detection accuracy of the abnormal data, thereby improving the quality of the target data, which is conducive to improving the subsequent analysis accuracy of the target data. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a flow chart of a preferred embodiment of the abnormal data processing method of the present invention.

[0059] Figure 2 It is a functional module diagram of a preferred embodiment of the abnormal data processing device of the present invention.

[0060] Figure 3 It is a structural schematic diagram of an electronic device of a preferred embodiment of the abnormal data processing method of the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0062] like Figure 1 FIG. 1 is a flow chart of a preferred embodiment of the abnormal data processing method of the present invention. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted.

[0063] The abnormal data processing method can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0064] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0065] The abnormal data processing method is applied to one or more electronic devices, which are devices that can automatically perform numerical calculations and / or information processing according to pre-set or stored computer-readable instructions, and whose hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0066] The electronic device may be any electronic product that can perform human-computer interaction with a user, such as a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), a game console, an interactive network television (IPTV), a smart wearable device, etc.

[0067] The electronic device may include a network device and / or a user device. The network device includes, but is not limited to, a single network electronic device, an electronic device group consisting of multiple network electronic devices, or a cloud consisting of a large number of hosts or network electronic devices based on cloud computing.

[0068] The network where the electronic device is located includes, but is not limited to: the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (VPN), etc.

[0069] S10, obtaining data to be analyzed.

[0070] In at least one embodiment of the present invention, the data to be analyzed refers to sample data that needs to be analyzed. For example, the data to be analyzed may be sample data for training any model.

[0071] The data to be analyzed may be any medical data, for example, the data to be analyzed may be personal health records, prescriptions, examination reports and other data.

[0072] In at least one embodiment of the present invention, the electronic device obtains the data to be analyzed from a preset storage path.

[0073] S11, identifying the abnormal type to which the data to be analyzed belongs.

[0074] In at least one embodiment of the present invention, the abnormality type refers to the type of the data to be analyzed, and the abnormality type includes a type with abnormal points and a type without abnormal points.

[0075] In at least one embodiment of the present invention, the electronic device identifying the abnormal type of the data to be analyzed includes:

[0076] Counting the total amount of the data to be analyzed;

[0077] Determine a first position serial number and a second position serial number in the data to be analyzed based on a first preset value, a second preset value and the total amount of data;

[0078] Sort the data to be analyzed in ascending order to obtain a data sequence and a sequence number of the data to be analyzed in the data sequence;

[0079] Calculating a first quantile based on the data to be analyzed, the sequence number, and the first position number, and calculating a second quantile based on the data to be analyzed, the sequence number, and the second position number;

[0080] Calculate the difference between the first quantile and the second quantile to obtain a data threshold;

[0081] Determine a target interval according to the first quantile, the second quantile and the data threshold;

[0082] If any data in the data to be analyzed is not in the target interval, the abnormal type is determined to be the preset type.

[0083] Among them, it has been proved through experiments that when the first preset value is usually set to 3 and the second preset value is usually set to 1, the accuracy of determining the abnormality type is higher.

[0084] The target interval refers to a data interval constructed with the difference between the second quantile and the preset multiple of the data threshold as the left interval and the sum of the second quantile and the preset multiple of the data threshold as the right interval.

[0085] Through the above implementation, the type of the data to be analyzed can be preliminarily determined, which is beneficial for subsequent processing of the data to be analyzed.

[0086] Specifically, the electronic device determines the first position serial number and the second position serial number based on the first preset value, the second preset value and the total amount of data, including:

[0087] Calculate the sum of the total amount of data and the configuration value to obtain a calculated sum;

[0088] Calculating the sum of the first preset value and the second preset value to obtain a preset sum;

[0089] Calculating the product of the first preset value and the operational sum to obtain a first operational product, and calculating the ratio of the first operational product to the preset sum to obtain the first position sequence number;

[0090] The product of the second preset value and the operational sum is calculated to obtain a second operational product, and the ratio of the second operational product to the preset sum is calculated to obtain the second position sequence number.

[0091] The configuration value is usually set to 1.

[0092] For example, the data sequence is 1, 3, 8, 11, 21, 54, the total data amount is 6, the first preset value is 3, the second preset value is 1, the configuration value is 1, after calculation, the operation sum is 7, the preset sum is 4, the first position number is: 3*7 / 4=5.25, and the second position number is 1*7 / 4=1.75.

[0093] Specifically, the electronic device calculates the first quantile based on the data to be analyzed, the sequence number and the first position number, including:

[0094] The first position sequence number is rounded to obtain a first rounded sequence number, and the sum of the first position sequence number and the configuration value is rounded to obtain a second rounded sequence number;

[0095] Calculate the difference between the first position sequence number and the first integer sequence number to obtain a first weight, and calculate the difference between the second integer sequence number and the first position sequence number to obtain a second weight;

[0096] Extracting data at the first rounding sequence number from the data sequence as first data, and extracting data at the second rounding sequence number from the data sequence as second data;

[0097] A weighted sum operation is performed on the first data and the second data based on the first weight and the second weight to obtain the first quantile.

[0098] Continuing with the above example, the first position number is 5.25. After processing, the first rounded number is 5, the second rounded number is 6, the first weight is 0.25, the second weight is 0.75, and the first quantile is: 21*0.25+54*0.75=45.75.

[0099] Specifically, the way in which the electronic device calculates the second quantile based on the data to be analyzed, the serial number and the second position number is the same as the way in which the electronic device calculates the first quantile based on the data to be analyzed, the serial number and the first position number, and the present invention will not go into details.

[0100] S12: If the abnormality type is a preset type, discretize the data to be analyzed to obtain discrete data.

[0101] In at least one embodiment of the present invention, the preset type refers to a type having an outlier point, that is, the data to be analyzed contains outlier point data.

[0102] The discrete data refers to data whose values ​​can only be calculated using natural numbers or integer units.

[0103] In at least one embodiment of the present invention, the electronic device discretizes the data to be analyzed to obtain discrete data, including:

[0104] Locating the data field corresponding to the data to be analyzed according to the data source of the data to be analyzed;

[0105] Acquire a plurality of preset ranges in the data domain;

[0106] The data to be analyzed is discretized into scores corresponding to the multiple preset ranges to obtain the discretization results.

[0107] The data source refers to the path location where the data to be analyzed is stored.

[0108] The data domain refers to a domain for data analysis based on the data to be analyzed. For example, the data domain may be a personnel capability rating.

[0109] By discretizing the data to be analyzed, it is convenient to map the data to be analyzed to a benchmark score, thereby reducing the amount of the data to be analyzed and improving the efficiency of locating subsequent abnormal data.

[0110] S13, dividing the discrete data according to the data frequency of the discrete data to obtain a plurality of sequence groups.

[0111] In at least one embodiment of the present invention, the data frequency refers to the number of discrete data corresponding to each discrete data in the data to be analyzed.

[0112] The difference between the maximum data frequency and the minimum data frequency in each sequence grouping is equal.

[0113] In at least one embodiment of the present invention, the electronic device divides the discrete data according to the data frequency of the discrete data to obtain a plurality of sequence groups including:

[0114] Sorting the discrete data according to the data frequency to obtain a discrete sequence;

[0115] Generating a plurality of frequency intervals according to the data frequency and a preset frequency distance;

[0116] The discrete sequence is divided according to the multiple frequency intervals and the data frequency to obtain the multiple sequence groups.

[0117] The preset frequency distance is generated by adjusting the result of outlier processing on historical sample data.

[0118] The frequencies in each frequency interval are equidistant, and the interval distance of each frequency interval is the preset frequency distance.

[0119] By sorting and then segmenting the discrete data, the disordered discrete data can be sorted according to the size of the data frequency, so that the discrete data can be directly divided according to the multiple frequency intervals later, thereby improving the efficiency of dividing the discrete data.

[0120] S14, searching for the target nearest neighbor group of each sequence group.

[0121] In at least one embodiment of the present invention, the multiple nearest neighbor groups refer to the sequence groups that are most adjacent to each sequence group along the search direction.

[0122] In at least one embodiment of the present invention, the electronic device searches for a target nearest neighbor group of each sequence group, including:

[0123] determining a discrete position of each sequence grouping in the discrete sequence;

[0124] Determining a search direction according to an arrangement direction of the plurality of sequence groups in the discrete sequence;

[0125] Traversing the plurality of sequence groups in the discrete sequence in the search direction based on the discrete position to obtain a plurality of search groups;

[0126] The search group having the smallest position distance to the discrete position is determined as the target nearest neighbor group.

[0127] The arrangement direction may include a horizontal direction or a vertical direction.

[0128] The search direction refers to a direction corresponding to the arrangement direction. For example, if the arrangement direction is a horizontal direction, the search direction may be a horizontal rightward direction; if the arrangement direction is a vertical direction, the search direction may be a vertical downward direction.

[0129] The position distance refers to the distance between the position of each search packet in the discrete sequence and the discrete position.

[0130] The search direction can be accurately determined by the arrangement direction, and then the target nearest neighbor group can be accurately located from the multiple sequence groups according to the search direction.

[0131] S15, locating the data inflection point of the data to be analyzed according to each sequence group and the target nearest neighbor group.

[0132] In at least one embodiment of the present invention, the data inflection point refers to a sequence grouping with the largest slope difference.

[0133] In at least one embodiment of the present invention, the electronic device locates the data inflection point of the data to be analyzed according to each sequence group and the target nearest neighbor group, including:

[0134] Obtaining the maximum frequency and the minimum frequency of each sequence grouping from the data frequency;

[0135] Calculating a ratio of a difference between the maximum frequency and the minimum frequency in the preset frequency distance to obtain a grouping slope of each sequence grouping;

[0136] Calculate the slope difference between each sequence group and the target nearest neighbor group based on the group slope;

[0137] A sequence group corresponding to a slope difference value having the largest value is selected from the slope differences as the data inflection point.

[0138] The maximum frequency refers to the data frequency with the largest value in each sequence group, and the minimum frequency refers to the data frequency with the smallest value in each sequence group.

[0139] By determining the maximum frequency and the minimum frequency, the group slope corresponding to each sequence group can be quickly generated, and then the data inflection point can be quickly determined according to the slope.

[0140] S16: Identify abnormal data in the data to be analyzed according to the data inflection point.

[0141] In at least one embodiment of the present invention, the abnormal data refers to the data to be analyzed corresponding to the discrete data whose data frequency is less than the maximum frequency in the feature group.

[0142] In at least one embodiment of the present invention, the electronic device identifying abnormal data in the data to be analyzed according to the data inflection point includes:

[0143] Determine the target nearest neighbor group corresponding to the data inflection point as a feature group;

[0144] Extracting, from the discrete sequence, a sequence group whose data frequency is less than the characteristic group as a target group;

[0145] The data to be analyzed corresponding to the target group is determined as the abnormal data.

[0146] The discrete sequence can quickly identify the target group whose data frequency is less than the characteristic group, thereby improving the efficiency of determining the abnormal data.

[0147] S17, removing the abnormal data from the data to be analyzed to obtain target data.

[0148] In at least one embodiment of the present invention, the target data refers to data obtained after filtering the abnormal data from the data to be analyzed.

[0149] It should be emphasized that in order to further ensure the privacy and security of the above target data, the above target data can also be stored in a node of a blockchain.

[0150] In this embodiment, the abnormal data is managed by filtering data with less frequency, so that the quality of the target data can be improved, so that a more stable and efficient model can be trained and generated, which is beneficial to data analysis.

[0151] In at least one embodiment of the present invention, after removing the abnormal data from the data to be analyzed to obtain the target data, the method further includes:

[0152] storing the target data;

[0153] Generate prompt information according to the storage path of the target data and the data identifier;

[0154] Encrypt the prompt information using symmetric encryption technology to obtain a ciphertext;

[0155] The ciphertext is sent to the terminal device of the designated contact.

[0156] It can be seen from the above technical solutions that the present invention can avoid data analysis and processing on data samples that do not contain abnormal data by detecting the abnormal type of the data to be analyzed, thereby improving the processing efficiency of abnormal data. Furthermore, when the abnormal type is the preset type, the data to be analyzed is discretized to improve the efficiency of determining the abnormal data. The data inflection point can be accurately located through the relationship between each sequence grouping and the target nearest neighbor grouping, which can improve the detection accuracy of the abnormal data, thereby improving the quality of the target data, which is conducive to improving the subsequent analysis accuracy of the target data.

[0157] like Figure 2 As shown, it is a functional module diagram of a preferred embodiment of the abnormal data processing device of the present invention. The abnormal data processing device 11 includes an acquisition unit 110, an identification unit 111, a processing unit 112, a division unit 113, a search unit 114, a positioning unit 115, a removal unit 116, a storage unit 117, a generation unit 118, an encryption unit 119 and a sending unit 120. The module / unit referred to in the present invention refers to a series of computer-readable instruction segments that can be acquired by the processor 13 and can complete a fixed function, which is stored in the memory 12. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0158] The acquisition unit 110 acquires the data to be analyzed.

[0159] In at least one embodiment of the present invention, the data to be analyzed refers to sample data that needs to be analyzed. For example, the data to be analyzed may be sample data for training any model.

[0160] The data to be analyzed may be any medical data, for example, the data to be analyzed may be personal health records, prescriptions, examination reports and other data.

[0161] In at least one embodiment of the present invention, the acquisition unit 110 acquires the data to be analyzed from a preset storage path.

[0162] The identification unit 111 identifies the abnormal type to which the data to be analyzed belongs.

[0163] In at least one embodiment of the present invention, the abnormality type refers to the type of the data to be analyzed, and the abnormality type includes a type with abnormal points and a type without abnormal points.

[0164] In at least one embodiment of the present invention, the identifying unit 111 identifies the abnormal type of the data to be analyzed, including:

[0165] Counting the total amount of the data to be analyzed;

[0166] Determine a first position serial number and a second position serial number in the data to be analyzed based on a first preset value, a second preset value and the total amount of data;

[0167] Sort the data to be analyzed in ascending order to obtain a data sequence and a sequence number of the data to be analyzed in the data sequence;

[0168] Calculating a first quantile based on the data to be analyzed, the sequence number, and the first position number, and calculating a second quantile based on the data to be analyzed, the sequence number, and the second position number;

[0169] Calculate the difference between the first quantile and the second quantile to obtain a data threshold;

[0170] Determine a target interval according to the first quantile, the second quantile and the data threshold;

[0171] If any data in the data to be analyzed is not in the target interval, the abnormal type is determined to be the preset type.

[0172] Among them, it has been proved through experiments that when the first preset value is usually set to 3 and the second preset value is usually set to 1, the accuracy of determining the abnormality type is higher.

[0173] The target interval refers to a data interval constructed with the difference between the second quantile and the preset multiple of the data threshold as the left interval and the sum of the second quantile and the preset multiple of the data threshold as the right interval.

[0174] Through the above implementation, the type of the data to be analyzed can be preliminarily determined, which is beneficial for subsequent processing of the data to be analyzed.

[0175] Specifically, the identification unit 111 determines the first position serial number and the second position serial number based on the first preset value, the second preset value and the total amount of data, including:

[0176] Calculate the sum of the total amount of data and the configuration value to obtain a calculated sum;

[0177] Calculating the sum of the first preset value and the second preset value to obtain a preset sum;

[0178] Calculating the product of the first preset value and the operational sum to obtain a first operational product, and calculating the ratio of the first operational product to the preset sum to obtain the first position sequence number;

[0179] The product of the second preset value and the operational sum is calculated to obtain a second operational product, and the ratio of the second operational product to the preset sum is calculated to obtain the second position sequence number.

[0180] The configuration value is usually set to 1.

[0181] For example, the data sequence is 1, 3, 8, 11, 21, 54, the total data amount is 6, the first preset value is 3, the second preset value is 1, the configuration value is 1, after calculation, the operation sum is 7, the preset sum is 4, the first position number is: 3*7 / 4=5.25, and the second position number is 1*7 / 4=1.75.

[0182] Specifically, the identification unit 111 calculates the first quantile based on the data to be analyzed, the sequence number and the first position number, including:

[0183] The first position sequence number is rounded to obtain a first rounded sequence number, and the sum of the first position sequence number and the configuration value is rounded to obtain a second rounded sequence number;

[0184] Calculate the difference between the first position sequence number and the first integer sequence number to obtain a first weight, and calculate the difference between the second integer sequence number and the first position sequence number to obtain a second weight;

[0185] Extracting data at the first rounding sequence number from the data sequence as first data, and extracting data at the second rounding sequence number from the data sequence as second data;

[0186] A weighted sum operation is performed on the first data and the second data based on the first weight and the second weight to obtain the first quantile.

[0187] Continuing with the above example, the first position number is 5.25. After processing, the first rounded number is 5, the second rounded number is 6, the first weight is 0.25, the second weight is 0.75, and the first quantile is: 21*0.25+54*0.75=45.75.

[0188] Specifically, the way in which the identification unit 111 calculates the second quantile based on the data to be analyzed, the sequence number and the second position number is the same as the way in which the identification unit 111 calculates the first quantile based on the data to be analyzed, the sequence number and the first position number, and the present invention will not go into details.

[0189] If the abnormality type is a preset type, the processing unit 112 discretizes the data to be analyzed to obtain discrete data.

[0190] In at least one embodiment of the present invention, the preset type refers to a type having an outlier point, that is, the data to be analyzed contains outlier point data.

[0191] The discrete data refers to data whose values ​​can only be calculated using natural numbers or integer units.

[0192] In at least one embodiment of the present invention, the processing unit 112 discretizes the data to be analyzed to obtain discrete data including:

[0193] Locating the data field corresponding to the data to be analyzed according to the data source of the data to be analyzed;

[0194] Acquire a plurality of preset ranges in the data domain;

[0195] The data to be analyzed is discretized into scores corresponding to the multiple preset ranges to obtain the discretization results.

[0196] The data source refers to the path location where the data to be analyzed is stored.

[0197] The data domain refers to a domain for data analysis based on the data to be analyzed. For example, the data domain may be a personnel capability rating.

[0198] By discretizing the data to be analyzed, it is convenient to map the data to be analyzed to a benchmark score, thereby reducing the amount of the data to be analyzed and improving the efficiency of locating subsequent abnormal data.

[0199] The division unit 113 divides the discrete data according to the data frequency of the discrete data to obtain a plurality of sequence groups.

[0200] In at least one embodiment of the present invention, the data frequency refers to the number of discrete data corresponding to each discrete data in the data to be analyzed.

[0201] The difference between the maximum data frequency and the minimum data frequency in each sequence grouping is equal.

[0202] In at least one embodiment of the present invention, the dividing unit 113 divides the discrete data according to the data frequency of the discrete data to obtain a plurality of sequence groups including:

[0203] Sorting the discrete data according to the data frequency to obtain a discrete sequence;

[0204] Generating a plurality of frequency intervals according to the data frequency and a preset frequency distance;

[0205] The discrete sequence is divided according to the multiple frequency intervals and the data frequency to obtain the multiple sequence groups.

[0206] The preset frequency distance is generated by adjusting the result of outlier processing on historical sample data.

[0207] The frequencies in each frequency interval are equidistant, and the interval distance of each frequency interval is the preset frequency distance.

[0208] By sorting and then segmenting the discrete data, the disordered discrete data can be sorted according to the size of the data frequency, so that the discrete data can be directly divided according to the multiple frequency intervals later, thereby improving the efficiency of dividing the discrete data.

[0209] The searching unit 114 searches for a target nearest neighbor group of each sequence group.

[0210] In at least one embodiment of the present invention, the multiple nearest neighbor groups refer to the sequence groups that are most adjacent to each sequence group along the search direction.

[0211] In at least one embodiment of the present invention, the searching unit 114 searches for the target nearest neighbor group of each sequence group including:

[0212] determining a discrete position of each sequence grouping in the discrete sequence;

[0213] Determining a search direction according to an arrangement direction of the plurality of sequence groups in the discrete sequence;

[0214] Traversing the plurality of sequence groups in the discrete sequence in the search direction based on the discrete position to obtain a plurality of search groups;

[0215] The search group having the smallest position distance to the discrete position is determined as the target nearest neighbor group.

[0216] The arrangement direction may include a horizontal direction or a vertical direction.

[0217] The search direction refers to a direction corresponding to the arrangement direction. For example, if the arrangement direction is a horizontal direction, the search direction may be a horizontal rightward direction; if the arrangement direction is a vertical direction, the search direction may be a vertical downward direction.

[0218] The position distance refers to the distance between the position of each search packet in the discrete sequence and the discrete position.

[0219] The search direction can be accurately determined by the arrangement direction, and then the target nearest neighbor group can be accurately located from the multiple sequence groups according to the search direction.

[0220] The positioning unit 115 locates the data inflection point of the data to be analyzed according to each sequence group and the target nearest neighbor group.

[0221] In at least one embodiment of the present invention, the data inflection point refers to a sequence grouping with the largest slope difference.

[0222] In at least one embodiment of the present invention, the positioning unit 115 positions the data inflection point of the to-be-analyzed data according to each sequence group and the target nearest neighbor group, including:

[0223] Obtaining the maximum frequency and the minimum frequency of each sequence grouping from the data frequency;

[0224] Calculating a ratio of a difference between the maximum frequency and the minimum frequency in the preset frequency distance to obtain a grouping slope of each sequence grouping;

[0225] Calculate the slope difference between each sequence group and the target nearest neighbor group based on the group slope;

[0226] A sequence group corresponding to a slope difference value having the largest value is selected from the slope differences as the data inflection point.

[0227] The maximum frequency refers to the data frequency with the largest value in each sequence group, and the minimum frequency refers to the data frequency with the smallest value in each sequence group.

[0228] By determining the maximum frequency and the minimum frequency, the group slope corresponding to each sequence group can be quickly generated, and then the data inflection point can be quickly determined according to the slope.

[0229] The identification unit 111 identifies abnormal data in the data to be analyzed according to the data inflection point.

[0230] In at least one embodiment of the present invention, the abnormal data refers to the data to be analyzed corresponding to the discrete data whose data frequency is less than the maximum frequency in the feature group.

[0231] In at least one embodiment of the present invention, the identifying unit 111 identifies abnormal data in the data to be analyzed according to the data inflection point, including:

[0232] Determine the target nearest neighbor group corresponding to the data inflection point as a feature group;

[0233] Extracting, from the discrete sequence, a sequence group whose data frequency is less than the characteristic group as a target group;

[0234] The data to be analyzed corresponding to the target group is determined as the abnormal data.

[0235] The discrete sequence can quickly identify the target group whose data frequency is less than the characteristic group, thereby improving the efficiency of determining the abnormal data.

[0236] The elimination unit 116 eliminates the abnormal data from the data to be analyzed to obtain target data.

[0237] In at least one embodiment of the present invention, the target data refers to data obtained after filtering the abnormal data from the data to be analyzed.

[0238] It should be emphasized that in order to further ensure the privacy and security of the above target data, the above target data can also be stored in a node of a blockchain.

[0239] In this embodiment, the abnormal data is managed by filtering data with less frequency, so that the quality of the target data can be improved, so that a more stable and efficient model can be trained and generated, which is beneficial to data analysis.

[0240] In at least one embodiment of the present invention, after removing the abnormal data from the data to be analyzed to obtain the target data, the method further includes:

[0241] The storage unit 117 stores the target data;

[0242] The generating unit 118 generates prompt information according to the storage path of the target data and the data identifier;

[0243] The encryption unit 119 uses a symmetric encryption technology to encrypt the prompt information to obtain a ciphertext;

[0244] The sending unit 120 sends the ciphertext to the terminal device of the designated contact.

[0245] It can be seen from the above technical solutions that the present invention can avoid data analysis and processing on data samples that do not contain abnormal data by detecting the abnormal type of the data to be analyzed, thereby improving the processing efficiency of abnormal data. Furthermore, when the abnormal type is the preset type, the data to be analyzed is discretized to improve the efficiency of determining the abnormal data. The data inflection point can be accurately located through the relationship between each sequence grouping and the target nearest neighbor grouping, which can improve the detection accuracy of the abnormal data, thereby improving the quality of the target data, which is conducive to improving the subsequent analysis accuracy of the target data.

[0246] like Figure 3 FIG. 1 is a schematic diagram of the structure of an electronic device of a preferred embodiment of the abnormal data processing method of the present invention.

[0247] In one embodiment of the present invention, the electronic device 1 includes, but is not limited to, a memory 12, a processor 13, and computer-readable instructions stored in the memory 12 and executable on the processor 13, such as an abnormal data processing program.

[0248] Those skilled in the art will understand that the schematic diagram is merely an example of the electronic device 1 and does not constitute a limitation on the electronic device 1. The electronic device 1 may include more or fewer components than shown in the diagram, or a combination of certain components, or different components. For example, the electronic device 1 may also include input and output devices, network access devices, buses, etc.

[0249] The processor 13 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 13 is the computing core and control center of the electronic device 1, and uses various interfaces and lines to connect various parts of the entire electronic device 1, and executes the operating system of the electronic device 1 and various installed applications, program codes, etc.

[0250] Exemplarily, the computer-readable instructions may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of completing specific functions, which are used to describe the execution process of the computer-readable instructions in the electronic device 1. For example, the computer-readable instructions may be divided into an acquisition unit 110, an identification unit 111, a processing unit 112, a division unit 113, a search unit 114, a positioning unit 115, a removal unit 116, a storage unit 117, a generation unit 118, an encryption unit 119, and a sending unit 120.

[0251] The memory 12 can be used to store the computer-readable instructions and / or modules, and the processor 13 implements various functions of the electronic device 1 by running or executing the computer-readable instructions and / or modules stored in the memory 12, and calling the data stored in the memory 12. The memory 12 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the electronic device, etc. The memory 12 can include non-volatile and volatile memories, such as: a hard disk, a memory, a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other storage devices.

[0252] The memory 12 may be an external memory and / or an internal memory of the electronic device 1. Furthermore, the memory 12 may be a memory in a physical form, such as a memory stick, a TF card (Trans-flash Card), and the like.

[0253] If the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also instruct the relevant hardware to complete it through computer-readable instructions. The computer-readable instructions can be stored in a computer-readable storage medium. When the computer-readable instructions are executed by the processor, the steps of the above-mentioned various method embodiments can be implemented.

[0254] The computer-readable instructions include computer-readable instruction codes, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer-readable instruction code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM).

[0255] The blockchain referred to in this invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0256] Combination Figure 1 The memory 12 in the electronic device 1 stores computer-readable instructions to implement an abnormal data processing method, and the processor 13 can execute the computer-readable instructions to implement:

[0257] Obtain the data to be analyzed;

[0258] Identify the abnormal type to which the data to be analyzed belongs;

[0259] If the abnormality type is a preset type, discretize the data to be analyzed to obtain discrete data;

[0260] Divide the discrete data according to the data frequency of the discrete data to obtain a plurality of sequence groups;

[0261] Search for the target nearest neighbor grouping for each sequence grouping;

[0262] Locating the data inflection point of the data to be analyzed according to each sequence grouping and the target nearest neighbor grouping;

[0263] Identifying abnormal data in the data to be analyzed according to the data inflection point;

[0264] The abnormal data is removed from the data to be analyzed to obtain target data.

[0265] Specifically, the specific implementation method of the processor 13 for the above-mentioned computer readable instructions can refer to Figure 1 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0266] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0267] The computer-readable storage medium stores computer-readable instructions, wherein the computer-readable instructions are used to implement the following steps when executed by the processor 13:

[0268] Obtain the data to be analyzed;

[0269] Identify the abnormal type to which the data to be analyzed belongs;

[0270] If the abnormality type is a preset type, discretize the data to be analyzed to obtain discrete data;

[0271] Divide the discrete data according to the data frequency of the discrete data to obtain a plurality of sequence groups;

[0272] Search for the target nearest neighbor grouping for each sequence grouping;

[0273] Locating the data inflection point of the data to be analyzed according to each sequence grouping and the target nearest neighbor grouping;

[0274] Identifying abnormal data in the data to be analyzed according to the data inflection point;

[0275] The abnormal data is removed from the data to be analyzed to obtain target data.

[0276] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0277] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0278] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0279] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. The multiple units or devices mentioned can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

[0280] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for processing abnormal data, characterized in that: The abnormal data processing method comprises: Acquire data to be analyzed, where the data to be analyzed is any medical data, including personal health records, prescriptions, and examination reports; Identifying the abnormal type to which the medical data belongs; If the abnormality type is a preset type, the medical data is discretized to obtain discrete data, including: locating a data field corresponding to the medical data according to a data source of the medical data; obtaining a plurality of preset ranges in the data field; discretizing the medical data into scores corresponding to the plurality of preset ranges to obtain the discrete data; Dividing the discrete data according to the data frequency of the discrete data to obtain a plurality of sequence groups includes: sorting the discrete data according to the data frequency to obtain a discrete sequence; generating a plurality of frequency intervals according to the data frequency and a preset frequency distance; dividing the discrete sequence according to the plurality of frequency intervals and the data frequency to obtain the plurality of sequence groups; Searching for a target nearest neighbor group of each sequence group includes: determining a discrete position of each sequence group in the discrete sequence; determining a search direction according to an arrangement direction of the plurality of sequence groups in the discrete sequence; traversing the plurality of sequence groups in the discrete sequence in the search direction based on the discrete position to obtain a plurality of search groups; and determining a search group having the smallest position distance to the discrete position as the target nearest neighbor group; The data inflection point of the medical data is located according to each sequence group and the target nearest neighbor group, including: obtaining the maximum frequency and the minimum frequency of each sequence group from the data frequency; calculating the ratio of the difference between the maximum frequency and the minimum frequency in the preset frequency distance to obtain the group slope of each sequence group; calculating the slope difference between each sequence group and the target nearest neighbor group based on the group slope; and selecting the sequence group corresponding to the slope difference with the largest value from the slope difference as the data inflection point; Identifying abnormal data in the medical data according to the data inflection point includes: determining a target nearest neighbor group corresponding to the data inflection point as a feature group; extracting a sequence group whose data frequency is less than the feature group from the discrete sequence as a target group; and determining the medical data corresponding to the target group as the abnormal data; The abnormal data is removed from the medical data to obtain target data.

2. The abnormal data processing method according to claim 1, characterized in that: The identifying the abnormal type to which the medical data belongs includes: Counting the total amount of the medical data; Determine a first position serial number and a second position serial number in the medical data based on a first preset value, a second preset value and the total amount of data; Sort the medical data in ascending order to obtain a data sequence and a sequence number of the medical data in the data sequence; Calculating a first quantile based on the medical data, the sequence number and the first position number, and calculating a second quantile based on the medical data, the sequence number and the second position number; Calculate the difference between the first quantile and the second quantile to obtain a data threshold; Determine a target interval according to the first quantile, the second quantile and the data threshold; If any data in the medical data is not in the target interval, the abnormal type is determined as the preset type.

3. An abnormal data processing device, characterized in that: The abnormal data processing device comprises: An acquisition unit, used for acquiring data to be analyzed, wherein the data to be analyzed is any medical data, including personal health records, prescriptions and examination reports; An identification unit, used for identifying the abnormal type to which the medical data belongs; A processing unit, configured to discretize the medical data to obtain discrete data if the abnormality type is a preset type, including: locating a data field corresponding to the medical data according to a data source of the medical data; obtaining a plurality of preset ranges in the data field; discretizing the medical data into scores corresponding to the plurality of preset ranges to obtain the discrete data; a division unit, configured to divide the discrete data according to the data frequency of the discrete data to obtain a plurality of sequence groups, comprising: sorting the discrete data according to the data frequency to obtain a discrete sequence; generating a plurality of frequency intervals according to the data frequency and a preset frequency distance; and dividing the discrete sequence according to the plurality of frequency intervals and the data frequency to obtain the plurality of sequence groups; A search unit is used to search for a target nearest neighbor group of each sequence group, including: determining a discrete position of each sequence group in the discrete sequence; determining a search direction according to an arrangement direction of the plurality of sequence groups in the discrete sequence; traversing the plurality of sequence groups in the discrete sequence in the search direction based on the discrete position to obtain a plurality of search groups; and determining a search group having the smallest position distance to the discrete position as the target nearest neighbor group; A positioning unit, used for locating the data inflection point of the medical data according to each sequence group and the target nearest neighbor group, including: obtaining the maximum frequency and the minimum frequency of each sequence group from the data frequency; calculating the ratio of the difference between the maximum frequency and the minimum frequency in the preset frequency distance to obtain the group slope of each sequence group; calculating the slope difference between each sequence group and the target nearest neighbor group based on the group slope; and selecting the sequence group corresponding to the slope difference with the largest value from the slope differences as the data inflection point; The identification unit is further used to identify abnormal data in the medical data according to the data inflection point, including: determining the target nearest neighbor group corresponding to the data inflection point as the feature group; extracting the sequence group whose data frequency is less than the feature group from the discrete sequence as the target group; and determining the medical data corresponding to the target group as the abnormal data; The elimination unit is used to eliminate the abnormal data from the medical data to obtain target data.

4. An electronic device, characterized in that: The electronic device comprises: a memory storing computer-readable instructions; and A processor executes computer-readable instructions stored in the memory to implement the abnormal data processing method according to any one of claims 1 to 2.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions are executed by a processor in an electronic device to implement the abnormal data processing method as described in any one of claims 1 to 2.

Images