Data processing method and device

Abstract

The invention provides a data processing method and device, and the method comprises the steps: determining initial data as to-be-processed data under the condition that the obtained initial data meets a preset judgment condition; detecting the to-be-processed data according to a first detection module of the initial object to obtain a detection value of the to-be-processed data, and determining a target detection value interval of the to-be-processed data based on the detection value of the to-be-processed data; detecting the to-be-processed data according to a second detection module of the initial object to obtain a target density distribution value and a target deviation threshold value of the to-be-processed data; the initial object is processed according to the to-be-processed data and the target detection value interval, the target density distribution value and the target deviation threshold value of the to-be-processed data, a target object is generated, and the target object comprises a first detection module and a second detection module and is used for determining whether the detection data is abnormal data or not. Whether the detection data is abnormal data is determined through the target object, and the probability of security risks is reduced.

Description

[0001] This application is a divisional application with application number 202110786822.7, application date July 12, 2021, and invention name “Data processing method and device”. Technical Field

[0002] The present invention relates to the field of computer technology, and more particularly to a data processing method, a data processing device, a computing device, and a computer-readable storage medium. Background Art

[0003] With the rapid development of the Internet and financial industries, a large number of Internet financial companies have emerged. Users can initiate loans from Internet financial companies, and Internet financial companies will assess the user's credit rating based on the user's personal data and lend to the user based on the user's credit rating, thereby avoiding security risks such as fraud and default in the process of Internet financial companies lending to users.

[0004] However, since the sources of user personal data are diverse and large in quantity, the existing technology can only verify user personal data through broad regulations and is unable to accurately identify abnormal data in user personal data, increasing the probability of security risks for Internet financial companies. Summary of the invention

[0005] In view of this, the embodiments of this specification provide a data processing method. This specification also relates to a data processing device, a computing device, and a computer-readable storage medium to solve the technical defects existing in the prior art.

[0006] According to a first aspect of an embodiment of this specification, a data processing method is provided, including: Inputting the acquired detection data into a first detection module and a second detection module of the target object, obtaining a detection value of the detection data output by the first detection module, and obtaining a density distribution value of the detection data output by the second detection module; Based on the detection value of the detection data and the density distribution value of the detection data, it is determined whether the detection data is abnormal data.

[0007] According to a second aspect of an embodiment of this specification, there is provided a data processing device, including: an acquisition module configured to input the acquired detection data into a first detection module and a second detection module of the target object, obtain a detection value of the detection data output by the first detection module, and obtain a density distribution value of the detection data output by the second detection module; The determination module is configured to determine whether the detection data is abnormal data based on the detection value of the detection data and the density distribution value of the detection data.

[0008] According to a third aspect of an embodiment of this specification, a computing device is provided, including: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of any of the data processing methods are implemented.

[0009] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps of any of the data processing methods are implemented.

[0010] The data processing method provided in the present specification obtains the detection value of the detection data output by the first detection module and the density distribution value of the detection data output by the second detection module by inputting the acquired detection data into the first detection module and the second detection module of the target object; thereby, based on the detection value of the detection data and the density distribution value of the detection data, it is possible to quickly and accurately determine whether the detection data is abnormal data, thereby reducing the probability of security risks incurred by the enterprise and avoiding losses caused to the enterprise by security risks. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a flow chart of a data processing method provided by an embodiment of this specification; Figure 2 It is a schematic diagram of training sample data that conforms to the normal distribution rule in a data processing method provided in an embodiment of this specification; Figure 3 It is a schematic diagram of calculating an outlier value by a Z-Score test method in a data processing method provided in an embodiment of the present specification; Figure 4 It is a schematic diagram of calculating an outlier by a box plot method in a data processing method provided in an embodiment of the present specification; Figure 5 It is a processing flow chart of a data processing method for detecting whether loan amount data is abnormal data in an embodiment of this specification; Figure 6 It is a structural schematic diagram of a data processing device provided in an embodiment of this specification; Figure 7 It is a structural block diagram of a computing device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0012] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0013] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0014] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0015] First, the terms involved in one or more embodiments of this specification are explained.

[0016] Numerical data: can be data such as principal, interest, penalty interest, interest rate, number of overdue days, etc.

[0017] IoT devices: Internet of Things (IoT) devices, referred to as IoT devices, refer to devices connected to the network according to agreed protocols. The devices exchange and communicate information through information transmission media to achieve intelligent identification, positioning, tracking, supervision and other functions.

[0018] Density distribution: also called probability density distribution. Probability refers to the chance of an event occurring randomly. For a uniform distribution function, the density distribution is equal to the probability of a certain interval (the range of values ​​of the event) divided by the length of the interval. Its value is non-negative and can be large or small.

[0019] KNN model: K-Nearest Neighbor model, referred to as KNN model, is a machine learning algorithm that can be used for both classification and regression.

[0020] OCSVM model: One class support vector machine model, referred to as OCSVM model, is a model used for abnormal data detection.

[0021] LOF model: Local Outlier Factor model, referred to as LOF model, is a model used to clean data and eliminate invalid and abnormal data.

[0022] HBOS model: Histogram-based Outlier Score model, which is an unsupervised training model.

[0023] Kolmogorov Smirnov (ks) test: Kolmogorov-Smirnov test, used to test whether data conform to a certain distribution. In the embodiment of this specification, it is used to test whether data conform to a normal distribution.

[0024] Lillifors test: is a test method that can be used to detect whether the data conforms to the normal distribution.

[0025] BOX-COX transformation: It is a data transformation commonly used in statistical modeling and is used when the continuous response variable does not satisfy the normal distribution.

[0026] Log change: Logarithmic transformation is a data conversion method.

[0027] Z-Score test method: Z-Score is also called Z value. Z-Score test method is a parameter anomaly detection method in one-dimensional or low-dimensional feature space. This method assumes that the data is normally distributed and the outliers are the data points at the tail of the distribution on both sides, so they are far away from the mean value of the data.

[0028] Tukey: Also known as the box plot method, it is a statistical graph used to show the dispersion of a set of data.

[0029] Grubbs test: also known as Grubbs test, Grubbs test is a method to find outliers from data. The use of Grubbs test requires that the data population conforms to the normal distribution.

[0030] Sky-integrated satellite: refers to the satellite in the sky-integrated system. The sky-integrated system refers to an information network system composed of satellites, drones, unmanned ships and other equipment, which is used to monitor target objects in real time and over a large area.

[0031] When granting credit to users (small, medium and micro enterprises and / or individual users), Internet financial companies can use the user data stored by the Internet financial companies themselves, combined with external data (single-source data) provided by user input, partners, integrated space satellites and IoT devices, as a source of credit enhancement; Internet financial companies use their own stored user data and external data to fully analyze, evaluate and grant credit to users, thereby ensuring that they can lend to users without collateral.

[0032] Whether the data generated by the integrated space satellites in space is normal, whether the self-certified data provided by farmers in the agricultural loan process is a reflection of the farmland that farmers actually own, whether the information in the vehicle loans provided by cooperative institutions (partners) is true, and whether the data generated by IoT devices is subject to human intervention, etc., will directly or indirectly affect the Internet financial companies' lending to users.

[0033] If there are problems with the quality of the single-source data obtained by Internet financial companies, on the one hand, it will affect the increase or decrease of loan amounts, such as whether the needs of small and medium-sized micro-enterprises are met to the greatest extent, and whether the loan amount is reduced in time to avoid bad debts and capital losses; on the other hand, in order to strictly comply with the requirements of the regulatory authorities, the data reported to the regulator must be true and accurate. Therefore, ensuring that there are no problems with the quality of single-source data is an urgent problem that Internet financial companies need to solve.

[0034] However, the sources of single-source data are not only diverse, but the data generated by users, software and hardware every day has reached the order of tens of millions. The data stored by Internet financial enterprises themselves can ensure their quality through their own testing and verification, but it is almost impossible to ensure the quality of single-source data through existing technologies. The reason is that you want to verify the correctness of single-source data, but there is no counterparty to verify. In the existing technology, the verification method is to prompt that the data is abnormal when the upstream data is not equal to the downstream data. In the scenario of single-source data, Internet financial enterprises can only get downstream data, but not source data, so they cannot do consistency verification. For example, whether the loan information provided by the cooperative institution, such as principal, interest, penalty interest, interest rate, overdue days, etc., is true and valid, there is no effective identification method in the existing technology.

[0035] Therefore, existing technologies can only rely on some broad hard rules to verify single-source data. Under broad rule governance, there will inevitably be some loopholes, and some hidden and incorrect data will be mixed in.

[0036] Based on this, in this specification, a data processing method is provided. This specification also involves a data processing device, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0037] See also Figure 1 , Figure 1 A flowchart of a data processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps: Step 102: input the acquired detection data into the first detection module and the second detection module of the target object, obtain the detection value of the detection data output by the first detection module, and obtain the density distribution value of the detection data output by the second detection module.

[0038] Among them, the detection data can be understood as user feature information provided by the user or provided by a third-party platform to the data processing platform. The detection data can be any type of data, such as numerical data, character data, and multimedia data; for ease of understanding, this specification only takes the detection data as numerical data as an example to introduce the data processing method in detail. In addition, the application scenarios of the data processing method are different, and the corresponding detection data are also different; for example, when the data processing method is applied to the library borrowing scenario, the detection data can be the student number, the number of borrowed books, and the borrowing date; when the data processing method is applied to the lending scenario, the detection data can be the loan amount data, principal, interest, penalty interest, interest rate, and overdue days.

[0039] The data processing platform can be understood as an abnormal data detection platform that detects whether the acquired detection data is abnormal data.

[0040] The third-party platform can be understood as a platform that can provide detection data to the data processing platform. The third-party platform may include but is not limited to a vehicle lending platform, a satellite platform, and an IoT device platform.

[0041] The target object can be understood as the object for detecting the detection data. In practical applications, the target object can be regarded as a detection model for detecting the detection data. Different application scenarios of the data processing method have different corresponding target objects. For example, in the application scenario of anomaly detection on images, the target object can be an abnormal image detection model, and in the application scenario of anomaly detection on numerical data, the target object can be an abnormal data detection model.

[0042] In the case where the target object is a detection model, the first detection module can be understood as the first detection layer of the detection model, and the second detection module can be understood as the second detection layer of the detection model.

[0043] The detection value of the detection data can be understood as the detection data after normalization processing; when the detection data is numerical data, the detection data received by the data processing platform can be loan amount data and loan interest rate; however, there are large differences between the loan amount data and the loan interest rate. For example, the loan amount data can be 9320000 and 25000000; the loan interest rate can be 0.167332 and 0.1288; based on this, the data processing platform normalizes the loan amount data and loan interest rate through the first detection module of the target object; scales the loan amount data and loan interest rate proportionally and maps them to a numerical range, which can be [0,1], [-1,1]; to facilitate subsequent processing of the detection data.

[0044] The density distribution value can be understood as a numerical value representing the density distribution of the detection data; when the detection data is loan amount data, the density distribution value can be understood as a numerical value representing the density distribution of the loan amount data.

[0045] Specifically, the data processing platform inputs the acquired detection data into the first detection module and the second detection module of the target object, detects the detection data based on the first detection module, and obtains the detection value of the detection data output by the first detection module; detects the detection data based on the second detection module, and obtains the density distribution value of the detection data output by the second detection module.

[0046] For example, taking the scenario in which the data processing method is applied to detect the user's loan amount as an example, the detection value and density distribution value of the detection data obtained through the target object are described in detail, wherein the detection data may be multiple user loan amount data provided by the lending platform, for example, the loan amount data is 100 pieces, the data processing platform may be an abnormal data detection platform, the third-party platform may be a lending platform, and the target object may be an anomaly detection model for detecting abnormal data in the loan amount data, hereinafter referred to as the anomaly detection model; the first detection module may be understood as the first anomaly detection layer of the anomaly detection model, and the second detection module may be understood as the second anomaly detection layer of the anomaly detection model.

[0047] The abnormal data detection platform is able to obtain 100 loan amount data provided by the lending platform, and input the loan amount data into the first anomaly detection layer and the second anomaly detection layer of the anomaly detection model.

[0048] After the first anomaly detection layer of the anomaly detection model receives 100 pieces of loan amount data, the loan amount data is normalized to obtain detection values ​​corresponding to the 100 pieces of loan amount data one by one, and the detection values ​​are output.

[0049] After the second anomaly detection layer of the anomaly detection model receives 100 pieces of loan amount data, it can detect the density distribution of the loan amount, thereby determining the density distribution value of the 100 loan amounts and outputting the density distribution value.

[0050] In practical applications, the first anomaly detection layer of the anomaly detection model can be an outlier detection model composed of a KNN model and an OCSVM model, and the detection data is detected through the KNN model and the OCSVM model; the second anomaly detection layer of the anomaly detection model can be a density detection model composed of a LOF model and a HBOS model, and the density distribution value of the detection data is detected through the LOF model and the HBOS model.

[0051] Step 104: Determine whether the detection data is abnormal data based on the detection value of the detection data and the density distribution value of the detection data.

[0052] Among them, abnormal data can be understood as detection data with abnormalities. For example, when the detection data is loan amount data, the abnormal data can be loan amount data with an amount of 0; when the detection data is the number of borrowed books, the abnormal data can be the number of borrowed books with a number of -1.

[0053] Specifically, after acquiring the detection value of the detection data and the density distribution value of the detection data through the target object, the data processing platform determines whether the detection data is abnormal data based on the detection value and the density distribution value of the detection data.

[0054] Further, based on the detection value of the detection data and the density distribution value of the detection data, a specific implementation method of determining whether the detection data is abnormal data is as follows: The determining whether the detection data is abnormal data based on the detection value of the detection data and the density distribution value of the detection data includes: When the detection value of the detection data is outside the target detection value interval, the detection data is determined to be abnormal data, or When the degree of deviation between the density distribution value of the detection data and the target density distribution value is greater than or equal to a target deviation threshold, the detection data is determined to be abnormal data.

[0055] Among them, the target detection value interval can be understood as an interval determined by historical detection data and used to determine whether the detection data is abnormal; in the embodiment of the present specification, the interval distribution law of abnormal data and non-abnormal data in the training sample data is calculated by the first detection module of the target object, and the distribution interval of non-abnormal data is used as the target detection value interval; when the detection value of the detection data is within the target detection value interval, the detection data is non-abnormal data, and when the detection value of the detection data is outside the target detection value interval, the detection data is abnormal data.

[0056] The target density distribution value can be understood as being determined by historical detection data, and is used to determine whether the density distribution of the detection data conforms to a normal density distribution value. In an embodiment of the present specification, the density distribution of the training sample data is detected by the second detection module of the target object to determine the density distribution law of the training sample data, and based on the density distribution law, the target density distribution value for detecting the detection data is determined.

[0057] The degree of deviation can be understood as a numerical value representing the deviation between the density distribution value of the test data and the target density distribution value, and the degree of deviation is a non-negative number; when the numerical value representing the deviation between the density distribution value of the test data and the target density distribution value is a negative number, the opposite of the numerical value is obtained, and the opposite number is used as the degree of deviation; when the numerical value representing the deviation between the density distribution value of the test data and the target density distribution value is a positive number, the numerical value is used as the degree of deviation.

[0058] The target deviation threshold can be understood as a threshold used to limit the deviation between the density distribution value of normal detection data and the target density distribution value. In practical applications, due to the diversity of detection data, the density distribution value of each detection data will not be completely consistent with the target density distribution value. Therefore, the target deviation threshold is used to limit the deviation between the density distribution value of the detection data and the target density distribution value. The target deviation threshold can be set according to actual needs, and this manual does not make any restrictions on this.

[0059] Specifically, after obtaining the detection value and density distribution value of the detection data, the data processing platform can determine whether the detection data is abnormal data based on the detection value and density distribution value; when the detection value of the detection data is outside the target detection value range, the detection data is determined to be abnormal data, or when the deviation between the density distribution value of the detection data and the target density distribution value is greater than or equal to the target deviation threshold, the detection data is determined to be abnormal data.

[0060] Continuing with the above example, the determination of whether the detected data is abnormal data through the detection value and the density distribution value is explained in detail.

[0061] After the abnormal data detection platform obtains the detection values ​​and density distribution values ​​of 100 loan amount data through the abnormal detection model, the first detection layer of the abnormal detection model can determine whether the 100 detection values ​​corresponding to the loan amount data are all within the target detection value range. If so, it means that there is no abnormal loan amount data in the loan amount data, and the loan amount data is normal data; if not, it means that there is abnormal loan amount data in the loan amount data, and the loan amount data is abnormal data.

[0062] Alternatively, the abnormal data detection platform may subtract the density distribution value of the loan amount data from the target density distribution value, and calculate the difference between the two. When the difference is a positive number, the difference is used as the deviation between the density distribution value of the loan amount data and the target density distribution value; when the difference is a negative number, the opposite of the difference is obtained, and the opposite is used as the deviation between the density distribution value of the loan amount data and the target density distribution value; determine whether the deviation is greater than or equal to the target deviation threshold. If so, it means that there is an abnormality in the density distribution of the loan amount data, and the loan amount data is determined as abnormal data; if not, it means that there is no abnormality in the density distribution of the loan amount data, and the loan amount data is non-abnormal data.

[0063] In the embodiments of the present specification, by determining whether the detection value of the detection data is within the target detection value range, or determining whether the deviation between the density distribution value of the detection data and the target density distribution value is greater than or equal to the target deviation threshold, these two detection dimensions are used to detect abnormal data, thereby accurately and effectively determining whether the acquired detection data is abnormal data, thereby avoiding losses to the enterprise caused by abnormal data.

[0064] In addition, in the embodiment of the present specification, before obtaining the detection value and density distribution value of the detection data through the target object, the target object needs to be generated; thereby determining whether the detection data is abnormal data based on the detection value and density distribution value of the detection data obtained by the target object; the specific method of generating the target object is as follows: Before inputting the acquired detection data into the first detection module and the second detection module of the target object, the method further includes steps 1042 to 1048: Step 1402: When the obtained initial data meets the preset determination condition, the initial data is determined as data to be processed.

[0065] The initial data may be understood as data that can be used to process the initial object. When the initial object is a detection model to be trained, the initial data may be understood as initial sample data that can be used to train the detection model to be trained.

[0066] The preset judgment condition can be set according to the actual application, for example, the preset judgment condition includes whether the initial data conforms to the normal distribution rule.

[0067] When the initial data is initial sample data, the data to be processed can be understood as initial sample data that meets preset judgment conditions. In the embodiment of this specification, the initial sample data that meets the preset judgment conditions is determined as training sample data for training the detection model.

[0068] Specifically, when the obtained initial data meets the preset determination conditions, the data processing platform determines the initial data as the data to be processed.

[0069] Furthermore, when the obtained initial data meets the preset determination condition, the initial data is determined as data to be processed, and further includes: When the obtained initial sample data meets the preset determination condition, the initial sample data is determined as training sample data.

[0070] Among them, the initial sample data can be understood as sample data used to train the detection model to be trained. In actual applications, the initial data will be different depending on the target object. For example, the target object is an anomaly detection model that detects abnormal amount data in loan amount data. Accordingly, the initial data can be historical loan amount data. For example, the historical loan amount data can be 9320000 or 2500000.

[0071] The preset judgment condition may be whether the initial sample data is normally distributed.

[0072] The training sample data may be understood as initial sample data that meets a preset determination condition. When the preset determination condition is whether the data is normally distributed, the training sample data may be initial sample data that meets a normal distribution.

[0073] Specifically, when the obtained initial sample data meets a preset determination condition, the initial sample data is determined as training sample data.

[0074] Continuing with the above example, the determination of the data to be processed is further explained in detail. Before the abnormal data detection platform obtains the detection value and density distribution value of the loan amount data through the abnormal data detection model, it is also necessary to train the abnormal data detection model to be trained, so as to obtain the trained abnormal data detection model, thereby realizing the detection value and density distribution value detected by the abnormal data detection model, and determining whether the loan amount data provided by the lending platform is abnormal data.

[0075] To train the anomaly detection model to be trained, it is first necessary to obtain initial sample data and judge the initial sample data. When the initial sample data satisfies the normal distribution rule, the initial sample data is determined as the training sample data for training the anomaly detection model to be trained.

[0076] In the embodiments of the present specification, by determining the initial sample data that meets the preset judgment conditions as the training sample data, the detection model to be trained is subsequently trained with the training sample data that meets the preset judgment, thereby effectively improving the performance of the trained anomaly detection model.

[0077] Step 1044: Detect the data to be processed according to the first detection module of the initial object to obtain the detection value of the data to be processed, and determine the target detection value interval of the data to be processed based on the detection value of the data to be processed.

[0078] In which, when the target object is a detection model, the initial object may be a detection model to be trained.

[0079] Specifically, the data processing platform detects the data to be processed according to the first detection module of the initial object, obtains the detection value of the data to be processed, and determines the target detection value interval of the data to be processed based on the detection value of the data to be processed.

[0080] Further, the detecting the data to be processed according to the first detection module of the initial object to obtain the detection value of the data to be processed, and determining the target detection value interval of the data to be processed based on the detection value of the data to be processed, further includes: The training sample data is detected according to the first detection layer of the detection model to be trained to obtain the detection value of the training sample data, and the target detection value interval of the training sample data is determined based on the detection value of the training sample data.

[0081] Among them, the first detection layer of the detection model to be trained can be an outlier detection model composed of KNN and OCSVM, and the target detection value interval of the training sample data can be determined through the outlier detection model.

[0082] Specifically, after determining the training sample data, the data processing platform detects the training sample data according to the first detection layer of the detection model to be trained to obtain the detection value of the training sample data, and determines the target detection value interval of the training sample data based on the detection value of the training sample data.

[0083] Continuing with the above example, determining the target detection value interval of the training sample data based on the detection value of the training sample data is described in further detail.

[0084] After determining the training sample data, the anomaly data detection platform inputs the training sample data into the first detection layer of the anomaly detection model to be trained. The first detection layer is an outlier detection model composed of KNN and OCSVM. The training sample data is normalized by the outlier detection model to obtain the detection value corresponding to the training sample data. Afterwards, the first detection layer uses the detection value of the training sample data as an operation parameter and calculates the target detection value interval corresponding to the training sample data through the detection value algorithm.

[0085] In the embodiments of the present specification, the training sample data is detected by the first detection layer of the detection model to be trained to obtain the detection value of the training sample data, thereby ensuring the consistency of the training sample data, and determining the target detection value range of the training sample data based on the detection value of the training sample data; thereby avoiding the occurrence of false alarms of abnormal values ​​in the subsequent detection of the detection data, and realizing the effective detection of whether the detection data is abnormal data through the detection model.

[0086] Step 1046: Detect the data to be processed according to the second detection module of the initial object to obtain the target density distribution value and the target deviation threshold of the data to be processed.

[0087] Specifically, the data to be processed is detected according to the second detection module of the initial object to obtain a target density distribution value and a target deviation threshold of the data to be processed.

[0088] Furthermore, the detecting the data to be processed according to the second detection module of the initial object to obtain the target density distribution value and the target deviation threshold of the data to be processed also includes: The training sample data is detected according to the second detection layer of the detection model to be trained to obtain a target density distribution value and a target deviation threshold of the training sample data.

[0089] Among them, the second detection layer of the detection model to be trained can be understood as a density detection model composed of the LOF model and the HBOS model.

[0090] Specifically, the training sample data is detected according to the second detection layer of the detection model to be trained to obtain the target density distribution value and the target deviation threshold of the training sample data.

[0091] Using the above example, the target density distribution value and target deviation threshold of the training sample data are explained in detail.

[0092] The abnormal data detection platform inputs the training sample data into the second detection layer of the anomaly detection model to be trained, and the second detection layer is a density detection model composed of the LOF model and the HBOS model. The training sample data is detected by using the LOF model and the HBOS model to obtain the target density distribution value of the training sample data, and the target deviation threshold is determined based on the target density distribution value.

[0093] In the embodiments of the present specification, the training sample data is detected according to the second detection layer of the detection model to be trained, and the target density distribution value and the target deviation threshold of the training sample data are obtained, so as to detect the detection data from the dimension of density distribution, thereby further improving the performance of the trained detection model.

[0094] Step 1048: Process the initial object according to the data to be processed and the target detection value interval, target density distribution value, and target deviation threshold of the data to be processed to generate a target object.

[0095] The target object includes a first detection module and a second detection module.

[0096] Specifically, the data processing platform processes the initial object according to the data to be processed and the target detection value interval, target density distribution value, and target deviation threshold of the data to be processed to generate the target object.

[0097] Furthermore, the processing of the initial object according to the data to be processed and the target detection value interval, target density distribution value, and target deviation threshold of the data to be processed to generate the target object also includes: The detection model to be trained is trained according to the training sample data and the target detection value interval, target density distribution value, and target deviation threshold of the training sample data to generate a trained detection model.

[0098] Specifically, the detection model to be trained is trained based on the target detection value interval of the training sample data obtained by the first detection layer of the detection model to be trained, the target density distribution value of the training sample data obtained by the second detection layer, the target deviation threshold and the training sample data to generate a trained detection model.

[0099] Continuing with the above example, the generation of the trained detection model is further explained in detail.

[0100] The data processing platform uses the training sample data and the target detection value interval, target density distribution value, and target deviation threshold corresponding to the training sample data as training input, and inputs them into the anomaly detection model to be trained, trains the anomaly detection model to be trained, and thus obtains a trained anomaly detection model.

[0101] In the embodiments of the present specification, the initial object is processed to generate a target object by obtaining the target detection value interval, target density distribution value, and target deviation threshold of the data to be processed through the data to be processed and the first detection module and the second detection module based on the initial object, thereby achieving the subsequent determination of whether the detection data is abnormal data through the target object, reducing the probability of security risks in the enterprise and avoiding losses to the enterprise caused by security risks such as breach of contract and overdue.

[0102] In the embodiments of the present specification, the initial object is processed by using the data to be processed that meets the preset judgment conditions, the target detection value interval determined by the first detection module of the initial object, the target density distribution value and the target deviation threshold obtained by the second detection module of the initial object, thereby ensuring the performance of the target object and avoiding the occurrence of false alarms in the subsequent target object during the detection of the detection data. Based on the generated target object, subsequent rapid and accurate identification of abnormal data is achieved, reducing the probability of security risks in the enterprise.

[0103] In the process of generating a target object, if the initial object is processed only by the initial data, the performance of the generated target object may be low. For example, in the process of generating a trained detection model, if the detection model to be trained is trained by the initial sample object instead of the data to be processed that conforms to the normal distribution rule, the accuracy of the trained detection model may be low during the detection process. In this case, the above problem can be solved by determining the initial data that meets the preset judgment conditions as the data to be processed. The specific implementation method is as follows: In the case where the obtained initial data meets the preset determination condition, determining the initial data as data to be processed also includes: Obtaining initial data, and determining whether the initial data satisfies a normal distribution, If yes, the initial data is determined as data to be processed; If not, the initial data is converted according to the normal distribution rule, and if the conversion is successful, the converted initial data is determined as the data to be processed.

[0104] Specifically, after obtaining the initial data, the data processing platform needs to determine whether the initial data satisfies the normal distribution. If so, the initial data is determined as the data to be processed; if not, the initial data is converted according to the normal distribution rules, and if the conversion is successful, the converted initial data is determined as the data to be processed.

[0105] Continuing with the above example, the determination of the initial data as the data to be processed is further explained in detail; in the process of generating a trained anomaly detection model, if the anomaly detection model to be trained is trained with sample data that conforms to a normal distribution, the performance of the trained anomaly detection model can be improved.

[0106] Based on this, see Figure 2 , Figure 2 A schematic diagram of training sample data that conforms to the normal distribution rule in a data processing method provided in an embodiment of this specification. After obtaining the initial sample data, the abnormal data detection platform verifies the initial sample data through a verification algorithm to determine whether the initial sample data conforms to the normal distribution.

[0107] Specifically, the initial sample data that conforms to the normal distribution can be obtained by Figure 2 The shaded area is used to represent the probability density of the initial sample data, where 0.0, 0.1, 0.2, 0.3, 0.4, etc. are used to represent the probability density of the initial sample data; the value 0 is used to represent the average value of the initial sample data, and the value is a standard deviation, the value The area between the value 0 is used to represent the initial sample data that is less than one standard deviation from the mean value. This area accounts for about 68.2% (34.1% + 34.1%) of all the initial sample data. is two standard deviations, the value The area between the value 0 is used to represent the initial sample data that is less than two standard deviations from the mean value. This area accounts for about 95.4% of all the initial sample data (68.2%+13.6%*2); the value is three standard deviations, the value The area between the value 0 is used to represent the initial sample data that is less than three standard deviations from the mean value. This area accounts for about 99.6% of all the initial sample data (95.4%+2.1%*2); the value The area far away from the value 0 represents the initial sample data that is more than three standard deviations away from the mean value. This area accounts for approximately 0.2% (0.1% + 0.1%) of all the initial sample data.

[0108] In the case that the initial sample data conforms to the normal distribution, the initial sample data is determined as the training sample data, and the anomaly detection model to be trained is trained by using the training sample data.

[0109] When the initial sample data does not conform to the normal distribution, the initial sample data is converted into initial sample data that conforms to the normal distribution through a transformation algorithm according to the normal distribution rules. When the conversion is successful, the successfully converted initial sample data is determined as training sample data to obtain the data required for training and that satisfies the normal distribution, and the training sample data is used to train the anomaly detection model to be trained.

[0110] In actual application, the test algorithm can be any algorithm for testing whether the data conforms to the normal distribution. In the embodiments of this specification, the initial sample data can be tested by any test algorithm according to the needs of the actual application, and the embodiments of this specification do not impose any restrictions on this; for example, the test algorithm can be a Kolmogorov Smirnov (ks) test, a Lillifors test, or other test algorithms.

[0111] The transformation algorithm may be any algorithm that can transform data that does not conform to the normal distribution into data that conforms to the normal distribution. In the embodiments of this specification, the initial sample data that does not conform to the normal distribution may be transformed into initial sample data that conforms to the normal distribution by any transformation algorithm according to the needs of the actual application, and the embodiments of this specification do not impose any limitation on this. For example, the transformation algorithm may be a transformation algorithm such as a BOX-COX transformation or a Log transformation.

[0112] In the embodiments of the present specification, by determining the initial data that satisfies the normal distribution as the data to be processed, the initial object is subsequently processed by the data to be processed that conforms to the normal distribution, thereby improving the performance of the target object. Moreover, when the initial data does not conform to the normal distribution, the initial data is converted according to the normal distribution rules, and the successfully converted initial data is determined as the data to be processed, thereby enhancing the adaptability of the data processing platform to the initial data and improving the efficiency of processing the initial object.

[0113] However, in another case, when the conversion fails during the conversion of the initial data according to the normal distribution rule, the data processing platform cannot determine the data to be processed for processing the initial object.

[0114] Based on this, after converting the initial data according to the normal distribution rule, it also includes: In case of conversion failure, the candidate data is reacquired and a normal distribution determination is performed on the candidate data.

[0115] The candidate data may be understood as data reacquired by the data processing platform when the initial data cannot be determined as the data to be processed.

[0116] Specifically, the data processing platform converts the initial data according to the normal distribution rules, and in case of conversion failure, re-acquires the candidate data and performs normal distribution determination on the candidate data.

[0117] Continuing with the above example, the re-acquisition of candidate data in case of conversion failure is explained in detail. In the process of the abnormal data detection platform converting the initial sample data that does not conform to the normal distribution through the transformation algorithm, if the conversion fails, the candidate data will be re-acquired from the offline data table and tested through the test algorithm to determine whether the candidate data conforms to the normal distribution.

[0118] In the embodiments of the present specification, in the event of a conversion failure, candidate data is reacquired and a normal distribution determination is performed on the candidate data, thereby avoiding the risk of using initial data that does not conform to a normal distribution to process the initial object, resulting in a generated target object with lower performance, and effectively ensuring the performance of the target object.

[0119] After obtaining the detection value of the data to be processed through the first detection model of the initial object, the target detection value interval of the data to be processed can be calculated through the detection value algorithm. In practical applications, if the target detection value interval is calculated using only one detection value algorithm, the diversity of the data to be processed may not be fully considered, resulting in anomalies in the calculated target detection value interval, which affects the performance of the target object.

[0120] Based on this, determining the target detection value interval of the data to be processed based on the detection value of the data to be processed includes: Calculating the detection value of the data to be processed by a first detection value algorithm to obtain a first detection value interval of the data to be processed; Calculating the detection value of the data to be processed by a second detection value algorithm to obtain a second detection value interval of the data to be processed; Calculating the detection value of the data to be processed by a third detection value algorithm to obtain a third detection value interval of the data to be processed; Based on the first detection value interval, the second detection value interval, and the third detection value interval, a target detection value interval of the data to be processed is determined.

[0121] The first detection value algorithm may be understood as a first algorithm for calculating a first detection value interval of the data to be processed, and may specifically be a Z-Score test method.

[0122] The second detection value algorithm can be understood as a second algorithm for calculating the second detection value interval of the data to be processed, and specifically can be the Tukey box plot method.

[0123] The third detection value algorithm may be understood as a second algorithm for calculating the third detection value interval of the data to be processed, and may specifically be a Grubbs test method.

[0124] Specifically, after obtaining the detection value of the data to be processed, the data processing platform first calculates the detection value of the data to be processed through a first detection value algorithm to obtain a first detection value interval of the data to be processed; secondly, calculates the detection value of the data to be processed through a second detection value algorithm to obtain a second detection value interval of the data to be processed; then, calculates the detection value of the data to be processed through a third detection value algorithm to obtain a third detection value interval of the data to be processed; based on the first detection value interval, the second detection value interval and the third detection value interval, determine the target detection value interval of the data to be processed.

[0125] Continuing with the above example, determining the target detection value interval by using the first detection value algorithm, the second detection value algorithm, and the third detection value algorithm is described in detail.

[0126] After the first detection layer of the anomaly detection model to be trained obtains the detection value of the training sample data, firstly, the first detection layer calculates the first detection value interval of the training sample data through the Z-Score test method.

[0127] Specifically, Figure 3 As shown, Figure 3 This is a schematic diagram of outliers calculated by the Z-Score test method in a data processing method provided in an embodiment of the present specification. The Z-Score test method assumes that the data is normally distributed, and outliers are data points at the tails of the distribution on both sides. Therefore, when the detection values ​​that conform to the normal distribution rules are detected by the Z-Score test method, the outliers in the detection values ​​are the detection values ​​located in the tail areas on both sides of the normal distribution. The detection values ​​in this area are determined as outliers, and the interval where the normal detection values ​​are located is determined as the first detection value interval of the training sample data. The distance between the area where the outliers are located and the center value of the normal distribution depends on the normalized data point calculated using the formula Setting threshold ,like It is believed that The corresponding detection value is an outlier. Calculate the normalized data point The formula is: ,in, is a detection value, For all detected values The standard deviation of For all detected values The average value of . It can be set according to actual application, and the embodiments of this specification do not impose any limitation on this. Can be set to 2.5, 3.0, or 3.5.

[0128] Secondly, the second detection value interval of the training sample data is calculated by the box plot method. Specifically, the average value of the detection value that conforms to the normal distribution rule is and the standard deviation of , which is easily affected by individual outliers, thus affecting the effectiveness of the judgment. Therefore, the first detection layer detects outliers in the test values ​​by introducing the box plot method. Outliers can be calculated by IQR (Inter Quartile Range). Calculate the first quartile Q1 and the third quartile Q3. Outliers are test values ​​outside the quartile range. .

[0129] like Figure 4 As shown, Figure 4 It is a schematic diagram of calculating outliers by a box plot method in a data processing method provided in an embodiment of this specification. In the figure, IQR (interquartile range) is obtained by Q3-Q1, and the interval (Q1, Q3) covers the middle 50% of the data of the detection values ​​that conform to the normal distribution, which is robust. If the detection value falls within the range of (Q1-1.5IQR, Q3+1.5IQR), it is considered to be a normal value. The first detection layer of the anomaly detection model to be trained determines the range of (Q1-1.5IQR, Q3+1.5IQR) as the second detection value interval; if the detection value falls outside the range of (Q1-1.5IQR, Q3+1.5IQR), it is considered to be an outlier.

[0130] Finally, the third detection value interval of the training sample data is calculated by the Grubbs test method. Specifically, the first detection layer of the anomaly detection model to be trained sorts the detection values ​​in ascending order and calculates the average value of the detection values ​​that conform to the normal distribution rule. and the standard deviation of .

[0131] Calculate the minimum detection value and the average value The difference between the maximum detection value and the average value The difference between the two values ​​is determined, and the detection value corresponding to the larger difference between the two values ​​is determined as the suspicious detection value. The standard score of the suspicious detection value is calculated. If the standard score is greater than the Grubbs critical value, then the suspicious detection value is an outlier.

[0132] After calculating the abnormal values ​​in the detection values ​​according to the Grubbs test method, based on the distribution rules of the abnormal values ​​and normal detection values ​​in the detection values, the first detection layer determines the interval in which the normal detection values ​​are located as the third detection value interval.

[0133] The first detection layer of the detection model to be trained determines the target detection value interval of the data to be processed based on the first detection value interval, the second detection value interval and the third detection value interval.

[0134] In the embodiments of the present specification, the detection value of the data to be processed is calculated by using a second detection value algorithm, a second detection value algorithm and a third detection value algorithm to obtain the first detection value interval, the second detection value interval and the third detection value interval of the data to be processed, and based on the first detection value interval, the second detection value interval and the third detection value interval, the target detection value interval of the data to be processed is accurately determined. Through the combination of the three methods, the false alarm of abnormal values ​​in the subsequent detection of the detection data is avoided, the accuracy of the detection is improved, and the subsequent effective detection of whether the detection data is abnormal data is achieved.

[0135] Further, determining a target detection value interval of the data to be processed based on the first detection value interval, the second detection value interval, and the third detection value interval includes: Obtaining the maximum value and the minimum value of the first detection value interval, the maximum value and the minimum value of the second detection value interval, and the maximum value and the minimum value of the third detection value interval; Based on the maximum and minimum values ​​of the first detection value interval, the second detection value interval, and the third detection value interval, a target detection value interval of the data to be processed is determined.

[0136] Specifically, after obtaining the first detection value interval, the second detection value interval, and the third detection value interval of the data to be processed, the data processing platform obtains the maximum value and minimum value of the first detection value interval, the maximum value and minimum value of the second detection value interval, and the maximum value and minimum value of the third detection value interval; based on the maximum value and minimum value of the first detection value interval, the second detection value interval, and the third detection value interval, the target detection value interval of the data to be processed is determined.

[0137] Continuing with the above example, determining the target detection value interval of the data to be processed by using the maximum value and the minimum value of the first detection value interval, the second detection value interval and the third detection value interval is further described in detail.

[0138] After obtaining the first detection value interval, the second detection value interval and the third detection value interval of the data to be processed, the first detection layer of the detection model to be trained obtains the maximum value and minimum value of the first detection value interval, the maximum value and minimum value of the second detection value interval and the maximum value and minimum value of the third detection value interval respectively.

[0139] The maximum value of the first detection value interval, the maximum value of the second detection value interval, and the maximum value of the third detection value interval are compared, and the maximum value among the three is obtained; the minimum value of the first detection value interval, the minimum value of the second detection value interval, and the minimum value of the third detection value interval are compared, and the minimum value among the three is obtained. Based on the maximum value and the minimum value among the three, the target detection value interval is determined.

[0140] In the embodiments of the present specification, based on the maximum and minimum values ​​of the first detection value interval, the second detection value interval, and the third detection value interval, the target detection value interval of the data to be processed is accurately determined, thereby avoiding false alarms of abnormal values ​​in the subsequent detection of the detection data, improving the accuracy of the detection, and realizing subsequent effective detection of whether the detection data is abnormal data.

[0141] In the embodiments of this specification, if the initial object is processed only by the target detection value interval generated by the first detection module, it may cause the generated target object to have low accuracy and high false alarm rate during the detection process. This is because if the detection data is only detected from the dimension of whether it meets the target detection value interval; then the target object cannot detect the detection data that meets the target detection value interval but has errors in the density distribution, resulting in false alarms during the detection process, increasing the security risk of the enterprise. Based on this, this specification obtains the target density distribution value and target deviation threshold of the data to be processed through the second detection module of the initial object, and realizes detection from the dimension of density distribution. The specific implementation method includes: The detecting the data to be processed according to the second detection module of the initial object to obtain the target density distribution value and the target deviation threshold of the data to be processed includes: Dividing the data to be processed into at least two data subsets to be processed according to the second detection module of the initial object, and detecting each data subset to be processed to obtain a density distribution value of each data subset to be processed; A target density distribution value and a target deviation threshold of the data to be processed are determined based on the density distribution value of each subset of the data to be processed.

[0142] Specifically, the data processing platform inputs the data to be processed into the second detection module of the initial object, divides the data to be processed into at least two data subsets to be processed according to the second detection module of the initial object, and detects each data subset to be processed to obtain the density distribution value of each data subset to be processed; based on the density distribution value of each data subset to be processed, the target density distribution value and the target deviation threshold of the data to be processed are determined.

[0143] Continuing with the above example, a detailed description is given of how to determine the target density distribution value and the target deviation threshold of the data to be processed based on the density distribution value of each subset of the data to be processed.

[0144] The data processing platform inputs the determined sample training data into the second detection layer to be trained and tested. The second detection layer is a density detection sub-model generated by combining the LOF model with the HBOS model.

[0145] After receiving the sample training data, the density detection submodel divides the sample training data into at least two sample training data subsets through the LOF model, and detects the density distribution value of each sample training data subset through the HBOS model to obtain the density distribution value of each sample training data subset.

[0146] The density detection submodel determines the target density distribution value of the training sample data based on the density distribution value of each sample training data subset, and determines the target deviation threshold of the training sample data according to the target density distribution value of the training sample data and the density distribution value of each sample training data subset.

[0147] In the embodiment of this specification, the data to be processed is divided into at least two subsets of data to be processed by the second detection module, and the density distribution value of each subset of data to be processed is obtained; then the target density distribution value and the target deviation threshold of the data to be processed are determined based on the density distribution value of each subset of data to be processed. The performance of the generated target object is further improved, and the accuracy of determining whether the detected data is abnormal data through the target object is achieved, thereby avoiding the loss of Internet financial enterprises.

[0148] Further, the determining of the target density distribution value and the target deviation threshold of the data to be processed based on the density distribution value of each subset of the data to be processed includes: Calculating an average density distribution value based on the density distribution value of each of the at least two data subsets to be processed, and using the average density distribution value as a target density distribution value of the data to be processed; Determining a maximum density distribution value and a minimum density distribution value based on the density distribution value of each of the at least two data subsets to be processed; Based on the maximum density distribution value, the minimum density distribution value and the average density distribution value, a target deviation threshold of the data to be processed is determined.

[0149] Specifically, the second detection model of the initial object calculates the average density distribution value based on the density distribution value of each data subset to be processed in at least two data subsets to be processed, and uses the average density distribution value as the target density distribution value of the data to be processed; determines the maximum density distribution value and the minimum density distribution value based on the density distribution value of each data subset to be processed in at least two data subsets to be processed; and determines the target deviation threshold of the data to be processed based on the maximum density distribution value, the minimum density distribution value and the average density distribution value.

[0150] Continuing with the above example, the determination of the target density distribution value and the target deviation threshold of the data to be processed based on the density distribution value of each subset of the data to be processed is further described in detail.

[0151] After obtaining the density distribution value of each training sample data subset in at least two training sample data subsets, the density detection submodel of the detection model to be trained adds the density distribution value of each training sample data subset to obtain a total density distribution value, and divides the total density distribution value by the number of each training sample data subset to calculate an average density distribution value, and uses the average density distribution value as the target density distribution value of the training sample data.

[0152] After determining the target density distribution value of the training sample data, the density distribution values ​​of each training sample data subset in at least two training sample data are compared to determine the maximum density distribution value and the minimum density distribution value.

[0153] Subtract the maximum density distribution value from the average density distribution value to obtain the difference between the maximum density distribution value and the average density distribution value, and determine the first difference based on the difference, and the first difference is a positive number; when the difference between the maximum density distribution value and the average density distribution value is a positive number, the difference is determined as the first difference; when the difference between the maximum density distribution value and the average density distribution value is a negative number, obtain the opposite of the difference and determine the opposite as the first difference.

[0154] Subtract the minimum density distribution value from the average density distribution value to obtain the difference between the minimum density distribution value and the average density distribution value, and determine the second difference based on the difference, and the second difference is a positive number; when the difference between the maximum density distribution value and the average density distribution value is a positive number, the difference is determined as the second difference; when the difference between the maximum density distribution value and the average density distribution value is a negative number, obtain the opposite of the difference and determine the opposite as the second difference. The first difference value is added to the second difference value and the resultant difference is divided by 2, thereby obtaining a target deviation threshold value of the training sample data.

[0155] In the embodiment of this specification, the average density distribution value is calculated through the density distribution value of each of the at least two data subsets to be processed, as the target density distribution value of the data to be processed; and based on the average density distribution value and the density distribution value of each of the at least two data subsets to be processed, the maximum density distribution value and the minimum density distribution value are determined to determine the target deviation threshold of the data to be processed. The performance of the generated target object is further improved, the accuracy of determining whether the detection data is abnormal data through the target object is achieved, and the probability of security risks in the enterprise is reduced.

[0156] The data processing method provided in the present specification obtains the detection value of the detection data output by the first detection module and the density distribution value of the detection data output by the second detection module by inputting the acquired detection data into the first detection module and the second detection module; thereby, based on the detection value of the detection data and the density distribution value of the detection data, it is possible to quickly and accurately determine whether the detection data is abnormal data, thereby reducing the probability of security risks incurred by the enterprise and avoiding losses to the enterprise caused by security risks such as breach of contract and overdue payment.

[0157] The following combination Figure 5 , taking the application of the data processing method provided in this specification in the scenario of detecting whether the loan amount data is abnormal data as an example, the data processing method is further explained. Figure 5 A processing flow chart of a data processing method for detecting whether loan amount data is abnormal data provided in an embodiment of this specification is shown, which specifically includes the following steps: Step 502: Obtain training data.

[0158] The training data may be understood as the initial data in the above embodiment.

[0159] Specifically, the data processing platform obtains numeric data from the offline table as training data. The numeric data is a data type that represents quantity and can perform numerical operations. Numeric data consists of numbers, decimal points, positive and negative signs, and the letter E representing power, and the numerical precision is up to 16 bits. Numeric data is divided into 4 different types: numeric, floating point, double precision, and integer. The numeric data can be price, principal, penalty interest, interest rate, fee rate, etc. For example, when the numeric data is price, it can be 9320000 or 25000000; when the numeric data is interest rate, it can be 0.167332 or 0.1288.

[0160] Step 504: Determine whether the data conforms to a normal distribution.

[0161] Specifically, the distribution of data is diverse, and most numerical data conform to the normal distribution rules, or the numerical data can conform to the normal distribution after conversion. Therefore, it is necessary to judge the training data and use test methods such as Kolmogorov Smirnov (ks) test and Lillifors test to determine whether the obtained training data conforms to the normal distribution.

[0162] If yes, then execute step 508, step 510 and step 512 to train the to-be-trained detection model using training data that conforms to the normal distribution.

[0163] In the embodiment of this specification, step 508 and step 510 are in parallel with step 512. In practical applications, the execution sequence of step 508 and step 510 and step 512 can be adjusted according to actual needs, or step 508 and step 510 and step 512 can be executed simultaneously according to actual needs, which is not specifically limited in this specification.

[0164] If not, execute step 506.

[0165] Step 506: The data are confirmed to be in normal distribution through BOX-COX change and Log change.

[0166] Specifically, when the training data does not conform to the normal distribution, the training data that does not conform to the normal distribution is transformed by using BOX-COX transformation, Log transformation and other transformation methods to obtain training data that conforms to the normal distribution, and the training data is used as training input to train the detection model to be trained.

[0167] Step 508: Use the Z-Score method, Box method, or Grubbs method to determine the abnormal threshold interval.

[0168] The Box method may be understood as the box plot method in the above embodiment; and the abnormal threshold interval may be understood as the target detection value interval in the above embodiment.

[0169] Specifically, when the training data conforms to the normal distribution, the training data is input into the outlier detection model composed of KNN and OCSVM. The outlier detection model detects the training data using the Z-Score method to obtain the abnormal threshold interval detected by the Z-Score method (Z-score min , Z-score max ).

[0170] By using the Box method to detect the training data, the abnormal threshold interval detected by the Box method is obtained (Tukey min , Tukey max ,).

[0171] By using the Grubbs method to detect the training data, the anomaly threshold interval detected by the Grubbs method is obtained. min , Grubbs max ).

[0172] Combine the abnormal threshold intervals obtained by the above three detection methods and select [min(Z-score min ,Tukey min , Grubbs min ), max(Z-score max , Tukey max , Grubbs max )] as the total abnormal threshold interval.

[0173] Step 510: Use KNN and OCSVM to find outliers except the maximum / minimum end by density / distance or the like.

[0174] Specifically, after obtaining the total anomaly threshold interval, the outlier detection model composed of KNN and OCSVM is trained through training data. The outlier detection model determines the outliers in the training data based on the anomaly threshold interval, and finds outliers other than the maximum and minimum ends through density / distance and other methods.

[0175] When a value in the training data is within the abnormal threshold interval, the value is a normal value. When a value in the training data is outside the abnormal threshold interval, the value is an abnormal value.

[0176] Step 512: Determine the distribution density of the data by combining LOF and HBOS.

[0177] Specifically, when the training data conforms to the normal distribution, the training data is input into a density detection model composed of the LOF model and the HBOS model to train the density detection model.

[0178] The LOF model splits the training data into different local training data; the HBOS model is then used to calculate the relative frequency of the discretized training data in each local bin, thereby determining the density distribution of the training data and fixing the relative frequency threshold of the training data.

[0179] Step 514: Smart Model (Threshold / Density).

[0180] Among them, the intelligent model can understand the target object in the above embodiments.

[0181] Specifically, after the outlier detection model composed of KNN and OCSVM and the density detection model composed of LOF model and HBOS model are trained, the outlier detection model and the density detection model are output as an intelligent model, thereby obtaining an intelligent model for detecting whether numerical data is abnormal.

[0182] Step 516: Get new data.

[0183] The new data may be understood as the detection data in the above embodiment. In this embodiment, the loan amount data is the loan amount data.

[0184] Specifically, the obtained loan amount data is input into the intelligent model, and the intelligent model uses the outlier detection model composed of KNN and OCSVM, and the density detection model composed of LOF model and HBOS model to detect the loan amount data.

[0185] Among them, the loan amount data is tested through the outlier detection model, including: The loan amount data is tested through the outlier detection model to determine whether the loan amount data is within the abnormal threshold range.

[0186] If yes, it means that there is no abnormality in the obtained loan amount data.

[0187] If not, it means that the obtained loan amount data is abnormal.

[0188] Among them, the loan amount data is tested through the density detection model, including: The density detection model is used to determine whether the frequency deviation between the density distribution of the loan amount data and the density distribution of the training data exceeds the relative frequency threshold.

[0189] If yes, it means that the obtained loan amount data is abnormal.

[0190] If not, it means that there is no abnormality in the obtained loan amount data.

[0191] Step 518: Display the abnormal data to the user.

[0192] Specifically, after the intelligent model detects the acquired loan amount data, the data detected as abnormal is displayed to the user.

[0193] The data processing method provided in this specification obtains an outlier detection model and a density detection model of the newly input intelligent model, thereby quickly and accurately determining whether the loan amount data is abnormal data based on the outliers of the loan amount data and the density distribution of the loan amount data, thereby reducing the probability of security risks in the enterprise.

[0194] Corresponding to the above method embodiment, this specification also provides a data processing device embodiment, Figure 6 FIG. 1 shows a schematic diagram of the structure of a data processing device provided by an embodiment of the present specification. Figure 6 As shown, the device comprises: An acquisition module 602 is configured to input the acquired detection data into a first detection module and a second detection module of the target object, obtain a detection value of the detection data output by the first detection module, and obtain a density distribution value of the detection data output by the second detection module; The determination module 604 is configured to determine whether the detection data is abnormal data based on the detection value of the detection data and the density distribution value of the detection data.

[0195] In an embodiment of the present specification, the determining module 604 further includes: a determination submodule, configured to determine that the detection data is abnormal data when the detection value of the detection data is within the target detection value interval, or When the degree of deviation between the density distribution value of the detection data and the target density distribution value is greater than or equal to a target deviation threshold, the detection data is determined to be abnormal data.

[0196] In one embodiment of the present specification, the data processing device further includes: A data determination module is configured to determine the initial data as data to be processed when the obtained initial data meets a preset determination condition; an interval determination module, configured to detect the data to be processed according to the first detection module of the initial object, obtain the detection value of the data to be processed, and determine the target detection value interval of the data to be processed based on the detection value of the data to be processed; a density determination module, configured to detect the data to be processed according to the second detection module of the initial object, and obtain a target density distribution value and a target deviation threshold of the data to be processed; A generation module is configured to process the initial object according to the data to be processed and the target detection value interval, target density distribution value, and target deviation threshold of the data to be processed to generate a target object, wherein the target object includes the first detection module and the second detection module.

[0197] In one embodiment of the present specification, the data determination module further includes: A judgment module is configured to obtain initial data and judge whether the initial data satisfies a normal distribution. If yes, the initial data is determined as data to be processed; The conversion module is configured to, if not, convert the initial data according to the normal distribution rule, and if the conversion is successful, determine the converted initial data as the data to be processed.

[0198] In one embodiment of the present specification, the conversion module further includes: The candidate data acquisition module is configured to reacquire the candidate data in case of conversion failure and perform normal distribution determination on the candidate data.

[0199] In one embodiment of the present specification, the interval determination module further includes: A first interval determination submodule is configured to calculate the detection value of the to-be-processed data through a first detection value algorithm to obtain a first detection value interval of the to-be-processed data; A second interval determination submodule is configured to calculate the detection value of the to-be-processed data through a second detection value algorithm to obtain a second detection value interval of the to-be-processed data; A third interval determination submodule is configured to calculate the detection value of the to-be-processed data by a third detection value algorithm to obtain a third detection value interval of the to-be-processed data; The target interval determination module is configured to determine a target detection value interval of the data to be processed based on the first detection value interval, the second detection value interval, and the third detection value interval.

[0200] In one embodiment of the present specification, the target interval determination module further includes: a first target interval determination submodule, configured to obtain a maximum value and a minimum value of the first detection value interval, a maximum value and a minimum value of the second detection value interval, and a maximum value and a minimum value of the third detection value interval; The second target interval determination submodule is configured to determine a target detection value interval of the data to be processed based on the maximum value and the minimum value of the first detection value interval, the second detection value interval, and the third detection value interval.

[0201] In one embodiment of the present specification, the density determination module further includes: A first density determination submodule is configured to divide the data to be processed into at least two data subsets to be processed according to the second detection module of the initial object, and detect each data subset to be processed to obtain a density distribution value of each data subset to be processed; The second density determination submodule is configured to determine a target density distribution value and a target deviation threshold of the data to be processed based on the density distribution value of each subset of the data to be processed.

[0202] In an embodiment of the present specification, the second density determination submodule further includes: an average density calculation module, configured to calculate an average density distribution value based on the density distribution value of each of the at least two data subsets to be processed, and use the average density distribution value as a target density distribution value of the data to be processed; A maximum value and a minimum value determination module, configured to determine a maximum density distribution value and a minimum density distribution value based on the density distribution value of each of the at least two data subsets to be processed; The deviation threshold determination module is configured to determine a target deviation threshold of the data to be processed based on the maximum density distribution value, the minimum density distribution value and the average density distribution value.

[0203] In one embodiment of the present specification, the data determination module further includes: The sample data determination module is configured to determine the initial sample data as training sample data when the obtained initial sample data meets a preset determination condition.

[0204] In one embodiment of the present specification, the interval determination module further includes: The sample interval determination module is configured to detect the training sample data according to the first detection layer of the detection model to be trained, obtain the detection value of the training sample data, and determine the target detection value interval of the training sample data based on the detection value of the training sample data.

[0205] In one embodiment of the present specification, the density determination module further includes: The sample density determination module is configured to detect the training sample data according to the second detection layer of the detection model to be trained, and obtain the target density distribution value and target deviation threshold of the training sample data.

[0206] In one embodiment of the present specification, the generating module further includes: The model generation module is configured to train the detection model to be trained according to the training sample data and the target detection value interval, target density distribution value, and target deviation threshold of the training sample data to generate a trained detection model.

[0207] The data processing device provided in the present specification obtains the detection value of the detection data output by the first detection module and the density distribution value of the detection data output by the second detection module by inputting the acquired detection data into the first detection module and the second detection module of the target object; thereby, based on the detection value of the detection data and the density distribution value of the detection data, it is possible to quickly and accurately determine whether the detection data is abnormal data, thereby reducing the probability of security risks incurred by the enterprise.

[0208] The above is a schematic scheme of a data processing device of this embodiment. It should be noted that the technical scheme of the data processing device and the technical scheme of the above data processing method belong to the same concept, and the details of the technical scheme of the data processing device that are not described in detail can be referred to the description of the technical scheme of the above data processing method.

[0209] Figure 7 The block diagram of a computing device 700 according to an embodiment of the present specification is shown. The components of the computing device 700 include but are not limited to a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and the database 750 is used to store data.

[0210] The computing device 700 also includes an access device 740 that enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 740 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0211] In one embodiment of the present specification, the above components of the computing device 700 and Figure 7 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 7 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0212] The computing device 700 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 700 may also be a mobile or stationary server.

[0213] The processor 720 is used to execute computer executable instructions, which can implement any step of the data processing method when executed by the processor.

[0214] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above data processing method.

[0215] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which can implement any step of the data processing method when executed by a processor.

[0216] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above data processing method.

[0217] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0218] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0219] It should be noted that, for the convenience of description, the aforementioned method embodiments are all described as a series of action combinations, but those skilled in the art should be aware that this specification is not limited by the order of the actions described, because according to this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this specification.

[0220] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0221] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method, comprising: In the case where the obtained initial data meets the preset determination condition, the initial data is determined as data to be processed, and the initial data is data used to process the initial object; Detecting the data to be processed according to the first detection module of the initial object to obtain a detection value of the data to be processed, and determining a target detection value interval of the data to be processed based on the detection value of the data to be processed; Detecting the data to be processed according to the second detection module of the initial object to obtain a target density distribution value and a target deviation threshold of the data to be processed; The initial object is processed according to the data to be processed and the target detection value interval, target density distribution value, and target deviation threshold of the data to be processed to generate a target object, wherein the target object includes the first detection module and the second detection module, and the first detection module and the second detection module are used to determine whether the detection data is abnormal data, and the detection data is user feature information.

2. The data processing method according to claim 1, wherein when the obtained initial data meets the preset determination condition, the initial data is determined as the data to be processed, further comprising: Obtaining initial data, and determining whether the initial data satisfies a normal distribution, If yes, the initial data is determined as data to be processed; If not, the initial data is converted according to the normal distribution rule, and if the conversion is successful, the converted initial data is determined as the data to be processed.

3. The data processing method according to claim 2, after converting the initial data according to the normal distribution rule, further comprising: In case of conversion failure, the candidate data is reacquired and a normal distribution determination is performed on the candidate data.

4. The data processing method according to claim 1, wherein determining the target detection value interval of the data to be processed based on the detection value of the data to be processed comprises: Calculating the detection value of the data to be processed by a first detection value algorithm to obtain a first detection value interval of the data to be processed; Calculating the detection value of the data to be processed by a second detection value algorithm to obtain a second detection value interval of the data to be processed; Calculating the detection value of the data to be processed by a third detection value algorithm to obtain a third detection value interval of the data to be processed; Based on the first detection value interval, the second detection value interval, and the third detection value interval, a target detection value interval of the data to be processed is determined.

5. The data processing method according to claim 4, wherein determining the target detection value interval of the data to be processed based on the first detection value interval, the second detection value interval, and the third detection value interval comprises: Obtaining the maximum value and the minimum value of the first detection value interval, the maximum value and the minimum value of the second detection value interval, and the maximum value and the minimum value of the third detection value interval; Based on the maximum and minimum values ​​of the first detection value interval, the second detection value interval, and the third detection value interval, a target detection value interval of the data to be processed is determined.

6. The data processing method according to claim 5, wherein determining the target detection value interval of the data to be processed based on the maximum and minimum values ​​of the first detection value interval, the second detection value interval, and the third detection value interval comprises: Compare the maximum value of the first detection value interval, the maximum value of the second detection value interval, and the maximum value of the third detection value interval, and obtain the maximum value among the three; Compare the minimum value of the first detection value interval, the minimum value of the second detection value interval, and the minimum value of the third detection value interval, and obtain the minimum value among the three; The target detection value interval is determined based on the maximum value and the minimum value among the three.

7. The data processing method according to claim 1, wherein the second detection module of the initial object detects the data to be processed to obtain a target density distribution value and a target deviation threshold of the data to be processed, comprising: Dividing the data to be processed into at least two data subsets to be processed according to the second detection module of the initial object, and detecting each data subset to be processed to obtain a density distribution value of each data subset to be processed; A target density distribution value and a target deviation threshold of the data to be processed are determined based on the density distribution value of each subset of the data to be processed.

8. The data processing method according to claim 7, wherein determining the target density distribution value and target deviation threshold of the data to be processed based on the density distribution value of each subset of the data to be processed comprises: Calculating an average density distribution value based on the density distribution value of each of the at least two data subsets to be processed, and using the average density distribution value as a target density distribution value of the data to be processed; Determining a maximum density distribution value and a minimum density distribution value based on the density distribution value of each of the at least two data subsets to be processed; Based on the maximum density distribution value, the minimum density distribution value and the average density distribution value, a target deviation threshold of the data to be processed is determined.

9. The data processing method according to claim 1, wherein when the obtained initial data meets a preset determination condition, the initial data is determined as data to be processed, further comprising: When the obtained initial sample data meets the preset determination condition, the initial sample data is determined as training sample data.

10. The data processing method according to claim 9, wherein the first detection module of the initial object detects the data to be processed to obtain the detection value of the data to be processed, and determines the target detection value interval of the data to be processed based on the detection value of the data to be processed, further comprising: The training sample data is detected according to the first detection layer of the detection model to be trained to obtain the detection value of the training sample data, and the target detection value interval of the training sample data is determined based on the detection value of the training sample data.

11. The data processing method according to claim 10, wherein the second detection module of the initial object detects the data to be processed to obtain the target density distribution value and the target deviation threshold of the data to be processed, further comprising: The training sample data is detected according to the second detection layer of the detection model to be trained to obtain a target density distribution value and a target deviation threshold of the training sample data.

12. The data processing method according to claim 11, wherein the initial object is processed according to the data to be processed and the target detection value interval, target density distribution value, and target deviation threshold of the data to be processed to generate the target object, further comprising: The detection model to be trained is trained according to the training sample data and the target detection value interval, target density distribution value, and target deviation threshold of the training sample data to generate a trained detection model.

13. A data processing device, comprising: A data determination module is configured to determine the obtained initial data as data to be processed when the obtained initial data meets a preset determination condition, wherein the initial data is data used to process the initial object; an interval determination module, configured to detect the data to be processed according to the first detection module of the initial object, obtain the detection value of the data to be processed, and determine the target detection value interval of the data to be processed based on the detection value of the data to be processed; a density determination module, configured to detect the data to be processed according to the second detection module of the initial object, and obtain a target density distribution value and a target deviation threshold of the data to be processed; A generation module is configured to process the initial object according to the data to be processed and the target detection value interval, target density distribution value, and target deviation threshold of the data to be processed to generate a target object, wherein the target object includes the first detection module and the second detection module, the first detection module and the second detection module are used to determine whether the detection data is abnormal data, and the detection data is user feature information.

14. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the data processing method according to any one of claims 1 to 12 are implemented.

15. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 12.

Images