Abnormal data evaluation method, device and equipment and storage medium thereof
By collecting and encrypting data from multiple data sources, combining security addition and anomaly detection models, the problems of insecurity in data protection and incomplete evaluation in abnormal data evaluation are solved, and more scientific and comprehensive abnormal detection is achieved.
Patent Information
- Application Number
- CN202411072968.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-06
- Publication Date
- 2024-11-12
AI Technical Summary
In the evaluation and detection of abnormal data, the existing technology has problems such as insufficient data protection and insufficient comprehensive and scientific evaluation.
Target evaluation data is collected from multiple data sources, encrypted and disturbed processing is performed, and initial integration is performed using a secure addition method. After cleaning and pre-processing, it is classified and processed according to the preset classification dimensions, and the data is input into the pre-trained abnormal detection model. , the abnormal data is identified by the detection component, and the outlier value is finally calculated based on the output results and the preset estimation algorithm.
While ensuring data security, more comprehensive and scientific abnormality detection is achieved, improving the accuracy and security of data evaluation.
Smart Images

Figure CN118940186A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of financial data processing, and in particular to an abnormal data evaluation method, device, equipment and storage medium thereof. Background Art
[0002] In the era of big data, the evaluation and detection of abnormal data can detect risks in advance and avoid them, ensuring the security of subsequent business processing and the stability of business systems.
[0003] As for whether there are potential risks in business systems, traditional evaluation and analysis decision-making systems mainly rely on the subjective judgment of experts, which is difficult to be scientific and objective. In addition, due to the multiple data sources, complex data structure, privacy protection, data imbalance and other issues, the analysis is difficult to a certain extent, which brings great challenges to traditional evaluation and analysis decision-making methods. At present, there are still problems in abnormal data evaluation and detection, such as insufficient data protection security and insufficient comprehensive and scientific evaluation. Summary of the invention
[0004] The purpose of the embodiments of the present application is to propose an abnormal data evaluation method, device, equipment and storage medium thereof to solve the problems in the prior art of abnormal data evaluation and detection, such as insufficient data protection security and insufficient comprehensive and scientific evaluation.
[0005] In order to solve the above technical problems, the present application embodiment provides an abnormal data evaluation method, which adopts the following technical solution:
[0006] An abnormal data evaluation method comprises the following steps:
[0007] Collect target assessment data from multiple data sources;
[0008] Encrypting and perturbing the target evaluation data to obtain data to be integrated;
[0009] Use the security addition method to preliminarily integrate all the data to be integrated to generate preliminary integrated evaluation data;
[0010] Cleaning and preprocessing the preliminarily integrated evaluation data, and classifying the cleaned and preprocessed evaluation data according to a preset classification dimension to obtain classified data;
[0011] Inputting the classified data into a pre-trained anomaly detection model to obtain an output result of the anomaly detection model;
[0012] According to the output result and a preset valuation algorithm, the outlier value of the target evaluation data is calculated.
[0013] Furthermore, the multiple data sources include a revenue and expenditure transaction data source, a tax management data source, a judicial announcement data source, a market supervision data source, and a credit inquiry data source. The step of encrypting and perturbing the target evaluation data to obtain the data to be integrated specifically includes:
[0014] identifying the source of said target assessment data;
[0015] According to different sources of the target evaluation data, different encryption technologies are selected for encryption processing;
[0016] The encrypted target evaluation data is perturbed using a preset perturbation function to obtain the perturbed data as the data to be integrated, wherein the preset perturbation function obeys Laplace distribution.
[0017] Furthermore, the target evaluation data includes basic enterprise information data, enterprise supervision data and enterprise transaction data. The step of preliminarily integrating all the data to be integrated in a secure addition manner to generate preliminarily integrated evaluation data specifically includes:
[0018] Identify the categories of data currently to be integrated;
[0019] If the current data to be integrated is basic information data of an enterprise, a clustering method is adopted to integrate all basic information data of the enterprise collected from multiple data sources into a preset first data set, wherein the basic information data of the enterprise includes the address information of the enterprise, the basic information of the commercial subject and senior executives of the enterprise, and the registered capital information of the enterprise;
[0020] If the current data to be integrated is enterprise supervision data, a clustering method is used to integrate all enterprise supervision data collected from multiple data sources into a preset second data set, wherein the enterprise supervision data includes the business scope declared by the enterprise, the filling status of the enterprise annual report, and the handling status of the evaluation procedures;
[0021] If the current data to be integrated is enterprise transaction data, all enterprise transaction data collected from multiple data sources are time-serialized and integrated in chronological order of transaction time, and all enterprise transaction data collected from multiple data sources are clustered and integrated according to transaction subjects, wherein the enterprise transaction data includes transaction data generated by the enterprise during the provision of business services.
[0022] Furthermore, the step of classifying the cleaned and preprocessed evaluation data according to a preset classification dimension to obtain classified data specifically includes:
[0023] Classify the cleaned and preprocessed evaluation data according to a preset first classification strategy to obtain first classified data, wherein the first classification strategy includes classifying the data based on whether the data is enterprise transaction data;
[0024] The first classified data is classified again based on a second classification strategy to obtain second classified data as the target classified data, wherein the second classification strategy includes classification based on whether the data has time series.
[0025] Furthermore, the pre-trained anomaly detection model includes a first detection component and a second detection component, and the step of inputting the classified data into the pre-trained anomaly detection model to obtain an output result of the anomaly detection model specifically includes:
[0026] According to the preset classification mark, identify whether the current classification data has time series;
[0027] If the current classified data does not have time series characteristics, then determining abnormal data in the current classified data through the first detection component;
[0028] If the current classified data has time series characteristics, the abnormal data in the current classified data is determined by the second detection component.
[0029] Furthermore, the step of determining abnormal data in the current classified data through the first detection component specifically includes:
[0030] Performing mathematical assignment processing on all data in the current classification data, wherein the mathematical assignment processing includes assigning a uniform mathematical value to data that is identical or has a similarity that reaches a preset threshold, and assigning other numerical values different from the mathematical value to data that has a similarity that does not reach the preset threshold;
[0031] Obtain the mathematical values corresponding to all data in the current classification data after mathematical assignment processing;
[0032] Z= X-μ
[0033] The preset algorithm formula: σ is used to calculate the Z score value corresponding to all the data in the current classification data, where X represents the mathematical value corresponding to the target data in the current classification data, μ represents the average mathematical value corresponding to all the data in the current classification data, and σ represents the standard deviation of the mathematical value corresponding to all the data in the current classification data;
[0034] By comparison, determine whether the Z score value corresponding to the current data is greater than the preset abnormal threshold;
[0035] If the Z score value corresponding to the current data is greater than the preset abnormal threshold, the current data is abnormal data.
[0036] Furthermore, the step of determining abnormal data in the current classified data by the second detection component specifically includes:
[0037] The frequency domain feature extraction algorithm is used to extract features from the data in the current classification data;
[0038] The extracted features are sorted according to the time series to obtain the feature vectors corresponding to all the data in the current classification data;
[0039] For the feature vectors corresponding to all the data in the current classification data, the Euclidean distance algorithm is used to compare them pairwise to obtain the feature distance;
[0040] Combining the feature distance and a preset density-based clustering algorithm, filtering out unclustered data, wherein the preset density-based clustering algorithm includes a DBSCAN algorithm;
[0041] The unclassified data are marked as abnormal data in the current classified data.
[0042] Furthermore, the step of calculating the outlier value of the target evaluation data according to the output result and the preset valuation algorithm specifically includes:
[0043] By using the output results, the data entry quantity of all abnormal data is counted;
[0044] Using a preset valuation algorithm, calculate the proportion of the data items of all abnormal data in the items of the preliminary integrated evaluation data, wherein the valuation algorithm includes a probability algorithm;
[0045] The proportion is used as the abnormal value, or a pre-divided abnormal level corresponding to the proportion is identified, and the abnormal level is used as the abnormal value.
[0046] In order to solve the above technical problems, the embodiment of the present application further provides an abnormal data evaluation device, which adopts the following technical solution:
[0047] An abnormal data evaluation device, comprising:
[0048] A target data collection module is used to collect target evaluation data from multiple data sources;
[0049] An encryption and disturbance processing module, used for encrypting and disturbing the target evaluation data to obtain data to be integrated;
[0050] The data preliminary integration module is used to perform preliminary integration of all the data to be integrated by using a secure addition method to generate preliminary integrated evaluation data;
[0051] A data classification processing module, used to clean and preprocess the preliminarily integrated evaluation data, and to classify the cleaned and preprocessed evaluation data according to a preset classification dimension to obtain classified data;
[0052] An anomaly detection module, used to input the classified data into a pre-trained anomaly detection model to obtain an output result of the anomaly detection model;
[0053] The outlier calculation module is used to calculate the outlier value of the target evaluation data according to the output result and a preset valuation algorithm.
[0054] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:
[0055] A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned abnormal data evaluation method when executing the computer-readable instructions.
[0056] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:
[0057] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the abnormal data evaluation method as described above.
[0058] Compared with the prior art, the embodiments of the present application have the following beneficial effects:
[0059] The abnormal data evaluation method described in the embodiment of the present application collects target evaluation data from multiple data sources; encrypts and perturbs the business data to obtain data to be integrated; performs preliminary integration of all data to be integrated to generate preliminary integrated evaluation data; cleans and preprocesses the preliminary integrated evaluation data, and classifies the cleaned and preprocessed business data according to preset classification dimensions to obtain classified data; inputs the classified data into an anomaly detection model to obtain an output result; calculates the abnormal value of the target evaluation data based on the output result and a preset valuation algorithm. By encrypting and perturbating the target evaluation data provided by multiple data sources, and using an anomaly detection model to identify abnormal task data, it is ensured that anomaly detection can be performed comprehensively in combination with multiple data sources while ensuring data security, which is more comprehensive and scientific. BRIEF DESCRIPTION OF THE DRAWINGS
[0060] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0061] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;
[0062] Figure 2 A flowchart of an embodiment of an abnormal data evaluation method according to the present application;
[0063] Figure 3 yes Figure 2 A flowchart of a specific embodiment of step 202 is shown;
[0064] Figure 4 yes Figure 2 A flowchart of a specific embodiment of step 203 is shown;
[0065] Figure 5 yes Figure 2 A flowchart of a specific embodiment of step 204 is shown;
[0066] Figure 6 yes Figure 2 A flowchart of a specific embodiment of step 205 is shown;
[0067] Figure 7 yes Figure 6 A flowchart of a specific embodiment of step 603 is shown;
[0068] Figure 8 A schematic structural diagram of an embodiment of an abnormal data evaluation device according to the present application;
[0069] Fig. 9 A schematic structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION
[0070] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.
[0071] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0072] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.
[0073] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.
[0074] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.
[0075] The terminal devices 101, 102, and 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.
[0076] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .
[0077] It should be noted that the abnormal data evaluation method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the abnormal data evaluation device is generally arranged in the server / terminal device.
[0078] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.
[0079] Continue to refer Figure 2 , shows a flow chart of an embodiment of an abnormal data evaluation method according to the present application. The abnormal data evaluation method comprises the following steps:
[0080] Step 201, collecting target evaluation data from multiple data sources, wherein the multiple data sources include revenue and expenditure transaction data source, tax management data source, judicial announcement data source, market supervision data source, and credit inquiry data source, and the target evaluation data includes enterprise basic information data, enterprise supervision data, and enterprise transaction data.
[0081] In this embodiment, the target evaluation data is collected from multiple data sources, the basic information data of the enterprise includes the address information of the residence or business premises declared by the enterprise, the amount of registered capital declared by the enterprise, the basic information of the business entity and senior executives of the enterprise, the enterprise supervision data includes the matters that require approval related to the business scope of the enterprise, the filling of the annual report of the enterprise, the credit information of the business entity and senior executives of the enterprise, and the application and replacement of various business certificates, such as: the application of tax registration certificate, organization code certificate, and the enterprise transaction data includes the transaction data generated by the enterprise during the provision of business services, such as: bank transaction flow data, business settlement data calculated by the company's financial personnel, etc. By collecting target evaluation data from multiple data sources, the evaluation of abnormal operations of the enterprise is made more comprehensive, helping financial institutions or regulatory agencies to comprehensively evaluate the business conditions of the enterprise from the perspective of multi-dimensional information. It can be considered that the target evaluation data described in this application is the pre-acquired enterprise business data, and the enterprise business data includes basic information data of the enterprise, enterprise supervision data and enterprise transaction data.
[0082] Step 202: encrypt and perturb the target evaluation data to obtain data to be integrated.
[0083] The target evaluation data is encrypted and disturbed to obtain the data to be integrated. The purpose is to make the evaluation data used in the subsequent enterprise business situation evaluation be encrypted and disturbed, thereby increasing data security. At the same time, data disturbance enriches the amount of evaluation data, making the enterprise business evaluation more accurate.
[0084] Continue to refer Figure 3 , Figure 3 yes Figure 2 The flowchart of a specific embodiment of step 202 shown includes:
[0085] Step 301, identifying the source of the target evaluation data;
[0086] Step 302, selecting different encryption technologies for encryption processing according to different sources of the target evaluation data;
[0087] Specifically, the privacy data in the target evaluation data are encrypted to obtain encrypted data, wherein the encryption of the privacy data in the target evaluation data comprises: selecting different encryption technologies for encryption according to different sources of the target evaluation data;
[0088] In this embodiment, firstly, the business data is not fully encrypted in the encryption processing, but only the privacy data is encrypted, thereby avoiding the leakage of privacy data. Secondly, according to the different sources of the target evaluation data, different encryption technologies are selected to encrypt the privacy data in the business data. Specifically, for example, different encryption means are set for the income and expenditure transaction data source, the tax management data source, the judicial announcement data source, the market supervision data source, and the credit inquiry data source. The encryption technology provided by the different data sources can be directly used to encrypt the privacy data of the corresponding data source, or a variety of encryption processing technologies can be pre-sorted. According to the different business data sources, selective encryption can be performed from a variety of encryption processing technologies. The purpose of this is to encrypt the privacy data of different data sources using different encryption means, thereby ensuring that once a data leakage incident occurs in a certain data source, it will not affect the data of other data sources, thereby enhancing data security.
[0089] In addition, when different encryption technologies are selected for encryption processing according to the different sources of the target evaluation data, security weights can be set for different encryption technologies, and selection weights can be set for different data sources, wherein the security weight refers to the security level of the encryption technology, and the selection weight refers to the screening priority level of the encryption technology for different data sources. This approach can better ensure that data with higher security requirements is matched to encryption technologies with higher security levels as much as possible.
[0090] Step 303: Use a preset perturbation function to perform data perturbation on the encrypted target evaluation data to obtain the perturbed data as the data to be integrated, wherein the preset perturbation function obeys the Laplace distribution.
[0091] In this embodiment, the target evaluation data after encryption processing is summarized. This is because direct encryption processing does not introduce noise data, while the disturbed data is the data after encryption. Noise data is introduced into the encrypted data on the basis of encryption, which enriches the data content and improves the encryption security of the data.
[0092] Specifically, the purpose of the data perturbation is to achieve secondary protection of privacy data by perturbing the encrypted business data, which is equivalent to reintroducing the Laplace mechanism on the basis of the first encryption to generate noise data, and to generate noise data similar to the encrypted business data, thereby increasing the security of the data to be integrated. For the perturbation function, it can be a function that obeys the Laplace distribution.
[0093] Step 203, using a secure addition method to perform preliminary integration on all the data to be integrated, and generate preliminary integrated evaluation data.
[0094] In this embodiment, the data to be integrated includes three categories, corresponding to enterprise basic information data, enterprise supervision data and enterprise transaction data respectively.
[0095] Specifically, the security addition method means that when all the data to be integrated are initially integrated, the acquisition, transmission and integration of the data to be integrated are all in a safe state, thereby increasing data security. Then, the evaluation data of the initial integration is performed according to the category of the data to be integrated, so as to achieve the initial sorting of the complex business data of the enterprise.
[0096] Continue to refer Figure 4 , Figure 4 yes Figure 2 The flowchart of a specific embodiment of step 203 shown includes:
[0097] Step 401, identifying the category of the data to be integrated;
[0098] Step 402: if the current data to be integrated is enterprise basic information data, then all enterprise basic information data collected from multiple data sources are integrated into a preset first data set by clustering, wherein the enterprise basic information data includes enterprise address information, enterprise business entity and senior executives basic information, and enterprise registered capital information;
[0099] Step 403: if the current data to be integrated is enterprise supervision data, all enterprise supervision data collected from multiple data sources are integrated into a preset second data set by clustering, wherein the enterprise supervision data includes the business scope declared by the enterprise, the filling of the enterprise annual report, and the handling of the enterprise business procedures;
[0100] Step 404, if the current data to be integrated is enterprise transaction data, then all enterprise transaction data collected from multiple data sources are time-serialized and integrated in chronological order of transaction time, and all enterprise transaction data collected from multiple data sources are clustered and integrated according to transaction subjects, wherein the enterprise transaction data includes transaction data generated by the enterprise during the provision of business services.
[0101] Step 204 , cleaning and preprocessing the initially integrated evaluation data, and classifying the cleaned and preprocessed evaluation data according to preset classification dimensions to obtain classified data.
[0102] In this embodiment, the preliminarily integrated evaluation data is cleaned and preprocessed, including processing missing values, outliers and duplicate values to ensure the accuracy and completeness of the data, and preprocessing operations such as standardization and normalization are performed on the data to facilitate subsequent analysis and calculation.
[0103] Continue to refer Figure 5 , Figure 5 yes Figure 2 The flowchart of a specific embodiment of step 204 shown includes:
[0104] Step 501, classifying the cleaned and preprocessed evaluation data according to a preset first classification strategy to obtain first classified data, wherein the first classification strategy includes classifying the data according to whether the data is enterprise transaction data;
[0105] Specifically, by classifying the data according to whether it is enterprise transaction data, the above-mentioned enterprise basic information data and enterprise supervision data can be separated from the enterprise transaction data, which is convenient for subsequent analysis and processing.
[0106] In this embodiment, the enterprise basic information data and the enterprise supervision data are pre-listed and sorted, and data that cannot be classified into the enterprise basic information data and the enterprise supervision data according to the list sorting result all belong to the enterprise transaction data.
[0107] Step 502, reclassifying the first classified data based on a second classification strategy to obtain second classified data as the target classified data, wherein the second classification strategy includes classifying the data based on whether there is a time series.
[0108] In this embodiment, the data is classified according to whether it has time series, so that the data with time series can be separated from the data without time series, which is convenient for subsequent analysis using different analysis methods.
[0109] Specifically, a time series identification form is pre-arranged, and when determining whether data has time series, it is possible to directly distinguish whether the data has time series based on the identification form.
[0110] In this embodiment, before executing the step of reclassifying the first classified data based on the second classification strategy to obtain the second classified data as the target classified data, the method also includes: classifying the enterprise transaction data according to different transaction categories, wherein the transaction categories include business categories, financial categories, tax categories, industrial and commercial categories, water and electricity categories, etc.
[0111] Step 205: input the classified data into the pre-trained anomaly detection model to obtain the output result of the anomaly detection model.
[0112] Among them, the pre-trained anomaly detection model includes a first detection component and a second detection component, the first detection component is used to perform abnormal operation detection on non-time series classification data, and the second detection component is used to perform abnormal operation detection on time series classification data.
[0113] Specifically, the anomaly detection model includes a detection model based on a progressive neural network, wherein, before executing the step of inputting the classified data into the pre-trained anomaly detection model to obtain the output result of the anomaly detection model, the method also includes: obtaining positive and negative sample data, the positive and negative sample data including labeled abnormal business data and normal business data, inputting the positive and negative sample data into the detection model based on the progressive neural network, and training the detection model based on the progressive neural network so that the detection model can learn the Z-score analysis steps and the time series analysis method, and can determine the abnormal business data in the target classified data by the Z-score method and the time series analysis method respectively.
[0114] Continue to refer Figure 6 , Figure 6 yes Figure 2 The flowchart of a specific embodiment of step 205 shown includes:
[0115] Step 601, identifying whether the current classification data has time series according to the preset classification identifier;
[0116] Step 602: if the current classified data does not have time series characteristics, then the first detection component is used to determine abnormal data in the current classified data;
[0117] In this embodiment, during the training phase of the anomaly detection model, the Z-score analysis method is pre-deployed to the first detection component, and the first detection component is trained to learn the Z-score analysis method.
[0118] In this embodiment, the step of determining abnormal data in the current classified data through the first detection component specifically includes: performing mathematical assignment processing on all data in the current classified data, wherein the mathematical assignment processing includes assigning a uniform mathematical value to data that are identical or have a similarity that reaches a preset threshold, and assigning other numerical values different from the mathematical value to data whose similarity does not reach the preset threshold; obtaining the mathematical values corresponding to all data in the current classified data after the mathematical assignment processing; and using a preset algorithm formula: Calculate the Z score value corresponding to all the data in the current classified data, where X represents the mathematical value corresponding to the target data in the current classified data, μ represents the average value of the mathematical values corresponding to all the data in the current classified data, and σ represents the standard deviation of the mathematical values corresponding to all the data in the current classified data; by comparison, determine whether the Z score value corresponding to the current data is greater than the preset abnormal threshold; if the Z score value corresponding to the current data is greater than the preset abnormal threshold, the current data is abnormal data.
[0119] Specifically, for example, the business location or registered address information in the basic information data of an enterprise is not time series data, and the registered address information obtained from multiple data sources may be different. Suppose that the registered address information is obtained from 100 data sources, and 20 pieces of address information are different from the other 80. The Z-score analysis method can be used to determine that the 20 pieces of address information are abnormal business data obtained.
[0120] Step 603: If the current classified data has time series characteristics, abnormal data in the current classified data is determined through the second detection component.
[0121] In this embodiment, during the training stage of the anomaly detection model, the frequency domain feature extraction algorithm, the Euclidean distance algorithm and the density-based clustering algorithm are pre-deployed to the second detection component, and through training, the second detection component learns the time series analysis method that combines the frequency domain feature extraction algorithm, the Euclidean distance algorithm and the density-based clustering algorithm.
[0122] Continue to refer Figure 7 , Figure 7 yes Figure 6 The flowchart of a specific embodiment of step 603 shown includes:
[0123] Step 701, using a frequency domain feature extraction algorithm to extract features from the data in the current classification data;
[0124] In this embodiment, the frequency domain feature extraction algorithm may use a power spectrum density method to obtain the feature distribution of data in the current classification data.
[0125] Step 702, sorting the extracted features according to time series to obtain feature vectors corresponding to all data in the current classification data;
[0126] Step 703, using the Euclidean distance algorithm to compare the feature vectors corresponding to all the data in the current classification data in pairs, to obtain the feature distance;
[0127] Step 704, combining the feature distance and a preset density-based clustering algorithm to filter out unclustered data, wherein the preset density-based clustering algorithm includes a DBSCAN algorithm;
[0128] In this embodiment, the DBSCAN algorithm determines whether a certain classified data is a noise point based on the feature distance, a pre-set clustering radius, and a pre-set minimum number of clusters. If so, the classified data corresponding to the noise point is abnormal operating data. Specifically, in this embodiment, the noise point refers to data that has not been clustered by the DBSCAN algorithm.
[0129] Step 705: Mark the unclustered data as abnormal data in the current classified data.
[0130] Step 206: Calculate the outlier value of the target evaluation data based on the output result and a preset valuation algorithm.
[0131] In this embodiment, the step of calculating the outlier value of the target evaluation data based on the output result and the preset valuation algorithm specifically includes: counting the number of data entries of all abnormal data through the output result; using the preset valuation algorithm to calculate the proportion of the number of data entries of all abnormal data in the preliminary integrated evaluation data, wherein the valuation algorithm includes a probabilistic algorithm; using the proportion as the outlier value, or identifying the pre-divided anomaly level corresponding to the proportion, and using the anomaly level as the outlier value.
[0132] In this embodiment, after obtaining the abnormal value, the method also includes: determining whether the abnormal value reaches a preset reporting threshold; if the abnormal value reaches the preset reporting threshold, sending an alarm message to a monitoring terminal; if the abnormal value does not reach the preset reporting threshold, continuing to evaluate abnormal data, wherein the monitoring terminal includes an internal monitoring platform of the company and a monitoring platform of the regulatory department.
[0133] By analyzing the business operation data provided by multiple data sources, using comprehensive analysis methods and anomaly detection models to identify the abnormal operation conditions of target companies, and using evaluation methods to determine the abnormal operation values of companies, it is more comprehensive and scientific.
[0134] This application collects target evaluation data from multiple data sources; encrypts and perturbs the business data to obtain data to be integrated; performs preliminary integration of all data to be integrated to generate preliminary integrated evaluation data; cleans and preprocesses the preliminary integrated evaluation data, and classifies the cleaned and preprocessed business data according to preset classification dimensions to obtain classified data; inputs the classified data into an anomaly detection model to obtain output results; calculates the abnormal value of the target evaluation data based on the output results and a preset valuation algorithm. By encrypting and perturbing the target evaluation data provided by multiple data sources, and using an anomaly detection model to identify abnormal task data, it is ensured that anomaly detection can be carried out comprehensively in combination with multiple data sources while ensuring data security, making it more comprehensive and scientific.
[0135] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0136] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0137] In the embodiment of the present application, target evaluation data is collected from multiple data sources; business data is encrypted and disturbed to obtain data to be integrated; all data to be integrated are preliminarily integrated to generate preliminarily integrated evaluation data; the preliminarily integrated evaluation data is cleaned and preprocessed, and the cleaned and preprocessed business data is classified according to the preset classification dimension to obtain classified data; the classified data is input into the anomaly detection model to obtain the output result; the outlier value of the target evaluation data is calculated according to the output result and the preset valuation algorithm. By encrypting and disturbing the target evaluation data provided by multiple data sources, and using the anomaly detection model to identify abnormal task data, it is ensured that anomaly detection can be carried out comprehensively in combination with multiple data sources under the condition of data security, which is more comprehensive and scientific.
[0138] Further references Figure 8 , as a response to the above Figure 2 The present application provides an embodiment of an abnormal data evaluation device, and the device embodiment is similar to Figure 2Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0139] like Figure 8 As shown, the abnormal data evaluation device 800 described in this embodiment includes: a target data acquisition module 801, an encryption and disturbance processing module 802, a data preliminary integration module 803, a data classification processing module 804, an abnormal operation detection module 805 and an abnormal operation evaluation module 806. Among them:
[0140] The target data collection module 801 is used to collect target evaluation data from multiple data sources;
[0141] The encryption and disturbance processing module 802 is used to encrypt and disturb the target evaluation data to obtain data to be integrated;
[0142] The data preliminary integration module 803 is used to perform preliminary integration of all the data to be integrated by using a secure addition method to generate preliminary integrated evaluation data;
[0143] The data classification processing module 804 is used to clean and preprocess the preliminarily integrated evaluation data, and classify the cleaned and preprocessed evaluation data according to a preset classification dimension to obtain classified data;
[0144] Anomaly detection module 805, used to input the classified data into a pre-trained anomaly detection model to obtain an output result of the anomaly detection model;
[0145] The outlier calculation module 806 is used to calculate the outlier value of the target evaluation data based on the output result and a preset valuation algorithm.
[0146] This application collects target evaluation data from multiple data sources; encrypts and perturbs the business data to obtain data to be integrated; performs preliminary integration of all data to be integrated to generate preliminary integrated evaluation data; cleans and preprocesses the preliminary integrated evaluation data, and classifies the cleaned and preprocessed business data according to preset classification dimensions to obtain classified data; inputs the classified data into an anomaly detection model to obtain output results; calculates the abnormal value of the target evaluation data based on the output results and a preset valuation algorithm. By encrypting and perturbing the target evaluation data provided by multiple data sources, and using an anomaly detection model to identify abnormal task data, it is ensured that anomaly detection can be carried out comprehensively in combination with multiple data sources while ensuring data security, making it more comprehensive and scientific.
[0147] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).
[0148] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.
[0149] To solve the above technical problems, the present application also provides a computer device. Fig. 9 , Fig. 9 This is a basic structural block diagram of the computer device in this embodiment.
[0150] The computer device 9 includes a memory 9a, a processor 9b, and a network interface 9c that are interconnected through a system bus. It should be noted that the figure only shows a computer device 9 having components 9a-9c, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital processors (DSPs), embedded devices, etc.
[0151] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.
[0152] The memory 9a includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 9a can be an internal storage unit of the computer device 9, such as a hard disk or memory of the computer device 9. In other embodiments, the memory 9a can also be an external storage device of the computer device 9, such as a plug-in hard disk equipped on the computer device 9, a smart memory card (Smart Med i aCard, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 9a can also include both the internal storage unit of the computer device 9 and its external storage device. In this embodiment, the memory 9a is generally used to store the operating system and various application software installed on the computer device 9, such as computer-readable instructions of the abnormal data evaluation method, etc. In addition, the memory 9a can also be used to temporarily store various types of data that have been output or are to be output.
[0153] The processor 9b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 9b is generally used to control the overall operation of the computer device 9. In the present embodiment, the processor 9b is used to run the computer-readable instructions or process data stored in the memory 9a, such as the computer-readable instructions for running the abnormal data evaluation method.
[0154] The network interface 9c may include a wireless network interface or a wired network interface. The network interface 9c is generally used to establish a communication connection between the computer device 9 and other electronic devices.
[0155] The computer device proposed in this embodiment belongs to the field of financial data processing technology. This application collects target evaluation data from multiple data sources; encrypts and perturbs the business data to obtain data to be integrated; performs preliminary integration of all data to be integrated to generate preliminary integrated evaluation data; cleans and preprocesses the preliminary integrated evaluation data, and classifies the cleaned and preprocessed business data according to preset classification dimensions to obtain classified data; inputs the classified data into an anomaly detection model to obtain an output result; calculates the outlier value of the target evaluation data based on the output result and a preset valuation algorithm. By encrypting and perturbating the target evaluation data provided by multiple data sources, and using an anomaly detection model to identify abnormal task data, it is ensured that anomaly detection can be performed comprehensively in combination with multiple data sources while ensuring data security, which is more comprehensive and scientific.
[0156] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by a processor to enable the processor to perform the steps of the abnormal data evaluation method as described above.
[0157] The computer-readable storage medium proposed in this embodiment belongs to the field of financial data processing technology. This application collects target evaluation data from multiple data sources; encrypts and perturbs the business data to obtain data to be integrated; performs preliminary integration of all data to be integrated to generate preliminary integrated evaluation data; cleans and preprocesses the preliminary integrated evaluation data, and classifies the cleaned and preprocessed business data according to preset classification dimensions to obtain classified data; inputs the classified data into an anomaly detection model to obtain an output result; calculates the outlier value of the target evaluation data based on the output result and a preset valuation algorithm. By encrypting and perturbating the target evaluation data provided by multiple data sources, and using an anomaly detection model to identify abnormal task data, it is ensured that anomaly detection can be performed comprehensively in combination with multiple data sources while ensuring data security, which is more comprehensive and scientific.
[0158] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.
[0159] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.
Claims
1. A method for evaluating abnormal data, characterized in that: The steps include: Collect target assessment data from multiple data sources; Encrypting and perturbing the target evaluation data to obtain data to be integrated; Use the security addition method to preliminarily integrate all the data to be integrated to generate preliminary integrated evaluation data; Cleaning and preprocessing the preliminarily integrated evaluation data, and classifying the cleaned and preprocessed evaluation data according to a preset classification dimension to obtain classified data; Inputting the classified data into a pre-trained anomaly detection model to obtain an output result of the anomaly detection model; According to the output result and a preset valuation algorithm, the outlier value of the target evaluation data is calculated.
2. The abnormal data evaluation method according to claim 1, characterized in that: The multiple data sources include a revenue and expenditure transaction data source, a tax management data source, a judicial announcement data source, a market supervision data source, and a credit inquiry data source; the step of encrypting and perturbing the target evaluation data to obtain the data to be integrated specifically includes: identifying the source of said target assessment data; According to different sources of the target evaluation data, different encryption technologies are selected for encryption processing; The encrypted target evaluation data is perturbed using a preset perturbation function to obtain the perturbed data as the data to be integrated, wherein the preset perturbation function obeys Laplace distribution.
3. The abnormal data evaluation method according to claim 1 or 2, characterized in that: The business data includes basic enterprise information data, enterprise supervision data and enterprise transaction data; The step of preliminarily integrating all the data to be integrated by using the security addition method to generate preliminarily integrated evaluation data specifically includes: Identify the categories of data currently to be integrated; If the current data to be integrated is basic information data of an enterprise, a clustering method is adopted to integrate all basic information data of the enterprise collected from multiple data sources into a preset first data set, wherein the basic information data of the enterprise includes the address information of the enterprise, the basic information of the commercial subject and senior executives of the enterprise, and the registered capital information of the enterprise; If the current data to be integrated is enterprise supervision data, a clustering method is used to integrate all enterprise supervision data collected from multiple data sources into a preset second data set, wherein the enterprise supervision data includes the business scope declared by the enterprise, the filling of the enterprise annual report, and the handling of the enterprise business procedures; If the current data to be integrated is enterprise transaction data, all enterprise transaction data collected from multiple data sources are time-serialized and integrated in chronological order of transaction time, and all enterprise transaction data collected from multiple data sources are clustered and integrated according to transaction subjects, wherein the enterprise transaction data includes transaction data generated by the enterprise during the provision of business services.
4. The abnormal data evaluation method according to claim 1, characterized in that: The step of classifying the cleaned and preprocessed evaluation data according to the preset classification dimension to obtain the classified data specifically includes: Classify the cleaned and preprocessed evaluation data according to a preset first classification strategy to obtain first classified data, wherein the first classification strategy includes classifying the data based on whether the data is enterprise transaction data; The first classified data is classified again based on a second classification strategy to obtain second classified data as the target classified data, wherein the second classification strategy includes classification based on whether the data has time series.
5. The abnormal data evaluation method according to claim 1, characterized in that: The pre-trained anomaly detection model includes a first detection component and a second detection component. The step of inputting the classification data into the pre-trained anomaly detection model to obtain an output result of the anomaly detection model specifically includes: According to the preset classification mark, identify whether the current classification data has time series; If the current classified data does not have time series characteristics, then determining abnormal data in the current classified data through the first detection component; If the current classified data has time series characteristics, the abnormal data in the current classified data is determined by the second detection component.
6. The abnormal data evaluation method according to claim 5, characterized in that: The step of determining abnormal data in the current classified data by the first detection component specifically includes: Performing mathematical assignment processing on all data in the current classification data, wherein the mathematical assignment processing includes assigning a uniform mathematical value to data that is identical or has a similarity that reaches a preset threshold, and assigning other numerical values different from the mathematical value to data that has a similarity that does not reach the preset threshold; Obtain the mathematical values corresponding to all data in the current classification data after mathematical assignment processing; Using the preset algorithm formula: Calculate the Z score value corresponding to all data in the current classification data, where X represents the mathematical value corresponding to the target data in the current classification data, μ represents the average mathematical value corresponding to all data in the current classification data, and σ represents the standard deviation of the mathematical value corresponding to all data in the current classification data; By comparison, determine whether the Z score value corresponding to the current data is greater than the preset abnormal threshold; If the Z score value corresponding to the current data is greater than the preset abnormal threshold, the current data is abnormal data.
7. The abnormal data evaluation method according to claim 5, characterized in that: The step of determining abnormal data in the current classified data by the second detection component specifically includes: The frequency domain feature extraction algorithm is used to extract features from the data in the current classification data; The extracted features are sorted according to the time series to obtain the feature vectors corresponding to all the data in the current classification data; For the feature vectors corresponding to all the data in the current classification data, the Euclidean distance algorithm is used to compare them pairwise to obtain the feature distance; Combining the feature distance and a preset density-based clustering algorithm, filtering out unclustered data, wherein the preset density-based clustering algorithm includes a DBSCAN algorithm; The unclassified data are marked as abnormal data in the current classified data.
8. The abnormal data evaluation method according to claim 1, characterized in that: The step of calculating the outlier value of the target evaluation data according to the output result and the preset valuation algorithm specifically includes: By using the output results, the data entry quantity of all abnormal data is counted; Using a preset valuation algorithm, calculate the proportion of the data items of all abnormal data in the items of the preliminary integrated evaluation data, wherein the valuation algorithm includes a probability algorithm; The proportion is used as the abnormal value, or a pre-divided abnormal level corresponding to the proportion is identified, and the abnormal level is used as the abnormal value.
9. An abnormal data evaluation device, characterized in that: include: A target data collection module is used to collect target evaluation data from multiple data sources; An encryption and disturbance processing module, used for encrypting and disturbing the target evaluation data to obtain data to be integrated; The data preliminary integration module is used to perform preliminary integration of all the data to be integrated by using a secure addition method to generate preliminary integrated evaluation data; A data classification processing module, used to clean and preprocess the preliminarily integrated evaluation data, and to classify the cleaned and preprocessed evaluation data according to a preset classification dimension to obtain classified data; An anomaly detection module, used to input the classified data into a pre-trained anomaly detection model to obtain an output result of the anomaly detection model; The outlier calculation module is used to calculate the outlier value of the target evaluation data according to the output result and a preset valuation algorithm.
10. A computer device, comprising a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the abnormal data evaluation method according to any one of claims 1 to 8 when executing the computer-readable instructions.
11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the abnormal data evaluation method according to any one of claims 1 to 8.
Citation Information
Patent Citations
Abnormal application data detection method and device, electronic equipment and storage medium
CN112989338A
Quantum security outsourcing machine learning method and system
CN116248260A
Frequency hopping signal detection method and device based on dynamic clustering
CN117318756A
Enterprise operation condition early warning method and device, server and readable storage medium
CN117522148A
Data asset processing method and system based on multilayer encryption auditing
CN118036035A
