Methods, devices, equipment and media for detecting anomalies in clearing data
By using the IPCA model for batch processing and principal component dimensionality reduction of liquidation data, the problems of low efficiency and unknown anomaly detection in traditional liquidation data are solved, and fast, accurate anomaly detection and interpretable output are achieved.
Patent Information
- Application Number
- CN202510033787.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-09
- Publication Date
- 2025-10-31
- Estimated Expiration
- 2045-01-09
AI Technical Summary
Traditional methods for detecting anomalies in clearing data are inefficient, struggle to detect unknown anomalies, and are costly to maintain.
Incremental Principal Component Analysis (IPCA) model is used for batch processing. Through principal component dimensionality reduction and reconstruction error calculation, anomalies in the clearing data are automatically detected.
It enables fast and accurate detection of data anomalies in clearing, reduces memory limitations, adapts to large-scale and streaming data, and provides interpretable anomaly output.
Smart Images
Figure CN119442118B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, equipment and medium for detecting anomalies in clearing data. Background Technology
[0002] Currently, clearing data is a critical element for ensuring the smooth operation of financial institutions. Traditional methods for detecting anomalies in clearing data are mainly based on rule engines, which rely on manually defined business rules. While rule engines are effective in detecting anomalies based on known patterns, their limitations are becoming increasingly apparent in the face of challenges such as rapidly growing data volume, increasing pattern complexity, and the diversification of anomaly types.
[0003] 1. Complex rule writing: A large number of rules need to be written for different business scenarios, which increases maintenance costs.
[0004] 2. Difficulty in handling unknown anomalies: Rule-based methods often struggle to capture anomalous patterns not present in historical data.
[0005] Therefore, how to efficiently and automatically detect anomalies in clearing data, and how to detect unknown anomalies, has become an urgent problem to be solved. Summary of the Invention
[0006] In view of the above, it is necessary to provide a method, apparatus, equipment and medium for detecting anomalies in clearing data, in order to solve the problems of low efficiency in detecting anomalies in clearing data and inability to detect unknown anomalies.
[0007] A method for detecting anomalies in clearing data, the method comprising:
[0008] Obtain the initial IPCA model and training samples, load the training samples into the initial IPCA model in batches for training, and obtain the target model and anomaly detection threshold.
[0009] In response to an anomaly detection command for the liquidation data, the liquidation data is loaded and cleaned to obtain data to be processed;
[0010] The target model is used to project the data to be processed to obtain dimensionality-reduced data.
[0011] The reconstruction error of the data to be processed is calculated based on the dimensionality reduction data and used as the target error.
[0012] Calculate the anomaly score of the data to be processed based on the target error;
[0013] By comparing the anomaly score with the anomaly detection threshold, the anomaly detection result of the liquidation data is obtained;
[0014] An anomaly report of the liquidation data is generated based on the anomaly detection results, and the anomaly report is sent to the designated terminal device.
[0015] According to a preferred embodiment of the present invention, the step of loading the training samples into the initial IPCA model in batches for training to obtain the target model and the anomaly detection threshold includes:
[0016] Obtain the file size of the training samples, the currently available memory, and the performance parameters of the training client;
[0017] The sample size for each batch is determined based on the file size, the currently available memory, and the performance parameters.
[0018] The IPCA model is initialized using the IncrementalPCA class, and the number of principal components after dimensionality reduction and the sample size per batch are configured to obtain the initial IPCA model.
[0019] The training samples are loaded into the initial IPCA model in batches according to the sample size of each batch for training, until all training samples have been trained, and the model obtained from the current training is determined as the target model.
[0020] During training, the covariance matrix of each batch of samples is calculated, and eigenvalue decomposition is performed on the covariance matrix of each batch of samples to extract the eigenvector direction corresponding to the largest eigenvalue as the principal component direction of each batch of samples.
[0021] Calculate the cumulative variance contribution rate corresponding to each principal component direction;
[0022] Obtain the contribution rate threshold, and select principal component directions from the principal component directions of each batch of samples whose cumulative variance contribution rate is greater than or equal to the contribution rate threshold as candidate directions;
[0023] The candidate directions are sorted in descending order of their cumulative variance contribution rates to obtain a candidate direction sequence.
[0024] The principal component space is constructed by obtaining the candidate directions at the previous preset positions from the candidate direction sequence;
[0025] The reconstruction error of each training sample in the principal component space is calculated, and the average reconstruction error of each training sample in the principal component space is calculated to obtain the mean reconstruction error.
[0026] The anomaly detection threshold is configured according to a preset multiple of the standard deviation of the mean of the reconstruction error.
[0027] According to a preferred embodiment of the present invention, loading the clearing data and cleaning the clearing data to obtain the data to be processed includes:
[0028] Load the liquidation data from multiple CSV files;
[0029] Delete all empty columns from the liquidation data, and delete rows from the liquidation data that include any missing values to obtain the first data;
[0030] Numerical features are filtered out from the first data to obtain the second data;
[0031] The second data is normalized to obtain the data to be processed.
[0032] According to a preferred embodiment of the present invention, the step of calculating the reconstruction error of the data to be processed as the target error based on the dimensionality reduction data includes:
[0033] Obtain the transpose of the principal component matrix corresponding to the principal component space;
[0034] Calculate the product of the transpose matrix and the dimensionality-reduced data to obtain intermediate data;
[0035] The difference between the data to be processed and the intermediate data is calculated to obtain the target error.
[0036] According to a preferred embodiment of the present invention, calculating the anomaly score of the data to be processed based on the target error includes:
[0037] Calculate the L2 norm of the target error to obtain the anomaly score.
[0038] According to a preferred embodiment of the present invention, the step of comparing the anomaly score with the anomaly detection threshold to obtain the anomaly detection result of the liquidation data includes:
[0039] When the anomaly score is greater than or equal to the anomaly detection threshold, the liquidation data is determined to be abnormal; or
[0040] When the anomaly score is less than the anomaly detection threshold, it is determined that the liquidation data is not abnormal.
[0041] According to a preferred embodiment of the present invention, the method further includes:
[0042] New liquidation data is collected at preset time intervals, and the new liquidation data is used to supplement the training of the target model.
[0043] A clearing data anomaly detection device, the clearing data anomaly detection device comprising:
[0044] The training unit is used to acquire the initial IPCA model and training samples, load the training samples into the initial IPCA model in batches for training, and obtain the target model and anomaly detection threshold.
[0045] A cleaning unit is used to load the clearing data in response to an anomaly detection command for the clearing data, and to clean the clearing data to obtain data to be processed.
[0046] A projection unit is used to project the data to be processed using the target model to obtain dimensionality-reduced data.
[0047] The calculation unit is used to calculate the reconstruction error of the data to be processed as the target error based on the dimensionality reduction data;
[0048] The calculation unit is also used to calculate the anomaly score of the data to be processed based on the target error;
[0049] The comparison unit is used to compare the anomaly score with the anomaly detection threshold to obtain the anomaly detection result of the liquidation data;
[0050] The generation unit is used to generate an anomaly report of the liquidation data based on the anomaly detection results, and send the anomaly report to a designated terminal device.
[0051] A computer device, the computer device comprising:
[0052] Memory, storing at least one instruction; and
[0053] The processor executes instructions stored in the memory to implement the liquidation data anomaly detection method.
[0054] A computer-readable storage medium storing at least one instruction, which is executed by a processor in a computer device to implement the liquidation data anomaly detection method.
[0055] As can be seen from the above technical solutions, on the one hand, the present invention achieves batch processing of clearing data through the IPCA model, which can avoid loading all data at once and solve the memory limitation problem in traditional big data processing scenarios, making it more suitable for processing large-scale or streaming clearing data; on the other hand, the present invention can detect abnormal data in clearing data more quickly and accurately through principal component dimensionality reduction and projection error calculation of the IPCA model; and on the other hand, the present invention uses the statistical characteristics of reconstruction error to identify fields with high deviation, realizing the interpretable output of abnormal data. Attached Figure Description
[0056] Figure 1This is a flowchart of a preferred embodiment of the liquidation data anomaly detection method of the present invention.
[0057] Figure 2 This is a functional block diagram of a preferred embodiment of the data anomaly detection device of the present invention.
[0058] Figure 3 This is a schematic diagram of the structure of a computer device that implements the method for detecting anomalies in clearing data according to the present invention. Detailed Implementation
[0059] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.
[0060] like Figure 1 The diagram shown is a flowchart of a preferred embodiment of the data anomaly detection method of the present invention. The order of the steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements.
[0061] The clearing data anomaly detection method is applied to one or more computer devices. The computer device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.
[0062] The computer device can be any electronic product that can interact with the user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), game console, interactive network television (IPTV), smart wearable device, etc.
[0063] The computer equipment may also include network equipment and / or user equipment. The network equipment includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.
[0064] The server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.
[0065] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0066] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.
[0067] The network in which the computer device is located includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, and virtual private network (VPN).
[0068] S10: Obtain the initial IPCA (Incremental Principal Component Analysis) model and training samples, and load the training samples into the initial IPCA model in batches for training to obtain the target model and anomaly detection threshold.
[0069] In this embodiment, the training samples can be historical liquidation data.
[0070] In this embodiment, the step of loading the training samples into the initial IPCA model in batches for training to obtain the target model and the anomaly detection threshold includes:
[0071] Obtain the file size of the training samples, the currently available memory, and the performance parameters of the training client;
[0072] The sample size for each batch is determined based on the file size, the currently available memory, and the performance parameters.
[0073] The IPCA model is initialized using the IncrementalPCA class, and the number of principal components after dimensionality reduction and the sample size per batch are configured to obtain the initial IPCA model.
[0074] The training samples are loaded into the initial IPCA model in batches according to the sample size of each batch for training, until all training samples have been trained, and the model obtained from the current training is determined as the target model.
[0075] During training, the covariance matrix of each batch of samples is calculated, and eigenvalue decomposition is performed on the covariance matrix of each batch of samples to extract the eigenvector direction corresponding to the largest eigenvalue as the principal component direction of each batch of samples.
[0076] Calculate the cumulative variance contribution rate corresponding to each principal component direction;
[0077] Obtain the contribution rate threshold, and select principal component directions from the principal component directions of each batch of samples whose cumulative variance contribution rate is greater than or equal to the contribution rate threshold as candidate directions;
[0078] The candidate directions are sorted in descending order of their cumulative variance contribution rates to obtain a candidate direction sequence.
[0079] The principal component space is constructed by obtaining the candidate directions at the previous preset positions from the candidate direction sequence;
[0080] The reconstruction error of each training sample in the principal component space is calculated, and the average reconstruction error of each training sample in the principal component space is calculated to obtain the mean reconstruction error.
[0081] The anomaly detection threshold is configured according to a preset multiple of the standard deviation of the mean of the reconstruction error.
[0082] Specifically, by configuring the sample size for each batch based on the file size, available memory, and training performance parameters, optimal resource utilization can be maximized while ensuring best operational performance, thereby improving data processing efficiency while maintaining system processing performance. Simultaneously, it solves the memory limitation problem of traditional PCA in big data processing scenarios, enabling batch training and real-time detection.
[0083] Specifically, when initializing the IPCA model, the number of principal components after dimensionality reduction and the sample size per batch are configured to ensure that memory usage is controllable.
[0084] The contribution rate threshold can be configured according to the actual use case, such as 95%.
[0085] The preset position can be customized, and the preset position is less than or equal to the dimension of the covariance matrix.
[0086] The preset multiple can be 3 times. In this case, the anomaly detection threshold is the mean of the reconstruction error + 3 times the standard deviation, or the mean of the reconstruction error - 3 times the standard deviation, to satisfy statistical laws.
[0087] S11, in response to the anomaly detection instruction for the liquidation data, load the liquidation data and clean the liquidation data to obtain the data to be processed.
[0088] In this embodiment, the anomaly detection command can be triggered by the personnel responsible for the clearing task according to actual needs, or it can be automatically triggered when clearing data is detected to be uploaded, so as to realize real-time anomaly detection of clearing data.
[0089] In this embodiment, loading the clearing data and cleaning the clearing data to obtain the data to be processed includes:
[0090] The liquidation data is loaded from multiple CSV (Comma-Separated Values) files;
[0091] Delete all empty columns from the liquidation data, and delete rows from the liquidation data that include any missing values to obtain the first data;
[0092] Numerical features are filtered out from the first data to obtain the second data;
[0093] The second data is normalized to obtain the data to be processed.
[0094] Specifically, by deleting completely empty columns and rows containing any missing values from the liquidation data, the integrity of the liquidation data can be ensured.
[0095] In this process, by filtering out numerical features from the first data, non-numerical fields that cannot participate in PCA analysis can be eliminated.
[0096] By normalizing the second data, the influence of different feature dimensions can be eliminated, ensuring that each feature contributes equally to the model.
[0097] S12, Project the data to be processed using the target model to obtain dimensionality-reduced data.
[0098] It is understandable that in the analysis of liquidation data, normal data usually follows a certain statistical pattern, while abnormal data, due to its characteristics deviating from statistical regularities, can manifest as a large projection error in the dimensionality-reduced space.
[0099] Therefore, this embodiment uses principal component analysis (PCA) to reduce dimensionality, which can help to quickly and accurately detect abnormal data in the liquidation data.
[0100] S13, calculate the reconstruction error of the data to be processed based on the dimensionality reduction data as the target error.
[0101] In this embodiment, calculating the reconstruction error of the data to be processed as the target error based on the dimensionality reduction data includes:
[0102] Obtain the transpose of the principal component matrix corresponding to the principal component space;
[0103] Calculate the product of the transpose matrix and the dimensionality-reduced data to obtain intermediate data;
[0104] The difference between the data to be processed and the intermediate data is calculated to obtain the target error.
[0105] Through the above embodiments, the projection error of the data to be processed can be calculated, and the fields with high deviation can be identified by utilizing the statistical characteristics of the reconstruction error, thereby achieving interpretable output to assist in quickly and accurately locating abnormal data.
[0106] S14, Calculate the anomaly score of the data to be processed based on the target error.
[0107] In this embodiment, calculating the anomaly score of the data to be processed based on the target error includes:
[0108] Calculate the L2 norm of the target error to obtain the anomaly score.
[0109] S15, compare the anomaly score with the anomaly detection threshold to obtain the anomaly detection result of the liquidation data.
[0110] In this embodiment, comparing the anomaly score with the anomaly detection threshold to obtain the anomaly detection result of the liquidation data includes:
[0111] When the anomaly score is greater than or equal to the anomaly detection threshold, the liquidation data is determined to be abnormal; or
[0112] When the anomaly score is less than the anomaly detection threshold, it is determined that the liquidation data is not abnormal.
[0113] S16, Generate an anomaly report of the liquidation data based on the anomaly detection results, and send the anomaly report to the designated terminal device.
[0114] In this embodiment, the designated terminal device can be the terminal device of the personnel responsible for detecting abnormal data clearing.
[0115] This embodiment records the characteristics and degree of deviation of anomalies in the anomaly report for subsequent review or other business processing. Furthermore, while detecting abnormal data, it can also provide detailed explanations of the anomalies, including:
[0116] (1) Exception fields: Directly output the specific field that caused the exception and its exception value.
[0117] (2) Deviation degree: Calculate the deviation of the field value from the principal component space and quantify the difference between outliers and normal patterns.
[0118] (3) Comprehensive anomaly scoring: Combine multi-dimensional features to output anomaly scores, providing a reference for subsequent business processing.
[0119] Therefore, this embodiment, through detailed anomaly reports, can help quickly locate the root cause of the problem and assist in the execution of review and subsequent business decisions.
[0120] In this embodiment, the method further includes:
[0121] New liquidation data is collected at preset time intervals, and the new liquidation data is used to supplement the training of the target model.
[0122] The preset time interval can be configured according to actual needs, such as 30 days.
[0123] For example, the latest liquidation data can be periodically fed into the IPCA model, i.e., the target model, to update the principal component orientation and anomaly detection threshold, ensuring that the model always adapts to changes in data distribution.
[0124] This embodiment employs incremental principal component analysis (PCA) to achieve continuous training and dynamic updates of the model. By periodically inputting the latest clearing data into the model, the principal component direction and anomaly detection threshold can be adjusted according to changes in data distribution, thereby maintaining the accuracy and robustness of the detection results. This feature eliminates the cumbersome process of frequently adding and adjusting rules in traditional rule-based auditing methods, significantly reducing maintenance costs and adapting to the increasingly complex and changing characteristics of financial data. Furthermore, the model can adapt to various clearing scenarios and adjust to dynamic changes in data distribution through an incremental update mechanism.
[0125] Of course, the clearing data anomaly detection method used in this embodiment can also be applied to other risk control scenarios and anomaly identification scenarios.
[0126] As can be seen from the above technical solutions, on the one hand, the present invention achieves batch processing of clearing data through the IPCA model, which can avoid loading all data at once and solve the memory limitation problem in traditional big data processing scenarios, making it more suitable for processing large-scale or streaming clearing data; on the other hand, the present invention can detect abnormal data in clearing data more quickly and accurately through principal component dimensionality reduction and projection error calculation of the IPCA model; and on the other hand, the present invention uses the statistical characteristics of reconstruction error to identify fields with high deviation, realizing the interpretable output of abnormal data.
[0127] like Figure 2 The diagram shown is a functional block diagram of a preferred embodiment of the clearing data anomaly detection device of the present invention. The clearing data anomaly detection device 11 includes a training unit 110, a cleaning unit 111, a projection unit 112, a calculation unit 113, a comparison unit 114, and a generation unit 115. The module / unit referred to in this invention refers to a series of computer program segments that can be executed by a processor and perform a fixed function, and are stored in memory. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.
[0128] The training unit 110 is used to obtain an initial IPCA model and training samples, load the training samples into the initial IPCA model in batches for training, and obtain a target model and anomaly detection threshold.
[0129] The cleaning unit 111 is used to load the clearing data in response to an anomaly detection command for the clearing data, and clean the clearing data to obtain data to be processed.
[0130] The projection unit 112 is used to project the data to be processed using the target model to obtain dimensionality-reduced data.
[0131] The calculation unit 113 is used to calculate the reconstruction error of the data to be processed as the target error based on the dimensionality reduction data.
[0132] The calculation unit 113 is also used to calculate the anomaly score of the data to be processed based on the target error;
[0133] The comparison unit 114 is used to compare the anomaly score with the anomaly detection threshold to obtain the anomaly detection result of the liquidation data.
[0134] The generation unit 115 is used to generate an anomaly report of the liquidation data based on the anomaly detection result, and send the anomaly report to a designated terminal device.
[0135] As can be seen from the above technical solutions, on the one hand, the present invention achieves batch processing of clearing data through the IPCA model, which can avoid loading all data at once and solve the memory limitation problem in traditional big data processing scenarios, making it more suitable for processing large-scale or streaming clearing data; on the other hand, the present invention can detect abnormal data in clearing data more quickly and accurately through principal component dimensionality reduction and projection error calculation of the IPCA model; and on the other hand, the present invention uses the statistical characteristics of reconstruction error to identify fields with high deviation, realizing the interpretable output of abnormal data.
[0136] like Figure 3 The diagram shown is a schematic representation of the computer device used to implement the method for detecting anomalies in clearing data according to the present invention.
[0137] The computer device 1 may include a memory 12, a processor 13 and a bus, and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a data anomaly detection program.
[0138] Those skilled in the art will understand that the schematic diagram is merely an example of computer device 1 and does not constitute a limitation on computer device 1. Computer device 1 can be either a bus topology or a star topology. Computer device 1 may also include more or fewer other hardware or software than shown in the diagram, or different component arrangements. For example, computer device 1 may also include input / output devices, network access devices, etc.
[0139] It should be noted that the computer device 1 described is merely an example. Other existing or future electronic products that are adaptable to this invention should also be included within the scope of protection of this invention and are incorporated herein by reference.
[0140] The memory 12 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic storage, magnetic disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the computer device 1, such as a portable hard drive of the computer device 1. In other embodiments, the memory 12 can be an external storage device of the computer device 1, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the computer device 1. Furthermore, the memory 12 can include both internal and external storage units of the computer device 1. The memory 12 can be used not only to store application software and various types of data installed on the computer device 1, such as the code for a data anomaly detection program, but also to temporarily store data that has been output or will be output.
[0141] In some embodiments, the processor 13 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control unit of the computer device 1, connecting various components of the computer device 1 via various interfaces and lines. It executes programs or modules stored in the memory 12 (e.g., executing a data anomaly detection program) and calls data stored in the memory 12 to perform various functions of the computer device 1 and process data.
[0142] The processor 13 executes the operating system of the computer device 1 and various installed applications. The processor 13 executes the applications to implement the steps in the various embodiments of the clearing data anomaly detection method described above, for example... Figure 1 The steps are shown.
[0143] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of performing specific functions, which describe the execution process of the computer program in the computer device 1. For example, the computer program may be divided into a training unit 110, a cleaning unit 111, a projection unit 112, a calculation unit 113, a comparison unit 114, and a generation unit 115.
[0144] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, computer equipment, or network device, etc.) or processor to execute portions of the clearing data anomaly detection method described in the various embodiments of the present invention.
[0145] If the modules / units integrated in the computer device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware devices. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above.
[0146] The computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory, etc.
[0147] Furthermore, the computer-readable storage medium may primarily include a stored program area and a stored data area, wherein the stored program area may store the operating system, an application program required for at least one function, etc.; and the stored data area may store data created based on the use of blockchain nodes, etc.
[0148] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.
[0149] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, in... Figure 3 The bus is represented by only one straight line, but this does not mean that there is only one bus or one type of bus. The bus is configured to enable communication between the memory 12 and at least one processor 13, etc.
[0150] Although not shown, the computer device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 13 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The computer device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.
[0151] Furthermore, the computer device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the computer device 1 and other computer devices.
[0152] Optionally, the computer device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the computer device 1 and to display a visual user interface.
[0153] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.
[0154] It will be understood by those skilled in the art that Figure 3 The structure shown does not constitute a limitation on the computer device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.
[0155] Combination Figure 1 The memory 12 in the computer device 1 stores multiple instructions to implement a data anomaly detection method, and the processor 13 can execute the multiple instructions to achieve the following:
[0156] Obtain the initial IPCA model and training samples, load the training samples into the initial IPCA model in batches for training, and obtain the target model and anomaly detection threshold.
[0157] In response to an anomaly detection command for the liquidation data, the liquidation data is loaded and cleaned to obtain data to be processed;
[0158] The target model is used to project the data to be processed to obtain dimensionality-reduced data.
[0159] The reconstruction error of the data to be processed is calculated based on the dimensionality reduction data and used as the target error.
[0160] Calculate the anomaly score of the data to be processed based on the target error;
[0161] By comparing the anomaly score with the anomaly detection threshold, the anomaly detection result of the liquidation data is obtained;
[0162] An anomaly report of the liquidation data is generated based on the anomaly detection results, and the anomaly report is sent to the designated terminal device.
[0163] Specifically, the processor 13's implementation method for the above instructions can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.
[0164] It should be noted that all the data involved in this case was legally obtained.
[0165] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.
[0166] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.
[0167] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0168] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.
[0169] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.
[0170] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.
[0171] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in this invention can also be implemented by a single unit or device through software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.
[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims
1. A method for detecting anomalies in clearing data, characterized in that, The method for detecting anomalies in liquidation data includes: The process involves obtaining an initial IPCA model and historical liquidation data as training samples. These training samples are then loaded into the initial IPCA model in batches for training to obtain a target model and an anomaly detection threshold. Specifically, the process includes obtaining the file size, available memory, and performance parameters of the training samples; determining the sample size for each batch based on these parameters; initializing the IPCA model using the IncrementalPCA class and configuring the number of principal components after dimensionality reduction and the sample size for each batch; and determining the initial IPCA model by loading the training samples into the initial IPCA model in batches for training to obtain the target model and anomaly detection threshold. This process includes loading the training samples into the initial IPCA model in batches according to the sample size for each batch until all training samples have completed training, at which point the currently trained model is determined as the target model. During training, the covariance matrix of each batch of samples is calculated, and eigenvalue decomposition is performed on the covariance matrix of each batch of samples to extract the eigenvector direction corresponding to the largest eigenvalue as the sample size for each batch. The process involves: determining the principal component directions; calculating the cumulative variance contribution rate for each principal component direction; obtaining a contribution rate threshold; selecting principal component directions from each batch of samples whose cumulative variance contribution rate is greater than or equal to the contribution rate threshold as candidate directions; sorting the candidate directions in descending order of cumulative variance contribution rate to obtain a candidate direction sequence; constructing a principal component space from the candidate direction sequence by selecting the first preset position of the candidate directions; calculating the reconstruction error of each training sample in the principal component space and calculating the average reconstruction error of each training sample in the principal component space to obtain the mean reconstruction error; and configuring the anomaly detection threshold according to a preset multiple of the standard deviation of the mean reconstruction error. In response to an anomaly detection command for the liquidation data, the liquidation data is loaded and cleaned to obtain data to be processed; The target model is used to project the data to be processed to obtain dimensionality-reduced data. The reconstruction error of the data to be processed is calculated based on the dimensionality reduction data and used as the target error. Calculate the anomaly score of the data to be processed based on the target error; By comparing the anomaly score with the anomaly detection threshold, the anomaly detection result of the liquidation data is obtained; An anomaly report of the liquidation data is generated based on the anomaly detection results, and the anomaly report is sent to the designated terminal device; Specifically, new liquidation data is collected at preset time intervals, and the new liquidation data is used to supplement the training of the target model in order to update the principal component direction of the target model and the anomaly detection threshold.
2. The method for detecting anomalies in clearing data as described in claim 1, characterized in that, The process of loading the liquidation data and cleaning the liquidation data to obtain the data to be processed includes: Load the liquidation data from multiple CSV files; Delete all empty columns from the liquidation data, and delete rows from the liquidation data that include any missing values to obtain the first data; Numerical features are filtered out from the first data to obtain the second data; The second data is normalized to obtain the data to be processed.
3. The method for detecting anomalies in clearing data as described in claim 1, characterized in that, The step of calculating the reconstruction error of the data to be processed as the target error based on the dimensionality reduction data includes: Obtain the transpose of the principal component matrix corresponding to the principal component space; Calculate the product of the transpose matrix and the dimensionality-reduced data to obtain intermediate data; The difference between the data to be processed and the intermediate data is calculated to obtain the target error.
4. The method for detecting anomalies in clearing data as described in claim 1, characterized in that, The step of calculating the anomaly score of the data to be processed based on the target error includes: Calculate the L2 norm of the target error to obtain the anomaly score.
5. The method for detecting anomalies in liquidation data as described in claim 1, characterized in that, The comparison of the anomaly score with the anomaly detection threshold to obtain the anomaly detection result of the liquidation data includes: When the anomaly score is greater than or equal to the anomaly detection threshold, the liquidation data is determined to be abnormal; or When the anomaly score is less than the anomaly detection threshold, it is determined that the liquidation data is not abnormal.
6. A clearing data anomaly detection device, characterized in that, The clearing data anomaly detection device includes: A training unit is used to acquire an initial IPCA model and historical liquidation data as training samples. The training samples are loaded into the initial IPCA model in batches for training to obtain a target model and anomaly detection threshold. Specifically, the unit acquires the file size, currently available memory, and performance parameters of the training samples; determines the sample size for each batch based on the file size, available memory, and performance parameters; initializes the IPCA model using the IncrementalPCA class and configures the number of principal components after dimensionality reduction and the sample size for each batch, thus obtaining the initial IPCA model. The step of loading the training samples into the initial IPCA model in batches to obtain the target model and anomaly detection threshold includes: loading the training samples into the initial IPCA model in batches according to the sample size for each batch until all training samples have completed training, then determining the currently trained model as the target model; during training, calculating the covariance matrix of each batch of samples and performing eigenvalue decomposition on the covariance matrix of each batch of samples to extract the eigenvector direction corresponding to the largest eigenvalue as the sample size for each batch. The process involves: determining the principal component directions; calculating the cumulative variance contribution rate for each principal component direction; obtaining a contribution rate threshold; selecting principal component directions from each batch of samples whose cumulative variance contribution rate is greater than or equal to the contribution rate threshold as candidate directions; sorting the candidate directions in descending order of cumulative variance contribution rate to obtain a candidate direction sequence; constructing a principal component space from the candidate direction sequence by selecting the first preset position of the candidate directions; calculating the reconstruction error of each training sample in the principal component space and calculating the average reconstruction error of each training sample in the principal component space to obtain the mean reconstruction error; and configuring the anomaly detection threshold according to a preset multiple of the standard deviation of the mean reconstruction error. A cleaning unit is used to load the clearing data in response to an anomaly detection command for the clearing data, and to clean the clearing data to obtain data to be processed. A projection unit is used to project the data to be processed using the target model to obtain dimensionality-reduced data. The calculation unit is used to calculate the reconstruction error of the data to be processed as the target error based on the dimensionality reduction data; The calculation unit is also used to calculate the anomaly score of the data to be processed based on the target error; The comparison unit is used to compare the anomaly score with the anomaly detection threshold to obtain the anomaly detection result of the liquidation data; The generation unit is configured to generate an anomaly report of the liquidation data based on the anomaly detection results, and send the anomaly report to a designated terminal device; Specifically, new liquidation data is collected at preset time intervals, and the new liquidation data is used to supplement the training of the target model in order to update the principal component direction of the target model and the anomaly detection threshold.
7. A computer device, characterized in that, The computer device includes: Memory, storing at least one instruction; and The processor executes instructions stored in the memory to implement the liquidation data anomaly detection method as described in any one of claims 1 to 5.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, which is executed by a processor in a computer device to implement the liquidation data anomaly detection method as described in any one of claims 1 to 5.