Clearing data anomaly detection method, device and equipment and medium

Abstract

The invention relates to the technical field of data processing, and provides a liquidation data anomaly detection method, device and equipment and a medium, on one hand, batch processing of liquidation data is achieved through an IPCA model, loading of all data at a time can be avoided, the problem of memory limitation in a traditional big data processing scene is solved, and the processing efficiency is improved; the method is more suitable for processing large-scale or streaming liquidation data; on one hand, the abnormal data in the clearing data can be detected more quickly and accurately through principal component dimensionality reduction and projection error calculation of the IPCA model; and on the other hand, high-deviation fields are identified by utilizing statistical characteristics of reconstruction errors, and interpretable output of abnormal data is realized.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, equipment and medium for detecting anomalies in clearing data. Background Art

[0002] At present, the clearing data of financial institutions is a key link to ensure the normal operation of the business. Traditional clearing data anomaly detection methods are mainly based on rule engines, which rely on manually set business rules. Although rule engines have good results in detecting data anomalies with known patterns, facing the challenges of rapid growth in data size, increasing pattern complexity, and diversified anomaly types, the limitations of traditional methods are gradually emerging: 1. Complex rule writing: A large number of rules need to be written for different business scenarios, which increases maintenance costs; 2. Difficulty dealing with unknown anomalies: Rule-based methods often have difficulty capturing abnormal patterns that do not appear in historical data.

[0003] Therefore, how to efficiently and automatically detect anomalies in clearing data and detect unknown anomalies has become an urgent problem to be solved. Summary of the invention

[0004] In view of the above, it is necessary to provide a clearing data anomaly detection method, device, equipment and medium, aiming to solve the problem of low efficiency of clearing data anomaly detection and inability to detect unknown anomalies.

[0005] A method for detecting anomalies in clearing data, the method comprising: Obtain an initial IPCA model and training samples, load the training samples into the initial IPCA model in batches for training, and obtain a target model and anomaly detection threshold; In response to an abnormality detection instruction for clearing data, the clearing data is loaded, and the clearing data is cleaned to obtain data to be processed; Projecting the data to be processed using the target model to obtain dimension-reduced data; Calculate the reconstruction error of the data to be processed according to the dimension reduction data as the target error; Calculating anomaly scores of the data to be processed according to the target error; Comparing the anomaly score with the anomaly detection threshold to obtain an anomaly detection result of the clearing data; An abnormality report of the clearing data is generated according to the abnormality detection result, and the abnormality report is sent to a designated terminal device.

[0006] According to a preferred embodiment of the present invention, loading the training samples into the initial IPCA model in batches for training to obtain the target model and the anomaly detection threshold comprises: Obtaining the file size of the training sample, the current available memory and the performance parameters of the training end; Determine the sample size for each batch according to the file volume, the currently available content and the performance parameter; Initialize the IPCA model using the IncrementalPCA class, and configure the number of principal components after dimensionality reduction and the sample size per batch corresponding to the IPCA model to obtain the initial IPCA model; Loading the training samples into the initial IPCA model in batches according to the sample size of each batch for training, until all training samples have completed training, and determining the model obtained by the current training as the target model; During the training process, the covariance matrix of each batch of samples is calculated, and the eigenvalue decomposition is performed on the covariance matrix of each batch of samples to extract the eigenvector direction corresponding to the maximum eigenvalue as the principal component direction of each batch of samples; Calculate the cumulative variance contribution rate corresponding to each principal component direction; Obtain a contribution rate threshold, and obtain, from the principal component directions of each batch of samples, principal component directions whose cumulative variance contribution rate is greater than or equal to the contribution rate threshold as candidate directions; Sorting the candidate directions in descending order of the cumulative variance contribution rate to obtain a candidate direction sequence; Acquire the previously preset candidate directions from the candidate direction sequence to construct a principal component space; Calculating the reconstruction error of each training sample in the principal component space, and calculating the average value of the reconstruction error of each training sample in the principal component space to obtain a reconstruction error mean; The anomaly detection threshold is configured according to a preset multiple standard deviation of the reconstruction error mean.

[0007] According to a preferred embodiment of the present invention, the loading of the clearing data and cleaning the clearing data to obtain the data to be processed includes: Load said liquidation data from multiple CSV files; Deleting all empty columns in the clearing data and deleting rows including any missing values ​​in the clearing data to obtain first data; Filtering out numerical features from the first data to obtain second data; The second data is normalized to obtain the data to be processed.

[0008] According to a preferred embodiment of the present invention, calculating the reconstruction error of the data to be processed as the target error according to the dimension reduction data comprises: Obtaining a transposed matrix of a principal component matrix corresponding to the principal component space; Calculate the product of the transposed matrix and the dimension-reduced data to obtain intermediate data; The difference between the data to be processed and the intermediate data is calculated to obtain the target error.

[0009] According to a preferred embodiment of the present invention, calculating the abnormality score of the data to be processed according to the target error includes: The L2 norm of the target error is calculated to obtain the anomaly score.

[0010] According to a preferred embodiment of the present invention, comparing the anomaly score with the anomaly detection threshold to obtain an anomaly detection result of the clearing data includes: When the abnormality score is greater than or equal to the abnormality detection threshold, determining that the clearing data is abnormal; or When the anomaly score is less than the anomaly detection threshold, it is determined that the clearing data has no anomaly.

[0011] According to a preferred embodiment of the present invention, the method further comprises: New clearing data is collected at preset time intervals, and the target model is supplemented with training using the new clearing data.

[0012] A clearing data anomaly detection device, the clearing data anomaly detection device comprising: A training unit, used to obtain an initial IPCA model and training samples, load the training samples into the initial IPCA model in batches for training, and obtain a target model and anomaly detection threshold; A cleaning unit, configured to load the clearing data in response to an abnormality detection instruction for the clearing data, and clean the clearing data to obtain data to be processed; A projection unit, used for projecting the data to be processed using the target model to obtain dimension-reduced data; A calculation unit, used for calculating the reconstruction error of the data to be processed as a target error according to the dimension reduction data; The calculation unit is further used to calculate the abnormality score of the data to be processed according to the target error; A comparison unit, used to compare the anomaly score with the anomaly detection threshold to obtain an anomaly detection result of the clearing data; A generating unit is used to generate an abnormality report of the clearing data according to the abnormality detection result, and send the abnormality report to a designated terminal device.

[0013] A computer device, comprising: a memory storing at least one instruction; and A processor executes instructions stored in the memory to implement the clearing data anomaly detection method.

[0014] A computer-readable storage medium stores at least one instruction, and the at least one instruction is executed by a processor in a computer device to implement the clearing data anomaly detection method.

[0015] It can be seen from the above technical solutions that, on the one hand, the present invention realizes batch processing of liquidation data through the IPCA model, which can avoid loading all data at one time, solves the problem of memory limitation in traditional big data processing scenarios, and is more suitable for processing large-scale or streaming liquidation data; on the one hand, the present invention can detect abnormal data in liquidation data more quickly and accurately through the principal component dimensionality reduction and projection error calculation of the IPCA model; on the other hand, the present invention utilizes the statistical characteristics of the reconstruction error to identify high-deviation fields, thereby realizing the interpretable output of abnormal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of a preferred embodiment of the method for detecting anomalies in clearing data of the present invention; Figure 2 It is a functional module diagram of a preferred embodiment of the clearing data anomaly detection device of the present invention; Figure 3 It is a structural schematic diagram of a computer device of a preferred embodiment of the liquidation data anomaly detection method of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0018] like Figure 1 FIG. 1 is a flow chart of a preferred embodiment of the method for detecting anomalies in clearing data of the present invention. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted.

[0019] The clearing data anomaly detection method is applied to one or more computer devices, and the computer device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0020] The computer device may be any electronic product that can perform human-computer interaction with a user, such as a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), a game console, an interactive network television (IPTV), a smart wearable device, etc.

[0021] The computer device may also include a network device and / or a user device, wherein the network device includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud consisting of a large number of hosts or network servers based on cloud computing.

[0022] The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0023] Among them, Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0024] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0025] The network where the computer device is located includes but is not limited to the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (VPN), etc.

[0026] S10, obtaining an initial IPCA (Incremental Principal Component Analysis) model and training samples, loading the training samples into the initial IPCA model in batches for training, and obtaining a target model and anomaly detection threshold.

[0027] In this embodiment, the training sample may be historical clearing data.

[0028] In this embodiment, the training samples are loaded into the initial IPCA model in batches for training to obtain the target model and the anomaly detection threshold, including: Obtaining the file size of the training sample, the current available memory and the performance parameters of the training end; Determine the sample size for each batch according to the file volume, the currently available content and the performance parameter; Initialize the IPCA model using the IncrementalPCA class, and configure the number of principal components after dimensionality reduction and the sample size per batch corresponding to the IPCA model to obtain the initial IPCA model; Loading the training samples into the initial IPCA model in batches according to the sample size of each batch for training, until all training samples have completed training, and determining the model obtained by the current training as the target model; During the training process, the covariance matrix of each batch of samples is calculated, and the eigenvalue decomposition is performed on the covariance matrix of each batch of samples to extract the eigenvector direction corresponding to the maximum eigenvalue as the principal component direction of each batch of samples; Calculate the cumulative variance contribution rate corresponding to each principal component direction; Obtain a contribution rate threshold, and obtain, from the principal component directions of each batch of samples, principal component directions whose cumulative variance contribution rate is greater than or equal to the contribution rate threshold as candidate directions; Sorting the candidate directions in descending order of the cumulative variance contribution rate to obtain a candidate direction sequence; Acquire the previously preset candidate directions from the candidate direction sequence to construct a principal component space; Calculating the reconstruction error of each training sample in the principal component space, and calculating the average value of the reconstruction error of each training sample in the principal component space to obtain a reconstruction error mean; The anomaly detection threshold is configured according to a preset multiple standard deviation of the reconstruction error mean.

[0029] Among them, the sample size of each batch is configured in a targeted manner according to the file volume, the currently available content and the performance parameters of the training end, which can maximize the utilization of resources while ensuring the best operating performance, thereby improving data processing efficiency while ensuring system processing performance. At the same time, it solves the memory limitation problem of traditional PCA in big data processing scenarios and realizes batch training and real-time detection.

[0030] When initializing the IPCA model, the number of principal components after dimensionality reduction and the sample size per batch corresponding to the IPCA model are configured to ensure that the memory usage is controllable.

[0031] The contribution rate threshold may be configured according to actual usage scenarios, such as 95%.

[0032] The pre-set positioning may be custom configured, and the pre-set positioning is smaller than or equal to the dimension of the covariance matrix.

[0033] The preset multiple may be 3 times, in which case the anomaly detection threshold is the reconstruction error mean + 3 times the standard deviation, or the reconstruction error mean - 3 times the standard deviation, to meet statistical rules.

[0034] S11, in response to an abnormality detection instruction for clearing data, loading the clearing data, and cleaning the clearing data to obtain data to be processed.

[0035] In this embodiment, the anomaly detection instruction can be triggered by the person in charge of the clearing task according to actual needs, or it can be automatically triggered when clearing data is detected to be uploaded, so as to realize real-time anomaly detection of clearing data.

[0036] In this embodiment, the loading of the clearing data and cleaning the clearing data to obtain the data to be processed includes: Load the liquidation data from multiple CSV (Comma-Separated Values) files; Deleting all empty columns in the clearing data and deleting rows including any missing values ​​in the clearing data to obtain first data; Filtering out numerical features from the first data to obtain second data; The second data is normalized to obtain the data to be processed.

[0037] The integrity of the clearing data can be ensured by deleting all empty columns in the clearing data and deleting rows including any missing values ​​in the clearing data.

[0038] Among them, by screening out numerical features from the first data, non-numerical fields that cannot participate in PCA analysis can be eliminated.

[0039] By normalizing the second data, the influence of different feature dimensions can be eliminated to ensure that each feature contributes equally to the model.

[0040] S12, projecting the data to be processed using the target model to obtain dimension-reduced data.

[0041] It is understandable that in the analysis of clearing data, normal data usually follows certain statistical patterns, while abnormal data, due to its characteristics deviating from statistical laws, can be manifested as large projection errors in the space after dimensionality reduction.

[0042] Therefore, this embodiment uses the principal component dimensionality reduction of IPCA to assist in the subsequent rapid and accurate detection of abnormal data in the clearing data.

[0043] S13, calculating the reconstruction error of the data to be processed according to the dimension reduction data as the target error.

[0044] In this embodiment, calculating the reconstruction error of the data to be processed according to the dimension reduction data as the target error includes: Obtaining a transposed matrix of a principal component matrix corresponding to the principal component space; Calculate the product of the transposed matrix and the dimension-reduced data to obtain intermediate data; The difference between the data to be processed and the intermediate data is calculated to obtain the target error.

[0045] Through the above embodiments, the projection error of the data to be processed can be calculated, and the statistical characteristics of the reconstruction error can be used to identify fields with high deviation, thereby achieving interpretable output to assist in quickly and accurately locating abnormal data.

[0046] S14: Calculate anomaly scores of the data to be processed according to the target error.

[0047] In this embodiment, calculating the abnormality score of the data to be processed according to the target error includes: The L2 norm of the target error is calculated to obtain the anomaly score.

[0048] S15, comparing the anomaly score with the anomaly detection threshold to obtain an anomaly detection result of the clearing data.

[0049] In this embodiment, comparing the anomaly score with the anomaly detection threshold to obtain an anomaly detection result of the clearing data includes: When the abnormality score is greater than or equal to the abnormality detection threshold, determining that the clearing data is abnormal; or When the anomaly score is less than the anomaly detection threshold, it is determined that the clearing data has no anomaly.

[0050] S16, generating an abnormality report of the clearing data according to the abnormality detection result, and sending the abnormality report to a designated terminal device.

[0051] In this embodiment, the designated terminal device may be a terminal device of a person responsible for clearing data anomaly detection.

[0052] This embodiment records the characteristics and deviation degree of the abnormal point in the abnormal report for subsequent review or other business processing. In addition, when abnormal data is detected, a detailed abnormal explanation can also be provided, including: (1) Abnormal field: directly output the specific field causing the abnormality and its abnormal value; (2) Deviation degree: Calculate the deviation between the field value and the principal component space, and quantify the difference between the outlier and the normal pattern; (3) Comprehensive anomaly scoring: Combine multi-dimensional features to output anomaly scores, providing a reference for subsequent business processing.

[0053] Therefore, this embodiment can assist in quickly locating the root cause of the problem through detailed exception reports, and assist in execution review and subsequent business decision-making.

[0054] In this embodiment, the method further includes: New clearing data is collected at preset time intervals, and the target model is supplemented with training using the new clearing data.

[0055] The preset time interval can be configured according to actual needs, such as 30 days.

[0056] For example, the latest clearing data can be regularly fed into the IPCA model, ie, the target model, to update the principal component directions and anomaly detection thresholds, ensuring that the model always adapts to changes in data distribution.

[0057] This embodiment uses an incremental principal component analysis algorithm to achieve continuous training and dynamic updating of the model. By regularly inputting the latest clearing data into the model, the principal component direction and anomaly detection threshold can be adjusted as the data distribution changes, thereby maintaining the accuracy and robustness of the detection effect. This feature eliminates the cumbersome process of frequently adding and adjusting rules in traditional rule auditing methods, greatly reduces maintenance costs, and can adapt to the increasingly complex and changing characteristics of financial data. In addition, the model can adapt to a variety of clearing scenarios and adapt to dynamic changes in data distribution through an incremental update mechanism.

[0058] Of course, the clearing data anomaly detection method adopted in this embodiment can also be applied to other risk control scenarios and anomaly identification scenarios.

[0059] It can be seen from the above technical solutions that, on the one hand, the present invention realizes batch processing of liquidation data through the IPCA model, which can avoid loading all data at one time, solves the problem of memory limitation in traditional big data processing scenarios, and is more suitable for processing large-scale or streaming liquidation data; on the one hand, the present invention can detect abnormal data in liquidation data more quickly and accurately through the principal component dimensionality reduction and projection error calculation of the IPCA model; on the other hand, the present invention utilizes the statistical characteristics of the reconstruction error to identify high-deviation fields, thereby realizing the interpretable output of abnormal data.

[0060] like Figure 2 , which is a functional module diagram of a preferred embodiment of the clearing data anomaly detection device of the present invention. The clearing data anomaly detection device 11 includes a training unit 110, a cleaning unit 111, a projection unit 112, a calculation unit 113, a comparison unit 114, and a generation unit 115. The module / unit referred to in the present invention refers to a series of computer program segments that can be executed by a processor and can perform fixed functions, which are stored in a memory. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0061] The training unit 110 is used to obtain an initial IPCA model and training samples, load the training samples into the initial IPCA model in batches for training, and obtain a target model and anomaly detection threshold; The cleaning unit 111 is used to load the clearing data in response to an abnormality detection instruction for the clearing data, and clean the clearing data to obtain data to be processed; The projection unit 112 is used to project the data to be processed using the target model to obtain dimension-reduced data; The calculation unit 113 is used to calculate the reconstruction error of the data to be processed as the target error according to the dimension reduction data; The calculation unit 113 is further used to calculate the abnormality score of the data to be processed according to the target error; The comparison unit 114 is used to compare the anomaly score with the anomaly detection threshold to obtain an anomaly detection result of the clearing data; The generating unit 115 is used to generate an abnormality report of the clearing data according to the abnormality detection result, and send the abnormality report to a designated terminal device.

[0062] It can be seen from the above technical solutions that, on the one hand, the present invention realizes batch processing of liquidation data through the IPCA model, which can avoid loading all data at one time, solves the problem of memory limitation in traditional big data processing scenarios, and is more suitable for processing large-scale or streaming liquidation data; on the one hand, the present invention can detect abnormal data in liquidation data more quickly and accurately through the principal component dimensionality reduction and projection error calculation of the IPCA model; on the other hand, the present invention utilizes the statistical characteristics of the reconstruction error to identify high-deviation fields, thereby realizing the interpretable output of abnormal data.

[0063] like Figure 3 The figure is a schematic diagram of the structure of a computer device of a preferred embodiment of the method for detecting anomalies in clearing data according to the present invention.

[0064] The computer device 1 may include a memory 12, a processor 13 and a bus, and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a clearing data anomaly detection program.

[0065] Those skilled in the art will appreciate that the schematic diagram is merely an example of the computer device 1 and does not constitute a limitation on the computer device 1. The computer device 1 may be a bus-type structure or a star-type structure. The computer device 1 may also include more or less other hardware or software than shown in the diagram, or a different arrangement of components. For example, the computer device 1 may also include input and output devices, network access devices, etc.

[0066] It should be noted that the computer device 1 is only an example, and other existing or future electronic products that are suitable for the present invention should also be included in the protection scope of the present invention and included here by reference.

[0067] Among them, the memory 12 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the computer device 1, such as a mobile hard disk of the computer device 1. In other embodiments, the memory 12 can also be an external storage device of the computer device 1, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 1. Further, the memory 12 can also include both an internal storage unit of the computer device 1 and an external storage device. The memory 12 can not only be used to store application software and various types of data installed in the computer device 1, such as the code of the clearing data anomaly detection program, etc., but also can be used to temporarily store data that has been output or is to be output.

[0068] In some embodiments, the processor 13 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 13 is the control core (Control Unit) of the computer device 1, and uses various interfaces and lines to connect various components of the entire computer device 1, and executes various functions and processes data of the computer device 1 by running or executing programs or modules stored in the memory 12 (for example, executing a clearing data anomaly detection program, etc.), and calling data stored in the memory 12.

[0069] The processor 13 executes the operating system of the computer device 1 and various installed applications. The processor 13 executes the applications to implement the steps in the above-mentioned embodiments of the clearing data anomaly detection method, for example Figure 1 Steps shown.

[0070] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the computer device 1. For example, the computer program may be divided into a training unit 110, a cleaning unit 111, a projection unit 112, a calculation unit 113, a comparison unit 114, and a generation unit 115.

[0071] The above-mentioned integrated unit implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, and includes a number of instructions for enabling a computer device (which can be a personal computer, a computer device, or a network device, etc.) or a processor to execute a part of the clearing data anomaly detection method described in each embodiment of the present invention.

[0072] If the module / unit integrated in the computer device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware devices through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of each of the above-mentioned method embodiments can be implemented.

[0073] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory, etc.

[0074] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0075] The blockchain referred to in this invention is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the blockchain underlying platform, platform product service layer, and application service layer.

[0076] The bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 The bus is represented by only one straight line, but it does not mean that there is only one bus or one type of bus. The bus is configured to realize the connection and communication between the memory 12 and at least one processor 13, etc.

[0077] Although not shown, the computer device 1 may also include a power source (such as a battery) for supplying power to various components. Preferably, the power source may be logically connected to the at least one processor 13 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include any components such as one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, and power status indicators. The computer device 1 may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0078] Furthermore, the computer device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is usually used to establish a communication connection between the computer device 1 and other computer devices.

[0079] Optionally, the computer device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), or a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the computer device 1 and to display a visual user interface.

[0080] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0081] It can be understood by those skilled in the art that Figure 3 The structure shown does not constitute a limitation on the computer device 1, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0082] Combination Figure 1 , the memory 12 in the computer device 1 stores a plurality of instructions to implement a clearing data anomaly detection method, and the processor 13 can execute the plurality of instructions to implement: Obtain an initial IPCA model and training samples, load the training samples into the initial IPCA model in batches for training, and obtain a target model and anomaly detection threshold; In response to an abnormality detection instruction for clearing data, the clearing data is loaded, and the clearing data is cleaned to obtain data to be processed; Projecting the data to be processed using the target model to obtain dimension-reduced data; Calculate the reconstruction error of the data to be processed according to the dimension reduction data as the target error; Calculating anomaly scores of the data to be processed according to the target error; Comparing the anomaly score with the anomaly detection threshold to obtain an anomaly detection result of the clearing data; An abnormality report of the clearing data is generated according to the abnormality detection result, and the abnormality report is sent to a designated terminal device.

[0083] Specifically, the specific implementation method of the processor 13 for the above instructions can refer to Figure 1 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0084] It should be noted that the data involved in this case were all obtained legally.

[0085] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0086] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0087] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0088] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0089] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0090] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0091] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the present invention can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for detecting anomalies in clearing data, characterized in that: The clearing data anomaly detection method comprises: Obtain an initial IPCA model and training samples, load the training samples into the initial IPCA model in batches for training, and obtain a target model and anomaly detection threshold; In response to an abnormality detection instruction for clearing data, the clearing data is loaded, and the clearing data is cleaned to obtain data to be processed; Projecting the data to be processed using the target model to obtain dimension-reduced data; Calculate the reconstruction error of the data to be processed according to the dimension reduction data as the target error; Calculating anomaly scores of the data to be processed according to the target error; Comparing the anomaly score with the anomaly detection threshold to obtain an anomaly detection result of the clearing data; An abnormality report of the clearing data is generated according to the abnormality detection result, and the abnormality report is sent to a designated terminal device.

2. The method for detecting anomalies in clearing data according to claim 1, characterized in that: The step of loading the training samples into the initial IPCA model in batches for training to obtain the target model and the anomaly detection threshold comprises: Obtaining the file size of the training sample, the current available memory and the performance parameters of the training end; Determine the sample size for each batch according to the file volume, the currently available content and the performance parameter; Initialize the IPCA model using the IncrementalPCA class, and configure the number of principal components after dimensionality reduction and the sample size per batch corresponding to the IPCA model to obtain the initial IPCA model; Loading the training samples into the initial IPCA model in batches according to the sample size of each batch for training, until all training samples have completed training, and determining the model obtained by the current training as the target model; During the training process, the covariance matrix of each batch of samples is calculated, and the eigenvalue decomposition is performed on the covariance matrix of each batch of samples to extract the eigenvector direction corresponding to the maximum eigenvalue as the principal component direction of each batch of samples; Calculate the cumulative variance contribution rate corresponding to each principal component direction; Obtain a contribution rate threshold, and obtain, from the principal component directions of each batch of samples, principal component directions whose cumulative variance contribution rate is greater than or equal to the contribution rate threshold as candidate directions; Sorting the candidate directions in descending order of the cumulative variance contribution rate to obtain a candidate direction sequence; Acquire the previously preset candidate directions from the candidate direction sequence to construct a principal component space; Calculating the reconstruction error of each training sample in the principal component space, and calculating the average value of the reconstruction error of each training sample in the principal component space to obtain a reconstruction error mean; The anomaly detection threshold is configured according to a preset multiple standard deviation of the reconstruction error mean.

3. The method for detecting anomalies in clearing data according to claim 1, characterized in that: The loading of the clearing data and cleaning the clearing data to obtain the data to be processed includes: Load said liquidation data from multiple CSV files; Deleting all empty columns in the clearing data and deleting rows including any missing values ​​in the clearing data to obtain first data; Filtering out numerical features from the first data to obtain second data; The second data is normalized to obtain the data to be processed.

4. The method for detecting anomalies in clearing data according to claim 2, characterized in that: Calculating the reconstruction error of the data to be processed as the target error according to the dimension reduction data includes: Obtaining a transposed matrix of a principal component matrix corresponding to the principal component space; Calculate the product of the transposed matrix and the dimension-reduced data to obtain intermediate data; The difference between the data to be processed and the intermediate data is calculated to obtain the target error.

5. The method for detecting anomalies in clearing data according to claim 1, characterized in that: Calculating the abnormal score of the data to be processed according to the target error includes: The L2 norm of the target error is calculated to obtain the anomaly score.

6. The method for detecting anomalies in clearing data according to claim 1, characterized in that: The comparing the anomaly score with the anomaly detection threshold to obtain an anomaly detection result of the clearing data includes: When the abnormality score is greater than or equal to the abnormality detection threshold, determining that the clearing data is abnormal; or When the anomaly score is less than the anomaly detection threshold, it is determined that the clearing data has no anomaly.

7. The method for detecting anomalies in clearing data according to claim 1, characterized in that: The method further comprises: New clearing data is collected at preset time intervals, and the target model is supplemented with training using the new clearing data.

8. A clearing data anomaly detection device, characterized in that: The clearing data anomaly detection device comprises: A training unit, used to obtain an initial IPCA model and training samples, load the training samples into the initial IPCA model in batches for training, and obtain a target model and anomaly detection threshold; A cleaning unit, configured to load the clearing data in response to an abnormality detection instruction for the clearing data, and clean the clearing data to obtain data to be processed; A projection unit, used for projecting the data to be processed using the target model to obtain dimension-reduced data; A calculation unit, used for calculating the reconstruction error of the data to be processed as a target error according to the dimension reduction data; The calculation unit is further used to calculate the abnormality score of the data to be processed according to the target error; A comparison unit, used to compare the anomaly score with the anomaly detection threshold to obtain an anomaly detection result of the clearing data; A generating unit is used to generate an abnormality report of the clearing data according to the abnormality detection result, and send the abnormality report to a designated terminal device.

9. A computer device, characterized in that: The computer device comprises: a memory storing at least one instruction; and A processor executes instructions stored in the memory to implement the clearing data anomaly detection method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, and the at least one instruction is executed by a processor in a computer device to implement the clearing data anomaly detection method as described in any one of claims 1 to 7.

Images