Clearing method and device based on auto-encoder, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides a clearing method, device and equipment based on an auto-encoder and a medium, on one hand, a reading task of clearing data, a generating task of reconstruction errors and a clearing task are executed in parallel based on different threads, the waiting time among the tasks can be shortened, and the working efficiency is improved. The liquidation efficiency is prevented from being influenced by data blockage; on the other hand, the threshold value is reasonably configured to detect abnormal liquidation data, execution of each thread is suspended in time when abnormality is detected so as to prevent the abnormal liquidation data from generating adverse effects on the accuracy of the liquidation result, and meanwhile, an interpretable abnormality report is generated based on the characteristics of the auto-encoder model, so that the accuracy of the liquidation result is improved. Therefore, the abnormal data can be quickly positioned and repaired in an assisted manner.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a clearing method, device, equipment and medium based on an autoencoder. Background Art

[0002] At present, in the financial transaction and clearing business, the scale of data processing is huge and the types of data are numerous. Various transaction instructions, subscription and redemption, fund transfer, share confirmation, settlement and reconciliation operations are highly dependent on accurate data processing and comparison.

[0003] However, with the continuous growth of business volume and the diversification of transaction scenarios, traditional clearing audit methods mainly rely on manual experience or hard-coded rules, which makes it difficult to detect abnormal data or potential risk points in a timely and accurate manner, resulting in the inability to carry out clearing tasks normally. Summary of the invention

[0004] In view of the above, it is necessary to provide a clearing method, device, equipment and medium based on an autoencoder, aiming to solve the problem that the normal execution of clearing tasks is affected by the inability to timely and accurately detect abnormal clearing data or potential risk points.

[0005] A clearing method based on an autoencoder, the clearing method based on an autoencoder comprising: When receiving a liquidation instruction, parsing the liquidation instruction to obtain a target system; Start a first thread to read clearing data from the target system according to a preset data collection strategy; Start the second thread to input the read clearing data into the pre-trained autoencoder model for processing, and obtain the reconstruction error of each clearing data; Get the configuration threshold; When it is detected that the reconstruction error of the clearing data is less than or equal to the configuration threshold, starting a third thread to perform clearing processing based on the detected clearing data; When it is detected that the reconstruction error of the clearing data is greater than the configured threshold, the first thread and the third thread are controlled to suspend execution, and the second thread is controlled to complete the processing of all the read clearing data; When it is detected that the second thread has been executed, obtaining clearing data whose corresponding reconstruction error is greater than the configured threshold as abnormal data; An exception report is generated according to the exception data, and the exception report is sent to a designated terminal device.

[0006] According to a preferred embodiment of the present invention, the first thread, the second thread and the third thread are threads executed in parallel.

[0007] According to a preferred embodiment of the present invention, starting the first thread to read the clearing data from the target system according to a preset data collection strategy includes: Start the first thread to read data from the target system as data to be processed according to a clearing cycle; Performing denoising on the data to be processed to obtain denoised data; The denoised data is standardized using the StandardScaler method to obtain the cleared data.

[0008] According to a preferred embodiment of the present invention, before obtaining the configuration threshold, the method further includes: Obtain historical liquidation data to construct training samples; Initialize the weights of the autoencoder model using the Kaiming initialization algorithm; Using the training samples to train the initialized autoencoder model; During the training process, after each training sample is input into the initialized autoencoder model, output data corresponding to each training sample is obtained as a reconstruction error of each training sample; Construct an error distribution curve based on the reconstruction error of each training sample; An error point is selected from the error distribution curve according to a preset quantile as the configuration threshold.

[0009] According to a preferred embodiment of the present invention, after selecting an error point from the error distribution curve as the configuration threshold according to a preset quantile, the method further includes: Collecting new data of the target system at preset time intervals; Performing supplementary training on the autoencoder model using the newly added data; After the training is completed, the F1 score of the autoencoder model is calculated; When the F1 score is greater than or equal to the configuration score, obtaining a reconstruction error of the newly added data; Updating the error distribution curve according to the reconstruction error of the newly added data; An error point is selected from the updated error distribution curve according to the preset quantile to replace the configuration threshold.

[0010] According to a preferred embodiment of the present invention, generating an exception report according to the abnormal data includes: Marking the abnormal data by field, and obtaining the source and data attribute information of the abnormal data; Acquire the abnormal data with the largest reconstruction error from the abnormal data as the maximum abnormal field; The exception report is generated according to the marked fields, the source and data attribute information of the abnormal data, and the maximum abnormal field.

[0011] According to a preferred embodiment of the present invention, after sending the abnormality report to the designated terminal device, the method further includes: When receiving a response from the designated terminal device to continue executing the liquidation instruction, starting the first thread, the second thread and the third thread; The first thread and the third thread continue to execute from the breakpoints.

[0012] A clearing device based on an autoencoder, the clearing device based on an autoencoder comprising: A parsing unit, configured to parse a liquidation instruction to obtain a target system when receiving the liquidation instruction; A reading unit, used to start a first thread to read clearing data from the target system according to a preset data collection strategy; A processing unit, used to start a second thread to input the read clearing data into a pre-trained autoencoder model for processing, so as to obtain a reconstruction error of each clearing data; An acquisition unit, used for acquiring a configuration threshold; A clearing unit, configured to start a third thread to perform clearing processing based on the detected clearing data when it is detected that the reconstruction error of the clearing data is less than or equal to the configuration threshold; A control unit, configured to control the first thread and the third thread to suspend execution and control the second thread to complete processing of all read clearing data when it is detected that the reconstruction error of the clearing data is greater than a configured threshold; The acquisition unit is further configured to acquire, when it is detected that the second thread has been executed, clearing data whose corresponding reconstruction error is greater than the configured threshold as abnormal data; A generating unit is used to generate an abnormality report according to the abnormal data, and send the abnormality report to a designated terminal device.

[0013] A computer device, comprising: a memory storing at least one instruction; and A processor executes instructions stored in the memory to implement the clearing method based on the autoencoder.

[0014] A computer-readable storage medium stores at least one instruction, and the at least one instruction is executed by a processor in a computer device to implement the clearing method based on an autoencoder.

[0015] It can be seen from the above technical solutions that, on the one hand, the reading task of clearing data, the generation task of reconstruction error and the clearing task are executed in parallel based on different threads, which can reduce the waiting time between tasks and avoid data congestion and affect the clearing efficiency; on the other hand, the threshold is reasonably configured to detect abnormal clearing data, and the execution of each thread is suspended in time when an anomaly is detected, so as to avoid the abnormal clearing data from having an adverse effect on the accuracy of the clearing results. At the same time, an explainable anomaly report is generated based on the characteristics of the autoencoder model, which can assist in quickly locating and repairing abnormal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of a preferred embodiment of the clearing method based on the autoencoder of the present invention; Figure 2 It is a functional module diagram of a preferred embodiment of a clearing device based on an autoencoder of the present invention; Figure 3 It is a structural diagram of a computer device of a preferred embodiment of the present invention for implementing a clearing method based on an autoencoder. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0018] like Figure 1 , which is a flow chart of a preferred embodiment of the clearing method based on the autoencoder of the present invention. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted.

[0019] The clearing method based on the autoencoder is applied to one or more computer devices, wherein the computer device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASIC), programmable gate arrays (FPGA), digital signal processors (DSP), embedded devices, etc.

[0020] The computer device may be any electronic product that can perform human-computer interaction with a user, such as a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), a game console, an interactive network television (IPTV), a smart wearable device, etc.

[0021] The computer device may also include a network device and / or a user device, wherein the network device includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud consisting of a large number of hosts or network servers based on cloud computing.

[0022] The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0023] Among them, Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0024] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0025] The network where the computer device is located includes but is not limited to the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (VPN), etc.

[0026] S10, when a liquidation instruction is received, the liquidation instruction is parsed to obtain a target system.

[0027] In this embodiment, the liquidation instruction may be triggered according to actual liquidation requirements.

[0028] In this embodiment, the target system may be a financial transaction system.

[0029] S11, starting a first thread to read clearing data from the target system according to a preset data collection strategy.

[0030] In this embodiment, starting the first thread to read the clearing data from the target system according to the preset data collection strategy includes: Start the first thread to read data from the target system as data to be processed according to a clearing cycle; Performing denoising on the data to be processed to obtain denoised data; The denoised data is standardized using the StandardScaler method to obtain the cleared data.

[0031] The settlement cycle may be configured according to business data, such as weekly, monthly, etc.

[0032] Of course, when it is detected that new data is generated, the first thread may be started to read data from the target system as the data to be processed.

[0033] The step of performing denoising on the data to be processed to obtain denoised data includes: The missing data in the data to be processed are deleted, and the invalid fields in the data to be processed are removed to obtain the denoised data.

[0034] The invalid fields may include, but are not limited to: date, time, etc.

[0035] Among them, the use of the StandardScaler method can effectively improve the convergence speed and reduce the impact of outliers, thereby improving the efficiency of subsequent data processing.

[0036] S12, starting the second thread to input the read clearing data into a pre-trained autoencoder (AE) model for processing to obtain a reconstruction error of each clearing data.

[0037] In this embodiment, the autoencoder model includes an encoder and a decoder.

[0038] Specifically, the autoencoder model may be a multi-layer encoding and decoding structure. For example, the autoencoder model may include: input layer -> hidden layer (8 neurons) -> hidden layer (4 neurons) -> hidden layer (8 neurons) -> output layer. This structure is suitable for processing complex data patterns.

[0039] The autoencoder model can also be a single-layer encoding and decoding structure. For example, the autoencoder model can include: input layer -> hidden layer (4 neurons) -> output layer. This structure is suitable for processing simple data patterns.

[0040] Through the encoder, the input data can be mapped to a low-dimensional latent representation space.

[0041] By means of the decoder, the original data can be reconstructed from the latent representation.

[0042] By training the autoencoder model, the input and output of the model are made as close as possible. For normal samples, the autoencoder can accurately reconstruct; for abnormal samples, the reconstruction error is usually large due to deviation from the distribution of the training data.

[0043] The autoencoder can output the fields of abnormal samples and the corresponding degree of deviation, thereby effectively enhancing the interpretability of detection.

[0044] In addition, by utilizing the deep learning autoencoder model, the complex data distribution of massive transaction data can be learned under unsupervised conditions without the need for prior knowledge, thereby automatically identifying abnormal data points, providing important support for financial institutions to improve data audit quality and reduce labor costs.

[0045] S13, obtaining a configuration threshold.

[0046] In this embodiment, before obtaining the configuration threshold, the method further includes: Obtain historical liquidation data to construct training samples; Initialize the weights of the autoencoder model using the Kaiming initialization algorithm; Using the training samples to train the initialized autoencoder model; During the training process, after each training sample is input into the initialized autoencoder model, output data corresponding to each training sample is obtained as a reconstruction error of each training sample; Construct an error distribution curve based on the reconstruction error of each training sample; An error point is selected from the error distribution curve according to a preset quantile as the configuration threshold.

[0047] Among them, the Kaiming initialization algorithm is used to initialize the weights of the autoencoder model, which can effectively alleviate the gradient vanishing and explosion problems and improve the training efficiency and stability, thereby ensuring excellent model convergence performance.

[0048] The preset quantile may be configured according to the characteristics of the data, for example, the preset quantile may be 95%.

[0049] Of course, the maximum reconstruction error may also be selected as the configuration threshold, which may be specifically configured according to the characteristics of the error distribution curve.

[0050] Through the above embodiments, the threshold can be reasonably configured according to the error distribution characteristics, so as to effectively detect abnormal data.

[0051] In this embodiment, after selecting an error point from the error distribution curve as the configuration threshold according to a preset quantile, the method further includes: Collecting new data of the target system at preset time intervals; Performing supplementary training on the autoencoder model using the newly added data; After the training is completed, the F1 score of the autoencoder model is calculated; When the F1 score is greater than or equal to the configuration score, obtaining a reconstruction error of the newly added data; Updating the error distribution curve according to the reconstruction error of the newly added data; An error point is selected from the updated error distribution curve according to the preset quantile to replace the configuration threshold.

[0052] The preset time interval may be configured as 1 month or the like.

[0053] Of course, the autoencoder model may also be supplemented with training according to the amount of newly added data. For example, when the amount of newly added data reaches a certain threshold, supplementary training of the autoencoder model may be triggered.

[0054] The configuration score may be optimized after a large number of experiments.

[0055] Through the above embodiments, when the performance of the model is not sufficient to accurately detect the newly added data, the supplementary training of the model and the update of the configuration threshold can be triggered in time, thereby improving the adaptability of the model and the rationality of the threshold configuration, thereby ensuring the accuracy of data anomaly detection and improving the robustness of the model. At the same time, continuously adapting to the latest data distribution can also reduce the cost of retraining.

[0056] S14, when it is detected that the reconstruction error of the clearing data is less than or equal to the configuration threshold, start the third thread to perform clearing processing based on the detected clearing data.

[0057] In this embodiment, when it is detected that the reconstruction error of the clearing data is less than or equal to the configuration threshold, it means that no abnormal clearing data is detected. At this time, the third thread can be started to perform clearing processing based on the detected clearing data.

[0058] In this embodiment, the first thread, the second thread and the third thread are threads executed in parallel. Based on different threads, the reading task of clearing data, the generation task of reconstruction error and the clearing task are executed in parallel, which can reduce the mutual waiting time between tasks, thereby avoiding the influence of clearing efficiency due to data congestion.

[0059] S15, when it is detected that the reconstruction error of the clearing data is greater than the configured threshold, the first thread and the third thread are controlled to suspend execution, and the second thread is controlled to complete the processing of all the read clearing data.

[0060] In this embodiment, when it is detected that the reconstruction error of the clearing data is greater than the configured threshold, it means that abnormal clearing data is detected, and clearing is suspended at this time to avoid errors.

[0061] S16, when it is detected that the second thread has been executed, obtaining the corresponding clearing data whose reconstruction error is greater than the configured threshold as abnormal data.

[0062] In this embodiment, the abnormal data is obtained for subsequent further processing.

[0063] S17, generating an abnormality report according to the abnormal data, and sending the abnormality report to a designated terminal device.

[0064] The designated terminal device may be a terminal device of a relevant staff member who is responsible for maintaining the clearing data.

[0065] In the above embodiment, by sending the abnormal report of the clearing data to the terminal device of the relevant staff in time, a quick response to the abnormality can be achieved to avoid affecting the efficiency and accuracy of clearing due to data abnormality.

[0066] In this embodiment, generating an exception report according to the abnormal data includes: Marking the abnormal data by field, and obtaining the source and data attribute information of the abnormal data; Acquire the abnormal data with the largest reconstruction error from the abnormal data as the maximum abnormal field; The exception report is generated according to the marked fields, the source and data attribute information of the abnormal data, and the maximum abnormal field.

[0067] Through the above embodiments, comprehensive detection results can be output to assist in quickly locating and repairing abnormal data.

[0068] In this embodiment, after sending the abnormality report to the designated terminal device, the method further includes: When receiving a response from the designated terminal device to continue executing the liquidation instruction, starting the first thread, the second thread and the third thread; The first thread and the third thread continue to execute from the breakpoints.

[0069] Through the above embodiments, clearing can be continued from the breakpoint after data recovery, thereby improving clearing efficiency.

[0070] It can be seen from the above technical solutions that, on the one hand, the reading task of clearing data, the generation task of reconstruction error and the clearing task are executed in parallel based on different threads, which can reduce the waiting time between tasks and avoid data congestion and affect the clearing efficiency; on the other hand, the threshold is reasonably configured to detect abnormal clearing data, and the execution of each thread is suspended in time when an anomaly is detected, so as to avoid the abnormal clearing data from having an adverse effect on the accuracy of the clearing results. At the same time, an explainable anomaly report is generated based on the characteristics of the autoencoder model, which can assist in quickly locating and repairing abnormal data.

[0071] like Figure 2 As shown, it is a functional module diagram of a preferred embodiment of the clearing device based on the autoencoder of the present invention. The clearing device 11 based on the autoencoder includes a parsing unit 110, a reading unit 111, a processing unit 112, an acquisition unit 113, a clearing unit 114, a control unit 115, and a generating unit 116. The module / unit referred to in the present invention refers to a series of computer program segments that can be executed by a processor and can perform fixed functions, which are stored in a memory. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0072] The parsing unit 110 is used to parse the liquidation instruction to obtain the target system when receiving the liquidation instruction; The reading unit 111 is used to start a first thread to read liquidation data from the target system according to a preset data collection strategy; The processing unit 112 is used to start the second thread to input the read clearing data into the pre-trained autoencoder model for processing to obtain the reconstruction error of each clearing data; The acquisition unit 113 is used to acquire a configuration threshold; The clearing unit 114 is configured to start a third thread to perform clearing processing based on the detected clearing data when it is detected that the reconstruction error of the clearing data is less than or equal to the configured threshold; The control unit 115 is used to control the first thread and the third thread to suspend execution and control the second thread to complete processing of all read clearing data when it is detected that the reconstruction error of the clearing data is greater than the configured threshold; The acquisition unit 113 is further configured to acquire, when it is detected that the second thread has been executed, clearing data whose corresponding reconstruction error is greater than the configured threshold as abnormal data; The generating unit 116 is configured to generate an abnormality report according to the abnormal data, and send the abnormality report to a designated terminal device.

[0073] It can be seen from the above technical solutions that, on the one hand, the reading task of clearing data, the generation task of reconstruction error and the clearing task are executed in parallel based on different threads, which can reduce the waiting time between tasks and avoid data congestion and affect the clearing efficiency; on the other hand, the threshold is reasonably configured to detect abnormal clearing data, and the execution of each thread is suspended in time when an anomaly is detected, so as to avoid the abnormal clearing data from having an adverse effect on the accuracy of the clearing results. At the same time, an explainable anomaly report is generated based on the characteristics of the autoencoder model, which can assist in quickly locating and repairing abnormal data.

[0074] like Figure 3 The figure is a schematic diagram of the structure of a computer device of a preferred embodiment of the present invention to implement the clearing method based on the autoencoder.

[0075] The computer device 1 may include a memory 12, a processor 13 and a bus, and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a clearing program based on an autoencoder.

[0076] Those skilled in the art will appreciate that the schematic diagram is merely an example of the computer device 1 and does not constitute a limitation on the computer device 1. The computer device 1 may be a bus-type structure or a star-type structure. The computer device 1 may also include more or less other hardware or software than shown in the diagram, or a different arrangement of components. For example, the computer device 1 may also include input and output devices, network access devices, etc.

[0077] It should be noted that the computer device 1 is only an example, and other existing or future electronic products that are suitable for the present invention should also be included in the protection scope of the present invention and included here by reference.

[0078] Among them, the memory 12 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the computer device 1, such as a mobile hard disk of the computer device 1. In other embodiments, the memory 12 can also be an external storage device of the computer device 1, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 1. Further, the memory 12 can also include both an internal storage unit of the computer device 1 and an external storage device. The memory 12 can not only be used to store application software and various types of data installed in the computer device 1, such as the code of the clearing program based on the autoencoder, but also can be used to temporarily store data that has been output or is to be output.

[0079] In some embodiments, the processor 13 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 13 is the control core (Control Unit) of the computer device 1, and uses various interfaces and lines to connect various components of the entire computer device 1, and executes various functions and processes data of the computer device 1 by running or executing programs or modules stored in the memory 12 (for example, executing a clearing program based on an autoencoder, etc.), and calling data stored in the memory 12.

[0080] The processor 13 executes the operating system of the computer device 1 and various installed applications. The processor 13 executes the applications to implement the steps in the above-mentioned embodiments of the clearing method based on the autoencoder, for example Figure 1 Steps shown.

[0081] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to implement the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the computer device 1. For example, the computer program may be divided into a parsing unit 110, a reading unit 111, a processing unit 112, an acquisition unit 113, a clearing unit 114, a control unit 115, and a generating unit 116.

[0082] The above-mentioned integrated unit implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, and includes a number of instructions for enabling a computer device (which can be a personal computer, a computer device, or a network device, etc.) or a processor to execute the part of the clearing method based on the autoencoder described in each embodiment of the present invention.

[0083] If the module / unit integrated in the computer device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware devices through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of each of the above-mentioned method embodiments can be implemented.

[0084] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory, etc.

[0085] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0086] The blockchain referred to in this invention is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the blockchain underlying platform, platform product service layer, and application service layer.

[0087] The bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 The bus is represented by only one straight line, but it does not mean that there is only one bus or one type of bus. The bus is configured to realize the connection and communication between the memory 12 and at least one processor 13, etc.

[0088] Although not shown, the computer device 1 may also include a power source (such as a battery) for supplying power to various components. Preferably, the power source may be logically connected to the at least one processor 13 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include any components such as one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, and power status indicators. The computer device 1 may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0089] Furthermore, the computer device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is usually used to establish a communication connection between the computer device 1 and other computer devices.

[0090] Optionally, the computer device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), or a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the computer device 1 and to display a visual user interface.

[0091] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0092] It can be understood by those skilled in the art that Figure 3 The structure shown does not constitute a limitation on the computer device 1, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0093] Combination Figure 1 , the memory 12 in the computer device 1 stores a plurality of instructions to implement a clearing method based on an autoencoder, and the processor 13 can execute the plurality of instructions to implement: When receiving a liquidation instruction, parsing the liquidation instruction to obtain a target system; Start a first thread to read clearing data from the target system according to a preset data collection strategy; Start the second thread to input the read clearing data into the pre-trained autoencoder model for processing, and obtain the reconstruction error of each clearing data; Get the configuration threshold; When it is detected that the reconstruction error of the clearing data is less than or equal to the configuration threshold, starting a third thread to perform clearing processing based on the detected clearing data; When it is detected that the reconstruction error of the clearing data is greater than the configured threshold, the first thread and the third thread are controlled to suspend execution, and the second thread is controlled to complete the processing of all the read clearing data; When it is detected that the second thread has been executed, obtaining clearing data whose corresponding reconstruction error is greater than the configured threshold as abnormal data; An exception report is generated according to the exception data, and the exception report is sent to a designated terminal device.

[0094] Specifically, the specific implementation method of the processor 13 for the above instructions can refer to Figure 1The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0095] It should be noted that the data involved in this case were all obtained legally.

[0096] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0097] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0098] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0099] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0100] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0101] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0102] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the present invention can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A clearing method based on an autoencoder, characterized in that: The clearing method based on the autoencoder includes: When receiving a liquidation instruction, parsing the liquidation instruction to obtain a target system; Start a first thread to read clearing data from the target system according to a preset data collection strategy; Start the second thread to input the read clearing data into the pre-trained autoencoder model for processing, and obtain the reconstruction error of each clearing data; Get the configuration threshold; When it is detected that the reconstruction error of the clearing data is less than or equal to the configuration threshold, starting a third thread to perform clearing processing based on the detected clearing data; When it is detected that the reconstruction error of the clearing data is greater than the configured threshold, the first thread and the third thread are controlled to suspend execution, and the second thread is controlled to complete the processing of all the read clearing data; When it is detected that the second thread has been executed, obtaining clearing data whose corresponding reconstruction error is greater than the configured threshold as abnormal data; An exception report is generated according to the exception data, and the exception report is sent to a designated terminal device.

2. The clearing method based on the autoencoder according to claim 1, characterized in that: The first thread, the second thread, and the third thread are threads executed in parallel.

3. The clearing method based on the autoencoder according to claim 1, characterized in that: The starting of the first thread to read the clearing data from the target system according to the preset data collection strategy includes: Start the first thread to read data from the target system as data to be processed according to a clearing cycle; Performing denoising on the data to be processed to obtain denoised data; The denoised data is standardized using the StandardScaler method to obtain the cleared data.

4. The clearing method based on the autoencoder according to claim 1, characterized in that: Before obtaining the configuration threshold, the method further includes: Obtain historical liquidation data to construct training samples; Initialize the weights of the autoencoder model using the Kaiming initialization algorithm; Using the training samples to train the initialized autoencoder model; During the training process, after each training sample is input into the initialized autoencoder model, output data corresponding to each training sample is obtained as a reconstruction error of each training sample; Construct an error distribution curve based on the reconstruction error of each training sample; An error point is selected from the error distribution curve according to a preset quantile as the configuration threshold.

5. The clearing method based on the autoencoder according to claim 4, characterized in that: After selecting an error point from the error distribution curve as the configuration threshold according to a preset quantile, the method further includes: Collecting new data of the target system at preset time intervals; Performing supplementary training on the autoencoder model using the newly added data; After the training is completed, the F1 score of the autoencoder model is calculated; When the F1 score is greater than or equal to the configuration score, obtaining a reconstruction error of the newly added data; Updating the error distribution curve according to the reconstruction error of the newly added data; An error point is selected from the updated error distribution curve according to the preset quantile to replace the configuration threshold.

6. The clearing method based on the autoencoder according to claim 1, characterized in that: Generating an abnormality report according to the abnormal data includes: Marking the abnormal data by field, and obtaining the source and data attribute information of the abnormal data; Acquire the abnormal data with the largest reconstruction error from the abnormal data as the maximum abnormal field; The exception report is generated according to the marked fields, the source and data attribute information of the abnormal data, and the maximum abnormal field.

7. The clearing method based on the autoencoder according to claim 1, characterized in that: After sending the abnormality report to the designated terminal device, the method further includes: When receiving a response from the designated terminal device to continue executing the liquidation instruction, starting the first thread, the second thread and the third thread; The first thread and the third thread continue to execute from the breakpoints.

8. A clearing device based on an autoencoder, characterized in that: The clearing device based on the autoencoder comprises: A parsing unit, configured to parse a liquidation instruction to obtain a target system when receiving the liquidation instruction; A reading unit, used to start a first thread to read clearing data from the target system according to a preset data collection strategy; A processing unit, used to start a second thread to input the read clearing data into a pre-trained autoencoder model for processing, so as to obtain a reconstruction error of each clearing data; An acquisition unit, used for acquiring a configuration threshold; A clearing unit, configured to start a third thread to perform clearing processing based on the detected clearing data when it is detected that the reconstruction error of the clearing data is less than or equal to the configuration threshold; A control unit, configured to control the first thread and the third thread to suspend execution and control the second thread to complete processing of all read clearing data when it is detected that the reconstruction error of the clearing data is greater than a configured threshold; The acquisition unit is further configured to acquire, when it is detected that the second thread has been executed, clearing data whose corresponding reconstruction error is greater than the configured threshold as abnormal data; A generating unit is used to generate an abnormality report according to the abnormal data, and send the abnormality report to a designated terminal device.

9. A computer device, characterized in that: The computer device comprises: a memory storing at least one instruction; and A processor executes instructions stored in the memory to implement the clearing method based on the autoencoder as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, and the at least one instruction is executed by a processor in a computer device to implement the clearing method based on the autoencoder as described in any one of claims 1 to 7.

Images