A large-scale data processing method and platform for digital economy monitoring

Abstract

The present application provides a large-scale data processing method and platform for digital economy monitoring, which relates to the technical field of digital economy monitoring. The method monitors the financial transaction process in real time to obtain financial transaction flow data; pre-processes the financial transaction flow data, performs heterogeneous screening on the pre-processed financial transaction flow data, and obtains homogeneous transaction flow data in the financial transaction process; performs auto-encoding on each data point in the homogeneous transaction flow data to obtain the encoding difference loss of each data point in the auto-encoding process, and determines the auto-encoding cost of the homogeneous transaction flow data in the auto-encoding process through all the encoding difference losses; uses the auto-encoding cost and the corresponding encoding difference loss to determine the data anomaly score of each data point in the homogeneous transaction flow data, and performs data anomaly processing on the homogeneous transaction flow data based on all the data anomaly scores. The present application can efficiently identify abnormal data in digital economy monitoring to improve the efficiency of data anomaly processing.

Description

Technical Field

[0001] The present application relates to the technical field of digital economy monitoring, and more specifically, to a large-scale data processing method and platform for digital economy monitoring. Background Art

[0002] Large-scale data processing for digital economy monitoring is a complex process that aims to reveal the development trends, risk points and potential opportunities of the digital economy in real time by collecting, storing, analyzing and mining large amounts of multi-source and heterogeneous data. This process usually involves key links such as data collection, cleaning, storage, processing and visualization, and relies on high-performance computing, big data analysis and artificial intelligence technologies to achieve efficient and accurate monitoring.

[0003] However, large-scale data often contains a large amount of redundant, incomplete or erroneous data, which can significantly affect the accuracy of the analysis results. For example, duplicate transaction records, abnormal numerical fields or missing geographic location information can lead to deviations in model predictions. Therefore, how to efficiently identify abnormal data in digital economic monitoring to improve the efficiency of data anomaly processing is an important problem. Summary of the invention

[0004] The present application provides a large-scale data processing method and platform for digital economy monitoring, which can efficiently identify abnormal data in digital economy monitoring to improve the efficiency of data anomaly processing.

[0005] In a first aspect, the present application provides a large-scale data processing method for digital economy monitoring, the processing method comprising the following steps:

[0006] Monitor the financial transaction process in real time and obtain financial transaction flow data;

[0007] Preprocessing the financial transaction flow data, and then performing heterogeneous screening on the preprocessed financial transaction flow data to obtain homogeneous transaction flow data in the financial transaction process;

[0008] Performing self-encoding on each data point in the homogeneous transaction flow data, thereby obtaining the encoding difference loss of each data point in the self-encoding process, and determining the self-encoding cost of the homogeneous transaction flow data in the self-encoding process through all the encoding difference losses;

[0009] The data anomaly score of each data point in the transaction flow homogeneous data is determined by using the self-encoding cost and the corresponding encoding difference loss, and data anomaly processing is performed on the transaction flow homogeneous data according to all the data anomaly scores.

[0010] In this embodiment, the financial transaction flow data refers to a data flow consisting of detailed information of each financial transaction in the financial market.

[0011] In this embodiment, preprocessing the financial transaction flow data includes removing duplicate data, processing missing data, and standardizing data formats.

[0012] In this embodiment, the pre-processed financial transaction flow data is subjected to heterogeneous screening to obtain transaction flow homogeneous data in the financial transaction process, specifically including:

[0013] Determining data anomaly values ​​of preprocessed financial transaction stream data;

[0014] The data points in the preprocessed financial transaction flow data are screened for heterogeneity according to the data heterogeneity value, thereby obtaining the transaction flow homogeneity data in the financial transaction process.

[0015] In this embodiment, determining the data abnormality value of the pre-processed financial transaction flow data specifically includes:

[0016] Determine the central aggregation value of the preprocessed financial transaction stream data;

[0017] Determining a data deviation value of the preprocessed financial transaction flow data according to the central aggregation value;

[0018] The data anomaly value of the preprocessed financial transaction flow data is determined according to the central aggregation value and the data deviation value.

[0019] In this embodiment, each data point in the transaction flow homogeneous data is self-encoded, and then the encoding difference loss of each data point in the self-encoding process is obtained, which specifically includes:

[0020] Obtaining preset encoding weights and decoding weights;

[0021] For each data point in the transaction flow homogeneous data, encoding the data point according to the encoding weight to obtain a potential feature representation of the data point;

[0022] Decoding the latent feature representation according to the decoding weight to obtain a reconstructed latent feature representation;

[0023] The encoding difference loss of the data point in the self-encoding process is determined by the reconstructed potential feature representation, and then the encoding difference loss of each data point in the self-encoding process is obtained.

[0024] In this embodiment, determining the self-encoding cost of the homogeneous transaction flow data in the self-encoding process through all encoding difference losses specifically includes:

[0025] Obtaining the weight corresponding to each data point in the transaction flow homogeneous data;

[0026] The self-encoding cost of the homogeneous data of the transaction flow in the self-encoding process is determined according to the weight corresponding to each data point and all encoding difference losses.

[0027] In this embodiment, using the self-encoding cost and the corresponding encoding difference loss to determine the data anomaly score of each data point in the transaction flow homogeneous data specifically includes:

[0028] For each data point in the transaction flow homogeneous data, the ratio of the encoding difference loss corresponding to the data point to the self-encoding cost is used as the data anomaly score of the data point, thereby obtaining the data anomaly score of each data point in the transaction flow homogeneous data.

[0029] In this embodiment, performing data anomaly processing on the transaction flow homogeneous data according to all data anomaly scores is to abnormally delete all data points in the transaction flow homogeneous data whose data anomaly scores are greater than a set threshold.

[0030] In a second aspect, the present application provides a large-scale data processing platform for digital economy monitoring, which is used to execute a large-scale data processing method for digital economy monitoring, and the processing platform includes:

[0031] The data monitoring module is used to monitor the financial transaction process in real time and obtain financial transaction flow data;

[0032] A preprocessing module, used to preprocess the financial transaction flow data, and then perform heterogeneous screening on the preprocessed financial transaction flow data to obtain transaction flow homogeneous data in the financial transaction process;

[0033] A self-encoding module, used for self-encoding each data point in the transaction flow homogeneous data, thereby obtaining the encoding difference loss of each data point in the self-encoding process, and determining the self-encoding cost of the transaction flow homogeneous data in the self-encoding process through all the encoding difference losses;

[0034] The exception processing module is used to determine the data anomaly score of each data point in the transaction flow homogeneous data using the self-encoding cost and the corresponding encoding difference loss, and perform data anomaly processing on the transaction flow homogeneous data according to all the data anomaly scores.

[0035] The technical solution provided by the embodiments disclosed in this application has the following beneficial effects:

[0036] Financial transaction flow data is obtained by real-time monitoring of the financial transaction process; the financial transaction flow data is preprocessed, and then the preprocessed financial transaction flow data is subjected to heterogeneous screening to obtain transaction flow homogeneous data in the financial transaction process; each data point in the transaction flow homogeneous data is self-encoded to obtain the encoding difference loss of each data point in the self-encoding process, and the self-encoding cost of the transaction flow homogeneous data in the self-encoding process is determined by all the encoding difference losses; the data anomaly score of each data point in the transaction flow homogeneous data is determined using the self-encoding cost and the corresponding encoding difference loss, and data anomaly processing is performed on the transaction flow homogeneous data according to all the data anomaly scores.

[0037] It can be seen that in the present application, firstly, the redundancy and noise in the data can be significantly reduced through the preprocessing step, the integrity and consistency of the data can be improved, and the data can be eliminated based on the data abnormal value, so as to reduce the scale of data to be processed and improve the processing efficiency, which can help to effectively deal with the high frequency, dynamic and complex characteristics of financial transactions and realize efficient processing of abnormal data; then, by evaluating the encoding difference loss of each data point, it can be effectively identified which data points deviate greatly from the reconstruction results of the model, and possible abnormal data can be more accurately identified, and the autoencoder has good automation characteristics when processing large-scale financial transaction flow data; finally, by calculating the data anomaly score of each data point, the degree of abnormality of the data point can be measured more accurately, and the platform can quickly locate potential abnormal data in large-scale transaction flow data, thereby improving the efficiency of data anomaly processing.

[0038] To sum up, the technical solution adopted in this application can efficiently identify abnormal data in digital economy monitoring to improve the efficiency of data anomaly processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0040] Figure 1 is a flow chart of a large-scale data processing method for digital economy monitoring provided by the present application;

[0041] Figure 2 is an exemplary flow chart for determining transaction flow homogeneity data in a financial transaction process provided by the present application;

[0042] Figure 3is an exemplary flow chart for determining the encoding difference loss of each data point in the self-encoding process provided by the present application;

[0043] Figure 4 This is a module structure diagram of a large-scale data processing platform for digital economy monitoring provided in this application. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0045] The embodiment of the present application provides a large-scale data processing method and platform for digital economic monitoring, the core of which is to monitor the financial transaction process in real time to obtain financial transaction flow data; pre-process the financial transaction flow data, and then perform heterogeneous screening on the pre-processed financial transaction flow data to obtain homogeneous transaction flow data in the financial transaction process; self-encode each data point in the homogeneous transaction flow data, and then obtain the encoding difference loss of each data point in the self-encoding process, and determine the self-encoding cost of the homogeneous transaction flow data in the self-encoding process through all the encoding difference losses; use the self-encoding cost and the corresponding encoding difference loss to determine the data anomaly score of each data point in the homogeneous transaction flow data, and perform data anomaly processing on the homogeneous transaction flow data based on all the data anomaly scores. The above scheme can efficiently identify abnormal data in digital economic monitoring to improve the efficiency of data anomaly processing.

[0046] Embodiment 1: In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods. Figure 1 As shown in FIG. 1 , this figure is an exemplary flow chart of a large-scale data processing method for digital economy monitoring according to this embodiment of the present application, and the processing method includes the following steps:

[0047] In step S1, the financial transaction process is monitored in real time to obtain financial transaction flow data.

[0048] In specific implementation, by designing an efficient real-time data collection and processing framework, we can monitor the key dynamics of the financial transaction process in the financial market in real time, use standardized interfaces (such as FIX protocol, WebSocket) to connect with exchanges and payment platforms, and collect real-time data streams, namely financial transaction flow data. This financial transaction flow data refers to a data stream composed of detailed information of each financial transaction in the financial market.

[0049] In step S2, the financial transaction flow data is preprocessed, and then the preprocessed financial transaction flow data is heterogeneously screened to obtain transaction flow homogeneous data in the financial transaction process.

[0050] In this embodiment, preprocessing the financial transaction flow data includes removing duplicate data, processing missing data, and standardizing data formats.

[0051] In the specific implementation, first of all, in the financial transaction flow data, duplicate data may come from overlapping records from multiple sources, repeated transmission caused by network delays, etc. The key to removing duplicate data is to ensure the uniqueness of the data. It can be removed by checking the unique identifier and deleting data records with the same unique identifier; then, missing data is caused by data transmission errors or incomplete records. Records with missing key fields (such as transaction amount, transaction time) can be directly deleted. For records with missing data, machine learning models (such as regression models or deep learning networks) can be used to predict missing values; finally, financial transaction flow data usually comes from various sources, and the field naming and data format may not be unified, so they need to be standardized, which will not be elaborated here.

[0052] Preferably, in this embodiment, reference Figure 2 As shown, this figure is an exemplary flow chart of determining homogeneous transaction flow data in a financial transaction process in an embodiment of the present application. In this embodiment, heterogeneous screening is performed on the pre-processed financial transaction flow data to obtain homogeneous transaction flow data in the financial transaction process, which can be specifically implemented by the following steps:

[0053] First, in step S21, the data anomaly value of the preprocessed financial transaction flow data is determined;

[0054] Then, in step S22, each data point in the pre-processed financial transaction flow data is screened for abnormality according to the data abnormality value, so as to obtain transaction flow homogeneous data in the financial transaction process.

[0055] In specific implementation, first, the data anomaly value of the preprocessed financial transaction flow data can be determined, and the data anomaly value represents the data oscillation and fluctuation degree of all data points in the financial transaction flow data, and the data anomaly value is proportional to the data fluctuation degree; then, the data points in the preprocessed financial transaction flow data are screened for anomalies based on the data anomaly value, and all data points in the financial transaction flow data whose data anomaly values ​​are greater than the anomaly threshold can be eliminated, and the data stream composed of all data points in the financial transaction flow data whose data anomaly values ​​are not greater than the anomaly threshold can be used as the transaction flow homogeneous data in the financial transaction process, wherein the anomaly threshold is a threshold used to measure the fluctuation degree of the data points, which can be set according to historical experiments and will not be elaborated here.

[0056] In this embodiment, the data anomaly value of the pre-processed financial transaction flow data may be determined in the following manner, namely:

[0057] Determine the central aggregation value of the preprocessed financial transaction stream data;

[0058] Determining a data deviation value of the preprocessed financial transaction flow data according to the central aggregation value;

[0059] The data anomaly value of the preprocessed financial transaction flow data is determined according to the central aggregation value and the data deviation value.

[0060] In specific implementation, first, the central aggregation value of the preprocessed financial transaction flow data can be determined, wherein the central aggregation value represents the data trend value of the preprocessed financial transaction flow data, and the ratio of the sum of all data points contained in the preprocessed financial transaction flow data to the total number of data points can be used as the central aggregation value of the preprocessed financial transaction flow data; the data deviation value of the preprocessed financial transaction flow data can be determined according to the central aggregation value, wherein the data deviation value represents the overall deviation degree of the preprocessed financial transaction flow data, and the deviation value corresponding to each data point in the preprocessed financial transaction flow data can be subtracted from the central aggregation value, so as to obtain the deviation value corresponding to each data point, so as to use the mean of the deviation values ​​corresponding to all data points as the data deviation value of the preprocessed financial transaction flow data; finally, the data anomaly value of the preprocessed financial transaction flow data can be determined according to the central aggregation value and the data deviation value, wherein the data anomaly value can be determined according to the following formula: in, Represents the data anomaly value of the preprocessed financial transaction stream data, represents the data deviation value of the preprocessed financial transaction stream data, represents the central aggregation value of the preprocessed financial transaction flow data, n represents the total number of data points in the preprocessed financial transaction flow data, Represents the i-th data point of the preprocessed financial transaction flow data. It should be noted that preprocessing steps (such as removing duplicate data, filling missing values ​​and format standardization) can significantly reduce redundancy and noise in the data, improve data integrity and consistency, and eliminate data based on data abnormal values, reduce the size of data to be processed, and improve processing efficiency, which can help effectively cope with the high frequency, dynamic and complex characteristics of financial transactions and achieve efficient processing of abnormal data.

[0061] In step S3, each data point in the transaction flow homogeneous data is self-encoded to obtain the encoding difference loss of each data point in the self-encoding process, and the self-encoding cost of the transaction flow homogeneous data in the self-encoding process is determined by all the encoding difference losses.

[0062] Preferably, in this embodiment, reference Figure 3 As shown, this figure is an exemplary flow chart of determining the encoding difference loss of each data point in the self-encoding process in an embodiment of the present application. In this embodiment, each data point in the homogeneous data of the transaction flow is self-encoded, and then the encoding difference loss of each data point in the self-encoding process is obtained, which can be specifically implemented by the following steps:

[0063] First, in step S31, a preset encoding weight and decoding weight are obtained;

[0064] Then, in step S32, for each data point in the transaction flow homogeneous data, the data point is encoded according to the encoding weight to obtain a potential feature representation of the data point;

[0065] Secondly, in step S33, the latent feature representation is decoded according to the decoding weight to obtain a reconstructed latent feature representation;

[0066] Finally, in step S34, the encoding difference loss of the data point in the self-encoding process is determined by the reconstructed latent feature representation, and then the encoding difference loss of each data point in the self-encoding process is obtained.

[0067] In the specific implementation, first, the encoding weight and the decoding weight can be set according to historical experiments, which will not be repeated here; then, for each data point in the homogeneous data of the transaction flow, the data point can be encoded according to the encoding weight, that is, the encoding weight and the data point are input into the encoding function in the unsupervised network framework for encoding operation to obtain the potential feature representation of the data point, which is a high-level abstract representation of the data point. The high-dimensional data point can be compressed into a low-dimensional vector space through the encoding function, and the potential feature representation can be obtained through compression; secondly, the potential feature representation can be decoded according to the decoding weight, that is, the decoding weight and the data point are input into the decoding function in the unsupervised network framework for decoding operation to obtain the reconstructed potential feature representation, which refers to the potential feature representation after decoding; finally, the encoding difference loss of the data point in the self-encoding process can be determined by reconstructing the potential feature representation, wherein the encoding difference loss represents the difference loss generated in the process of self-encoding the potential feature representation of the data point, and the Euclidean norm of the difference between the data point and the reconstructed potential feature representation can be used as the encoding difference loss of the data point in the self-encoding process. The encoding difference loss of each data point in the self-encoding process can be obtained in the above manner.

[0068] In this embodiment, the self-encoding cost of the homogeneous transaction flow data in the self-encoding process is determined by all encoding difference losses in the following manner, namely:

[0069] Obtaining the weight corresponding to each data point in the transaction flow homogeneous data;

[0070] The self-encoding cost of the homogeneous data of the transaction flow in the self-encoding process is determined according to the weight corresponding to each data point and all encoding difference losses.

[0071] In the specific implementation, first, the weight corresponding to each data point in the transaction flow homogeneous data can be obtained. The weight is learned by the deep learning model in the training process, and the corresponding weight reflects the contribution of each data point to the deep learning model; then, the self-encoding cost of the transaction flow homogeneous data in the self-encoding process can be determined according to the weight corresponding to each data point and all encoding difference losses, where the self-encoding cost refers to the overall difference cost generated by the transaction flow homogeneous data in the self-encoding process. The weight corresponding to each data point and the encoding difference loss corresponding to the data point can be multiplied, and the sum of all results can be used as the self-encoding cost of the transaction flow homogeneous data in the self-encoding process.

[0072] It should be noted that by evaluating the encoding difference loss of each data point, it is possible to effectively identify which data points deviate greatly from the reconstruction results of the model, and to more accurately identify possible abnormal data. In addition, the autoencoder has good automation characteristics when processing large-scale financial transaction flow data. By calculating the encoding difference loss of each data point, the abnormality of the data can be evaluated quickly and automatically, avoiding manual inspection and human intervention.

[0073] In step S4, the data anomaly score of each data point in the transaction flow homogeneous data is determined using the self-encoding cost and the corresponding encoding difference loss, and data anomaly processing is performed on the transaction flow homogeneous data according to all the data anomaly scores.

[0074] In this embodiment, the data anomaly score of each data point in the transaction flow homogeneous data is determined by using the self-encoding cost and the corresponding encoding difference loss, which can be specifically determined in the following manner, namely:

[0075] For each data point in the transaction flow homogeneous data, the ratio of the encoding difference loss corresponding to the data point to the self-encoding cost is used as the data anomaly score of the data point, thereby obtaining the data anomaly score of each data point in the transaction flow homogeneous data.

[0076] It should be noted that in this application, the data anomaly score is used to indicate the degree of abnormality of a data point. In actual implementation, if the anomaly score of a data point is low, it means that the encoding difference loss of the data point is small, and it is not much different from the reconstruction result of the model, and it is usually considered to be normal data. If the anomaly score of a data point is high, it means that its encoding difference loss is large, indicating that the reconstruction error of the data point is large during the self-encoding process, and it may be abnormal data or an outlier.

[0077] In this embodiment, performing data anomaly processing on the transaction flow homogeneous data according to all data anomaly scores is to abnormally delete all data points in the transaction flow homogeneous data whose data anomaly scores are greater than a set threshold.

[0078] In specific implementation, a threshold can be set based on domain knowledge and historical data. If the data anomaly score of a data point is greater than the set threshold, the data point is considered to be an abnormal data point and is removed. The above method can complete data anomaly processing for homogeneous data in the transaction flow.

[0079] It should be noted that by calculating the data anomaly score for each data point, the degree of anomaly of the data point can be measured more accurately. The platform can quickly locate potential abnormal data in large-scale transaction flow data without the need for manual inspection or screening through traditional rule definitions.

[0080] It can be seen that in the present application, firstly, the redundancy and noise in the data can be significantly reduced through the preprocessing step, the integrity and consistency of the data can be improved, and the data can be eliminated based on the data abnormal value, so as to reduce the scale of data to be processed and improve the processing efficiency, which can help to effectively deal with the high frequency, dynamic and complex characteristics of financial transactions and realize efficient processing of abnormal data; then, by evaluating the encoding difference loss of each data point, it can be effectively identified which data points deviate greatly from the reconstruction results of the model, and possible abnormal data can be more accurately identified, and the autoencoder has good automation characteristics when processing large-scale financial transaction flow data; finally, by calculating the data anomaly score of each data point, the degree of abnormality of the data point can be measured more accurately, and the platform can quickly locate potential abnormal data in large-scale transaction flow data, thereby improving the efficiency of data anomaly processing.

[0081] To sum up, the technical solution adopted in this application can efficiently identify abnormal data in digital economy monitoring to improve the efficiency of data anomaly processing.

[0082] Embodiment 2: This application provides a large-scale data processing platform for digital economy monitoring. Figure 4 As shown, this figure is a schematic diagram of a processing platform according to this embodiment of the present application, and the processing platform includes:

[0083] The data monitoring module 100 is used to monitor the financial transaction process in real time and obtain financial transaction flow data;

[0084] A preprocessing module 200 is used to preprocess the financial transaction flow data, and then perform heterogeneous screening on the preprocessed financial transaction flow data to obtain transaction flow homogeneous data in the financial transaction process;

[0085] The self-encoding module 300 is used to perform self-encoding on each data point in the transaction flow homogeneous data, thereby obtaining the encoding difference loss of each data point in the self-encoding process, and determining the self-encoding cost of the transaction flow homogeneous data in the self-encoding process through all the encoding difference losses;

[0086] The exception processing module 400 is used to determine the data anomaly score of each data point in the transaction flow homogeneous data using the self-encoding cost and the corresponding encoding difference loss, and perform data anomaly processing on the transaction flow homogeneous data according to all the data anomaly scores.

[0087] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0088] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, the storage medium including a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically-erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0089] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

Claims

1. A large-scale data processing method for digital economy monitoring, characterized in that: The processing method comprises the following steps: Monitor the financial transaction process in real time and obtain financial transaction flow data; Preprocessing the financial transaction flow data, and then performing heterogeneous screening on the preprocessed financial transaction flow data to obtain homogeneous transaction flow data in the financial transaction process; Performing self-encoding on each data point in the homogeneous transaction flow data, thereby obtaining the encoding difference loss of each data point in the self-encoding process, and determining the self-encoding cost of the homogeneous transaction flow data in the self-encoding process through all the encoding difference losses; Determine the data anomaly score of each data point in the transaction flow homogeneous data using the autoencoding cost and the corresponding encoding difference loss, and perform data anomaly processing on the transaction flow homogeneous data according to all the data anomaly scores; The self-encoding of each data point in the homogeneous transaction flow data to obtain the encoding difference loss of each data point in the self-encoding process specifically includes: Obtaining preset encoding weights and decoding weights; For each data point in the transaction flow homogeneous data, encoding the data point according to the encoding weight to obtain a potential feature representation of the data point; Decoding the latent feature representation according to the decoding weight to obtain a reconstructed latent feature representation; Determine the encoding difference loss of the data point in the self-encoding process by reconstructing the latent feature representation, and then obtain the encoding difference loss of each data point in the self-encoding process, wherein the encoding difference loss represents the difference loss generated in the process of self-encoding the latent feature representation of the data point, and the Euclidean norm of the difference between the data point and the reconstructed latent feature representation is used as the encoding difference loss of the data point in the self-encoding process; The self-encoding cost of the homogeneous transaction flow data in the self-encoding process is determined by all encoding difference losses, specifically including: Obtaining the weight corresponding to each data point in the transaction flow homogeneous data; Determine the self-encoding cost of the homogeneous transaction flow data in the self-encoding process according to the weight corresponding to each data point and all encoding difference losses, wherein the self-encoding cost refers to the overall difference cost generated by the homogeneous transaction flow data in the self-encoding process, and multiply the weight corresponding to each data point and the encoding difference loss corresponding to the data point, and take the sum of all the results as the self-encoding cost of the homogeneous transaction flow data in the self-encoding process; The method of using the autoencoding cost and the corresponding encoding difference loss to determine the data anomaly score of each data point in the transaction flow homogeneous data specifically includes: For each data point in the transaction flow homogeneous data, the ratio of the encoding difference loss corresponding to the data point to the self-encoding cost is used as the data anomaly score of the data point, thereby obtaining the data anomaly score of each data point in the transaction flow homogeneous data, wherein the data anomaly score is used to indicate the degree of abnormality of the data point.

2. A large-scale data processing method for digital economy monitoring according to claim 1, characterized in that: The financial transaction flow data refers to a data flow consisting of detailed information of each financial transaction in the financial market.

3. A large-scale data processing method for digital economy monitoring according to claim 1, characterized in that: Preprocessing the financial transaction flow data includes removing duplicate data, processing missing data and standardizing data formats.

4. A large-scale data processing method for digital economy monitoring according to claim 1, characterized in that: The pre-processed financial transaction flow data is subjected to heterogeneous screening to obtain the transaction flow homogeneous data in the financial transaction process, including: Determining data anomaly values ​​of preprocessed financial transaction stream data; The data points in the preprocessed financial transaction flow data are screened for heterogeneity according to the data heterogeneity value, thereby obtaining transaction flow homogeneity data in the financial transaction process.

5. A large-scale data processing method for digital economy monitoring as claimed in claim 4, characterized in that: Determining the data anomaly value of the preprocessed financial transaction flow data specifically includes: Determine the central aggregation value of the preprocessed financial transaction stream data; Determining a data deviation value of the preprocessed financial transaction flow data according to the central aggregation value; The data anomaly value of the preprocessed financial transaction flow data is determined according to the central aggregation value and the data deviation value.

6. A large-scale data processing method for digital economy monitoring according to claim 1, characterized in that: The data anomaly processing of the transaction flow homogeneous data according to all data anomaly scores is to delete abnormally all data points in the transaction flow homogeneous data whose data anomaly scores are greater than a set threshold.

7. A large-scale data processing platform for digital economy monitoring, used to execute a large-scale data processing method for digital economy monitoring as claimed in any one of claims 1 to 6, characterized in that: The processing platform comprises: The data monitoring module is used to monitor the financial transaction process in real time and obtain financial transaction flow data; A preprocessing module, used to preprocess the financial transaction flow data, and then perform heterogeneous screening on the preprocessed financial transaction flow data to obtain transaction flow homogeneous data in the financial transaction process; A self-encoding module, used for self-encoding each data point in the transaction flow homogeneous data, thereby obtaining the encoding difference loss of each data point in the self-encoding process, and determining the self-encoding cost of the transaction flow homogeneous data in the self-encoding process through all the encoding difference losses; The exception processing module is used to determine the data anomaly score of each data point in the transaction flow homogeneous data using the self-encoding cost and the corresponding encoding difference loss, and perform data anomaly processing on the transaction flow homogeneous data according to all the data anomaly scores.

Images