Large-scale dynamic data exception identification and early warning method and system oriented to supervision field

Abstract

The invention relates to a large-scale dynamic data exception identification and early warning method and system oriented to the supervision field, and belongs to the technical field of big data supervision. The method comprises the following steps: integrating and preprocessing data through a large-scale data integration platform; carrying out feature extraction on the preprocessed data model; identifying the abnormal data through the abnormal detection model; setting an early warning rule and an early warning threshold value in the anomaly detection model, and performing real-time identification and early warning on the abnormal data; and processing the abnormal early warning data according to the response strategy. The system comprises a large-scale data integration platform and an anomaly detection and early warning module, and is used for executing the method. By introducing an advanced machine learning algorithm and a data analysis technology, the accuracy and timeliness of anomaly recognition are improved, the supervision cost is reduced, the supervision efficiency is improved, and more comprehensive, accurate and timely supervision support is provided for supervision departments.

Description

Technical Field

[0001] The present invention belongs to the technical field of big data supervision, and relates to a method and system for identifying and warning of anomalies in large-scale dynamic data for the supervision field. Background Art

[0002] With the continuous improvement of the informatization level of enterprises, the amount of data shows an explosive growth trend. Not only does the quantity increase sharply, but the data types also become increasingly diverse, covering various forms such as structured data, semi-structured data, and unstructured data. The emergence of this data flood has brought unprecedented opportunities to the operation and management of enterprises, while also posing severe challenges to traditional supervision means, especially in the field of financial supervision.

[0003] Financial supervision, as one of the core links of enterprise management, its accuracy and efficiency are directly related to the capital safety, stable operation, and scientific decision-making of enterprises. However, traditional financial supervision means are unable to cope when faced with massive and polymorphic data. On the one hand, traditional supervision systems often rely on fixed rules and models, lacking sufficient flexibility and adaptability, and are difficult to respond to the rapidly changing market environment and business requirements. When the enterprise's business model is adjusted, new products are launched, or market policies change, traditional supervision systems often require a long time to update rules and models and cannot effectively capture new risk points in a timely manner.

[0004] On the other hand, many problems have emerged in the practical application of existing anomaly detection technologies. Due to the diversity and complexity of data types, traditional anomaly detection algorithms often have difficulty accurately identifying real abnormal behaviors, and the false alarm rate remains high. This not only increases the workload of financial supervision personnel but also may cause real risks to be ignored, bringing potential losses to the enterprise. In addition, traditional anomaly detection technologies are usually applicable to static data or small-scale data sets and are unable to meet the real-time detection requirements for large-scale dynamic data. In financial supervision scenarios with high real-time requirements, such as transaction monitoring and capital flow analysis, traditional anomaly detection technologies often cannot meet the dual requirements of real-time and accuracy.

[0005] To address these challenges, enterprises need to explore new financial supervision means and technologies. On the one hand, advanced technologies such as machine learning and artificial intelligence can be introduced to enhance the adaptive ability and intelligent level of the supervision system. By training models to automatically learn and identify normal business models and abnormal behaviors, human intervention can be reduced, and the accuracy and efficiency of supervision can be improved. On the other hand, real-time anomaly detection technologies applicable to large-scale dynamic data need to be developed to meet the real-time requirements of enterprise financial supervision. This includes aspects such as optimizing algorithm performance, improving data processing speed, and enhancing the scalability and stability of the system.

[0006] In summary, with the rapid increase in enterprise data volume and the diversification of data types, traditional regulatory means face huge challenges in the field of financial supervision. To address these challenges, enterprises need to continuously innovate and explore new regulatory means and technologies to improve the accuracy and efficiency of financial supervision and ensure the safety of enterprise funds and the soundness of operations. Summary of the Invention

[0007] In view of this, the purpose of the present invention is to provide a method and system for identifying and warning of anomalies in large-scale dynamic data for the regulatory field. By introducing advanced machine learning algorithms and data analysis techniques, the accuracy and timeliness of anomaly identification are improved, the regulatory cost is reduced, the regulatory efficiency is enhanced, and more comprehensive, accurate, and timely regulatory support is provided for regulatory authorities.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] On the one hand, a method for identifying and warning of anomalies in large-scale dynamic data for the regulatory field is provided. The method includes the following steps:

[0010] S1. Integrate and preprocess the data through a large-scale data integration platform;

[0011] S2. Extract features from the preprocessed data model;

[0012] S3. Identify abnormal data through an anomaly detection model;

[0013] S4. Set warning rules and warning thresholds in the anomaly detection model to identify and warn of abnormal data in real time;

[0014] S5. Dispose of the abnormal warning data according to the response strategy.

[0015] Further, in step S1, a large-scale data integration platform is constructed, which integrates multiple data source interfaces and captures and integrates key data in the corresponding regulatory field through a built-in regulatory data model. The data model includes an enterprise operation status model, an equity structure model, a financial condition model, an operation activity model, and a judicial record model;

[0016] Preprocess the captured and integrated data. Among them, the preprocessing includes data cleaning and deduplication, and data standardization processing. In the data cleaning process, data cleaning techniques are used to remove noise and redundancy in the data. The data cleaning techniques include regular expression matching, data filling, and outlier processing methods. Data deduplication uses uniqueness verification and algorithms. Data standardization processing refers to uniformly converting and standardizing data from different sources and in different formats to form a structured high-quality data foundation.

[0017] Further, in each model, the enterprise operation status model obtains registration, operation, and qualification information from the enterprise internal system, industrial and commercial departments, and industry associations; during data preprocessing, regular expressions are used to correct the format, fill in missing key information, and process outliers according to logical rules;

[0018] The equity structure model obtains shareholder and equity change information from the stock exchange and industrial and commercial registration archives; during data preprocessing, shareholder identity information is encrypted, duplicate records are removed, and data from different data sources are compared to ensure consistency;

[0019] The financial condition model obtains asset-liability, profit, and cash flow data from the enterprise financial system, banks, and tax platforms; during data preprocessing, financial indicator relationships are used to verify data, missing values are filled, and outliers are processed according to the fluctuation range;

[0020] The business activities model obtains sales revenue, market share, and customer data with the help of the enterprise sales system, market research institutions, and e-commerce platforms; during data preprocessing, duplicate sales revenue data are removed, the calculation basis of the market share is updated regularly, and abnormal customer activities are mined;

[0021] The judicial record model interfaces with the court and credit platforms to obtain litigation-related data; during data preprocessing, natural language processing technology is used to structurally extract data, case information is associated and integrated, and the trend of the number of cases is analyzed to review and update the data.

[0022] Further, in step S2, first, based on business rules and domain knowledge, combined with the actual needs of enterprise operation, key feature indicators that can comprehensively reflect the health status of the enterprise are designed and extracted. The key feature indicators include financial indicators, equity structure indicators, business activities indicators, and judicial record indicators. Among them, the financial indicators include asset-liability ratio, profit margin, and cash flow status; the equity structure indicators include shareholder shareholding ratio and equity concentration; the business activities indicators include sales revenue and market share; the judicial record indicators include the number of litigations and judgment results;

[0023] Then, a multi-dimensional feature vector is constructed according to the extracted key feature indicators. The extracted key feature indicators are combined and encoded to form a multi-dimensional feature vector. The formation process of the multi-dimensional feature vector is as follows:

[0024] Different standardization methods are used for different types of data. The data types include ratio-type data, numerical-type data, and frequency-type data. For ratio-type data, the maximum-minimum normalization method is used to normalize it to the [0,1] interval; for numerical-type data, the Z-score standardization method is used for processing; for frequency-type data, frequency binning is performed first and then normalization is carried out;

[0025] The judgment result is processed by one-hot encoding;

[0026] Arrange the standardized indicators in the order of financial, equity structure, business activities, and judicial record indicators to form a multi-dimensional feature vector.

[0027] Furthermore, in step S3, first establish the mapping relationship between the data model and the anomaly recognition rules. Among them, the model-rule mapping relationship is established by combining automated mapping and manual fine-tuning. Automated mapping uses model-driven technology to automatically parse the attribute information of the data model and match it with the preset anomaly recognition rules; manual fine-tuning refers to manually adjusting and optimizing the mapping relationship.

[0028] Then establish an anomaly detection model, and use machine learning algorithms to train and learn the multi-dimensional feature vector; among them, the anomaly detection model structure adopts a neural network deep learning architecture, which includes an input layer, multiple hidden layers, and an output layer.

[0029] The input layer inputs the multi-dimensional feature vector.

[0030] The hidden layer receives data from the input layer. The hidden layer is a multi-layer perceptron structure, and the neurons are connected by weights and activated using the ReLU function. The data features are mined through linear transformation and non-linear activation.

[0031] The output layer is a single node, and the value between 0 and 1 output represents the anomaly probability.

[0032] During training, use the backpropagation algorithm to calculate the gradient and update the weights in reverse according to the error between the output and the true label.

[0033] Furthermore, in step S4, first preset the warning rules and thresholds according to the actual situation of the enterprise and regulatory requirements. Among them,

[0034] For the financial indicator of asset-liability ratio, set the upper threshold of the asset-liability ratio with reference to the industry average and the enterprise's historical data; for the profit margin, set the lower threshold in combination with the enterprise's profit trend and industry standards; for the cash flow situation, set the lower threshold of the cash flow based on the enterprise's cash flow history and business plan.

[0035] For the equity structure indicator of the shareholder shareholding ratio, set the threshold of the change range of the shareholding ratio according to the equity dispersion; for the equity concentration, measure it with the HHI index and set the threshold of the change range of the HHI index.

[0036] For the business activity indicator of sales revenue, set the decline ratio threshold according to seasonality and growth trend; for the market share, set the continuous decline threshold according to the decline in two consecutive quarters.

[0037] For the judicial record indicator of the number of lawsuits, set the upper threshold of the average annual number of lawsuits based on the enterprise and industry history, and set the upper threshold of the loss of losing lawsuits.

[0038] Compare and evaluate the results output by the anomaly detection model with the preset warning rules and thresholds in real time to identify potential abnormal behaviors or patterns;

[0039] Once an abnormal behavior or pattern is detected, immediately trigger the warning mechanism to generate a warning report, which includes abnormal information, estimated risk points, and recommended handling measures.

[0040] Furthermore, in step S5, first establish a response strategy library according to different types of abnormal behaviors and potential risks. The response strategy library contains preset handling measures for different abnormal types and severities. The preset handling measures include further investigation, risk assessment, and notification to relevant enterprises;

[0041] Among them, according to the content of the warning report, automatically or semi-automatically trigger the corresponding response strategy; track and feedback on the triggered response strategy. At the same time, make necessary adjustments and optimizations to the response strategy according to the handling results and actual situations.

[0042] On the other hand, a system for implementing the aforementioned large-scale dynamic data anomaly identification and warning method for the regulatory field is also proposed. The system includes a large-scale data integration platform and an anomaly detection and warning module. Among them,

[0043] The large-scale data integration platform incorporates enterprise operation status models, equity structure models, financial status models, business activity models, and judicial record models. It captures and integrates key data in the corresponding regulatory fields through the built-in regulatory data models, and then constructs multi-dimensional feature vectors based on the extracted key feature indicators;

[0044] The anomaly detection and warning module internally includes a mapping relationship establishment model, an anomaly detection model, and a warning strategy model. The mapping relationship establishment model establishes the mapping relationship between the model and the rules in a combination of automated mapping and manual fine-tuning. The anomaly detection model uses machine learning algorithms to train and learn the multi-dimensional feature vectors. The warning strategy model presets warning rules and thresholds according to the actual situation of the enterprise and regulatory requirements, and compares and evaluates the results output by the anomaly detection model with the preset warning rules and thresholds in real time to identify potential abnormal behaviors or patterns;

[0045] Once an abnormal behavior or pattern is detected, immediately trigger the warning mechanism to generate a warning report, which includes abnormal information, estimated risk points, and recommended handling measures.

[0046] Furthermore, the optimization and iteration method for this system is:

[0047] Continuously monitor the model performance. By continuously monitoring the performance and output results of the anomaly detection model, promptly discover the problems and deficiencies existing in the model. At the same time, collect user feedback and opinions to understand the effect of the model in actual applications and user requirements.

[0048] Adaptive adjustment and optimization. According to the monitoring results and user feedback, perform adaptive adjustment and optimization on the anomaly detection model, including adjusting model parameters, improving algorithms, and adding feature indicators.

[0049] Update and improve system functions: According to the changes in regulatory policies and market demands, continuously update and improve system functions, including adding new data source interfaces, optimizing data processing processes, and enhancing the sensitivity of the early warning mechanism.

[0050] The beneficial effects of the present invention are as follows:

[0051] By constructing a large-scale data integration platform, the present invention can collect and integrate multi-source data in real-time or regularly, including enterprise operation status, equity structure, financial status, etc., to ensure the integrity and timeliness of the data. Precise feature extraction and modeling: Based on business rules and domain knowledge, the present invention designs key feature indicators that comprehensively reflect the health status of enterprises and constructs multi-dimensional feature vectors. These feature vectors can accurately describe the operation status and potential risks of enterprises, providing high-quality inputs for the anomaly detection model.

[0052] The present invention uses machine learning algorithms to construct an anomaly detection model, which can analyze large-scale dynamic data in real-time and accurately identify potential abnormal behaviors or patterns. At the same time, by presetting warning rules and thresholds, the system can trigger the early warning mechanism in real-time and promptly notify the regulatory authorities to take corresponding measures.

[0053] Generally speaking, by introducing advanced machine learning algorithms and data analysis technologies, the present invention improves the accuracy and timeliness of anomaly identification, reduces regulatory costs, enhances regulatory efficiency, and provides more comprehensive, accurate, and timely regulatory support for regulatory authorities. Moreover, the present invention can improve the accuracy and timeliness of anomaly identification, reduce the false alarm rate and missed alarm rate, and enhance regulatory efficiency; the automated processing process reduces manual intervention, reduces regulatory costs, and enhances regulatory effectiveness. The strong adaptive ability can effectively cope with market changes and new risk challenges.

[0054] Other advantages, objectives, and features of the present invention will be described to some extent in the subsequent specification, and to some extent, will be obvious to those skilled in the art based on the study of the following text, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be described in detail and preferably below in conjunction with the accompanying drawings, where:

[0056] Figure 1 It is a schematic diagram of the overall process of the method for identifying and warning of abnormal large-scale dynamic data in the field of supervision according to the present invention. Specific embodiments

[0057] The following uses specific specific examples to illustrate the implementation manners of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific implementation manners. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the drawings provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner. Without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0058] Among them, the drawings are only for illustrative purposes, showing only schematic diagrams, not physical diagrams, and should not be construed as a limitation to the present invention; in order to better illustrate the embodiments of the present invention, some components in the drawings will be omitted, enlarged, or reduced, which do not represent the dimensions of actual products; for those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0059] In the drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components; in the description of the present invention, it should be understood that if there are terms such as "upper", "lower", "left", "right", "front", "rear", etc. indicating the orientation or positional relationship, they are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, the terms describing the positional relationship in the drawings are only for illustrative purposes and should not be construed as a limitation to the present invention. For those of ordinary skill in the art, the specific meanings of the above terms can be understood according to specific circumstances.

[0060] Please refer to Figure 1 , which is a method and system for identifying and warning of abnormal large-scale dynamic data in the field of supervision.

[0061] Embodiment 1

[0062] This embodiment provides a method and system for identifying and warning of abnormal large-scale dynamic data in the field of supervision. As Figure 1 shown, it includes the following steps:

[0063] Step S1: First, integrate and preprocess the data through a large-scale data integration platform:

[0064] Build a large-scale data integration platform: The platform integrates multiple data source interfaces and can collect multi-source data such as business status, equity structure, financial status (including detailed financial indicators such as cash flow analysis and interest-bearing debt analysis), business activity records, and judicial records in real time or regularly. Ensure the integrity and timeliness of data through efficient data transmission and storage mechanisms. In the large-scale data integration platform, define regulatory data models, including business status models, equity structure models, financial status models (covering cash flow analysis and interest-bearing debt analysis), business activity models, and judicial record models. These models are used to comprehensively capture and integrate key data in the regulatory field.

[0065] The integrated data is preprocessed, which includes:

[0066] Data cleaning and deduplication: For the collected raw data, we use advanced data cleaning technologies, such as regular expression matching, data filling, and outlier processing, to eliminate noise and redundancy in the data. At the same time, we use uniqueness verification and algorithms to deduplication data and ensure data accuracy and consistency.

[0067] Data standardization: Data from different sources and formats are uniformly converted and standardized to form a structured, high-quality data foundation. This includes data format conversion, data type unification, and data granularity adjustment, which facilitates subsequent feature extraction and modeling.

[0068] More specifically, the enterprise operation status model obtains registration, operation and qualification information from the enterprise's internal system, industrial and commercial departments and industry associations. During processing, regular expressions are used to correct the format, fill in missing key information, and handle abnormal values ​​according to logical rules.

[0069] The equity structure model obtains shareholder and equity change information from the stock exchange and industrial and commercial registration files. During processing, the shareholder identity information is encrypted, duplicate records are removed, and data from different data sources are compared to ensure consistency.

[0070] The financial status model obtains asset and liability, profit and cash flow data from the enterprise financial system, bank and tax platform. It uses the relationship between financial indicators to verify the data, fill in missing values, and handle outliers according to the fluctuation range.

[0071] The business activity model uses the enterprise sales system, market research institutions and e-commerce platforms to obtain sales revenue, market share and customer data. When processing, it removes duplicate sales revenue data, regularly updates the basis for market share calculations, and mines abnormal customer activities.

[0072] The judicial record model interfaces with the court and the credit platform to obtain litigation-related data. It uses natural language processing technology to structurally extract data, correlates and integrates case information, and analyzes the trend of the number of cases to review and update the data.

[0073] S2. Extract features from the preprocessed data model:

[0074] Design key feature indicators: Based on business rules and domain knowledge, combined with the actual needs of enterprise operations, design and extract key feature indicators that can comprehensively reflect the health status of the enterprise. These indicators include, but are not limited to, financial indicators (such as asset-liability ratio, profit margin, cash flow status, etc.), equity structure indicators (such as shareholder shareholding ratio, equity concentration, etc.), business activity indicators (such as sales revenue, market share, etc.), and judicial record indicators (such as number of lawsuits, judgment results, etc.).

[0075] Construct multi-dimensional feature vectors: Combine and encode the extracted key feature indicators to form multi-dimensional feature vectors. These feature vectors can comprehensively and accurately describe the operation status and potential risks of the enterprise. Among them, the process of obtaining multi-dimensional feature vectors is as follows:

[0076] Data standardization processing: First, standardize each indicator. For ratio data such as asset-liability ratio and shareholder shareholding ratio, normalize them to the interval [0, 1]; for numerical data such as cash flow status and sales revenue, use the Z-score standardization method. For the number of lawsuits, normalize it after binning by frequency; use one-hot encoding for judgment results.

[0077] Combine into feature vectors: Arrange the standardized indicators of various types in the order of financial, equity structure, business activity, and judicial record indicators. For example, first place financial indicators such as asset-liability ratio and profit margin, and then successively place equity structure, business activity, and judicial record indicators, so as to form multi-dimensional feature vectors for subsequent analysis by the anomaly detection model.

[0078] S3. Identify abnormal data through the anomaly detection model:

[0079] Establish the mapping relationship between the data model and the anomaly identification rules. Through model-rule mapping, associate the elements in the regulatory data model with the preset anomaly identification rules so that the system can automatically detect abnormal patterns in the data. This includes defining which features or indicators in the data model will be used for anomaly detection and how these features or indicators match the anomaly rules.

[0080] When establishing the mapping relationship between the data model and the anomaly recognition rules, an approach combining automated mapping and manual fine-tuning can be adopted. Automated mapping mainly utilizes model-driven technology to automatically parse the attribute information of the data model and match it with the preset anomaly recognition rules. Manual fine-tuning allows users to adjust and optimize the mapping relationship when necessary to meet specific regulatory requirements.

[0081] Construct an anomaly detection model: Use machine learning algorithms such as deep learning and clustering analysis to train and learn from multi-dimensional feature vectors to construct an anomaly detection model. This model can achieve real-time analysis of large-scale dynamic data and accurately identify potential abnormal behaviors or patterns.

[0082] Specifically, a neural network deep learning architecture is adopted, including an input layer, multiple hidden layers, and an output layer.

[0083] Input: It is the multi-dimensional feature vector constructed in step S2, covering the indicator data of all aspects of enterprise operation.

[0084] Intermediate data processing process: The data in the input layer enters the hidden layer, and the hidden layer is a multi-layer perceptron structure. Neurons are connected by weights, activated using the ReLU function, and the data features are mined through linear transformation and non-linear activation. During training, the backpropagation algorithm is used to calculate the gradient and update the weights in reverse according to the error between the output and the true label to improve the accuracy of the model.

[0085] Output: The output layer is a single node, and the value between 0 and 1 output represents the anomaly probability. The model is optimized by learning from a large number of labeled samples, and in actual applications, the anomaly possibility of new data is judged based on this.

[0086] S4. Set warning rules and warning thresholds in the anomaly detection model to perform real-time identification and warning of abnormal data:

[0087] Preset warning rules and thresholds: According to the actual situation of the enterprise and regulatory requirements, preset reasonable warning rules and thresholds. These rules and thresholds can reflect abnormal situations and potential risks in enterprise operation. Specifically, the following warning rules and thresholds are included:

[0088] Financial indicators: The asset-liability ratio refers to the industry average and the enterprise's historical data. For example, if the industry average is 60% and the enterprise is usually 50%-60%, the warning upper limit can be set at 70%.

[0089] The profit margin is combined with the enterprise's profit trend and industry standards. If the enterprise's average in the past three years is 15% and the industry is 12%, the lower limit can be set at 10%.

[0090] The cash flow situation is based on the enterprise's cash flow history and business plan. For example, if the net cash flow from operating activities is less than 1 million yuan for three consecutive months, it is set as the warning threshold.

[0091] Equity structure indicators: Early warning for significant changes in shareholders' shareholding ratios (such as an increase or decrease exceeding 10%), and the threshold is adjusted according to the degree of equity dispersion. The equity concentration is measured by the HHI index, and a change exceeding 200 (adjusted according to industry competition) triggers an early warning.

[0092] Operating activity indicators: For sales revenue, based on seasonality and growth trends, such as a year-on-year decrease of 30% or a month-on-month decrease of 20% in the first quarter triggers an early warning. If the market share decreases by more than 5% for two consecutive quarters, it triggers an early warning.

[0093] Judicial record indicators: The number of lawsuits is based on the history of the enterprise and the industry. For example, if the average number of lawsuits per enterprise is 2 times per year and 3 times for the industry, an early warning can be set for more than 5 times in a year. A major loss in a lawsuit (the amount exceeds 5% of the enterprise's net assets) triggers an immediate early warning.

[0094] Real-time assessment and anomaly identification: Compare and evaluate the results output by the anomaly detection model with the preset early warning rules and thresholds in real time to identify potential abnormal behaviors or patterns. This process enables a quick response and timely handling of abnormal situations.

[0095] Trigger the early warning mechanism: Once an abnormal behavior or pattern is detected, immediately trigger the early warning mechanism to generate an early warning report. The early warning report includes detailed abnormal information, possible risk points, and recommended handling measures, etc.

[0096] Notify the regulatory authorities: Notify the regulatory authorities of the early warning report in a timely manner through various means such as text messages, emails, and system prompts. Ensure that the regulatory authorities can learn about the abnormal situation of the enterprise in a timely manner and take corresponding regulatory measures.

[0097] S5. Dispose of the abnormal early warning data according to the response strategy:

[0098] Establish a response strategy library: According to different types of abnormal behaviors and potential risks, establish a response strategy library. The response strategy library contains preset handling measures for different abnormal types and severity levels, such as further investigation, risk assessment, notifying relevant enterprises, etc.

[0099] Trigger the response strategy: Automatically or semi-automatically trigger the corresponding response strategy according to the content of the early warning report. This process enables a quick response and effective handling of abnormal situations.

[0100] Tracking and feedback: Track and feedback on the triggered response strategy to ensure the effectiveness and timeliness of the handling measures. At the same time, make necessary adjustments and optimizations to the response strategy according to the handling results and actual situations.

[0101] Example 2

[0102] Based on the aforementioned large-scale dynamic monitoring data anomaly identification and early warning method, this embodiment establishes a large-scale dynamic data anomaly identification and early warning system for the supervision field. Based on the data model, model-rule mapping, and early warning logic, through the anomaly identification engine and early warning trigger engine, the supervision data model is converted into an actual anomaly identification and early warning system. This engine can be used as an independent service or library, supporting integration and invocation through APIs or using programming languages. The system abstracts the anomaly identification and early warning process into model construction, model mapping, early warning logic design, and system construction through a model-driven method, reducing the workload of directly writing anomaly detection algorithms and early warning rules and simplifying the construction process of the entire system.

[0103] The system includes a large-scale data integration platform and an anomaly detection and early warning module. Among them, the large-scale data integration platform internally has an enterprise operation status model, an equity structure model, a financial condition model, an operation activity model, and a judicial record model, and several data models respectively capture and integrate key data in the supervision field; the anomaly detection and early warning module internally includes a mapping relationship establishment model, an anomaly detection model, and an early warning strategy model.

[0104] For system optimization and iteration, it is carried out in the following manner:

[0105] Continuously monitor the model performance: By continuously monitoring the performance and output results of the anomaly detection model, timely discover the problems and deficiencies of the model. At the same time, collect user feedback and opinions to understand the effects of the model in actual applications and user needs.

[0106] Adaptive adjustment and optimization: According to the monitoring results and user feedback, adaptively adjust and optimize the anomaly detection model. This includes adjusting model parameters, improving algorithms, adding feature indicators, etc., to improve the accuracy and stability of the model.

[0107] Update and improve system functions: According to changes in supervision policies and market demands, continuously update and improve system functions. This includes adding new data source interfaces, optimizing data processing processes, enhancing the sensitivity of the early warning mechanism, etc., to ensure that the system can continuously meet supervision requirements and market changes.

[0108] Combining the above methods and systems, the present invention gives a specific implementation process:

[0109] (1) Start

[0110] Start the large-scale dynamic data anomaly identification and early warning system for the supervision field, and prepare to receive and process large-scale dynamic data from the supervision field.

[0111] (2) Data input

[0112] The data is sourced from various storage media such as databases, sensor networks, and log systems in the regulatory domain. These raw data may include various types such as time series data, image data, and text data. The task in the data input stage is to import this data into the anomaly recognition and warning system in real-time or in batches.

[0113] (3) Data preprocessing

[0114] Perform processing such as cleaning, denoising, normalization, and format conversion on the raw data to improve the quality and consistency of the data. Since the data in the regulatory domain often has complexity and diversity, special attention needs to be paid to the integrity and accuracy of the data in the data preprocessing stage to ensure the smooth progress of subsequent steps.

[0115] (4) Feature extraction

[0116] Extract features useful for anomaly recognition from the preprocessed data. These features may include statistical features, time-domain features, frequency-domain features, and specific features based on business rules, etc. The purpose of feature extraction is to convert the raw data into a form more suitable for machine learning algorithms to process while retaining the key information in the data.

[0117] (5) Anomaly detection model construction

[0118] Construct an anomaly detection model for the regulatory domain. This model should be able to handle large-scale dynamic data and have the ability to detect anomalies in real-time or near real-time. The model can include multiple machine learning algorithms such as support vector machine (SVM), random forest, neural network, etc., as well as customized algorithms for specific requirements in the regulatory domain. In the training stage, the model will learn the distribution pattern of normal data and set reasonable thresholds to distinguish normal data from abnormal data.

[0119] (6) Anomaly recognition and warning

[0120] According to the prediction results of the anomaly detection model and the threshold setting mechanism, identify abnormal data in real-time or near real-time. Once abnormal data is identified, the system will immediately trigger the warning mechanism and send warning information to relevant personnel. The warning information may include key information such as the detailed information of the abnormal data, the type of anomaly, the occurrence time, and the possible scope of influence, etc., so that relevant personnel can take corresponding measures in a timely manner.

[0121] (7) Result output and feedback

[0122] Generate an anomaly detection report, which details the recognition results of abnormal data, warning information, and subsequent processing situations. At the same time, the system should provide a user feedback mechanism, allowing relevant personnel to verify and correct the anomaly detection results to improve the accuracy and reliability of the system.

[0123] (8) Continuous Monitoring and Optimization

[0124] Anomaly identification and early warning in the regulatory field is a continuous process. The system should be able to continuously receive new data and continuously optimize and update the model to adapt to the changes and developments in the regulatory field. At the same time, the system should also have the ability of self-monitoring and fault troubleshooting to ensure the stability and reliability of the system.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the present technical solution, and they should all be covered within the scope of the claims of the present invention.

Claims

1. A large-scale dynamic data anomaly identification and early warning method for the regulatory field, characterized by: The method comprises the following steps: S1. Integrate and pre-process data through a large-scale data integration platform; S2, extracting features from the preprocessed data model; S3, identifying abnormal data through anomaly detection model; S4. Set warning rules and warning thresholds in the anomaly detection model to identify and warn abnormal data in real time; S5. Handle abnormal warning data according to the response strategy.

2. According to claim 1, a large-scale dynamic data anomaly identification and early warning method for the regulatory field is characterized by: In step S1, a large-scale data integration platform is constructed, which integrates multiple data source interfaces and captures and integrates key data in the corresponding regulatory fields through built-in regulatory data models. The data models include enterprise operating status model, equity structure model, financial status model, business activity model and judicial record model; Preprocessing is performed on the captured and integrated data, including data cleaning and deduplication, and data standardization. During the data cleaning process, data cleaning techniques are used to remove noise and redundancy in the data, including regular expression matching, data filling, and outlier processing. Data deduplication uses uniqueness verification and algorithms. Data standardization refers to the unified conversion and standardization of data from different sources and in different formats to form a structured, high-quality data foundation.

3. The large-scale dynamic data anomaly identification and early warning method for the regulatory field according to claim 2 is characterized by: Among the various models, the enterprise operation status model obtains registration, operation and qualification information from the enterprise's internal system, industrial and commercial departments and industry associations; during data preprocessing, regular expressions are used to correct the format, fill in missing key information, and handle outliers according to logical rules; The equity structure model obtains information on shareholders and equity changes from stock exchanges and industrial and commercial registration files; During data preprocessing, encrypt shareholder identity information, remove duplicate records, and compare data from different data sources to ensure consistency; The financial status model obtains asset-liability, profit and cash flow data from the enterprise financial system, bank and tax platform; During data preprocessing, financial indicator relationships are used to verify data, fill missing values, and process outliers according to fluctuation ranges; The business activity model uses the enterprise sales system, market research institutions and e-commerce platforms to obtain sales revenue, market share and customer data; during data preprocessing, it removes duplicate sales revenue data, regularly updates the basis for market share calculations, and mines abnormal customer activities; The judicial record model connects to the court and credit platform to obtain litigation-related data; During data preprocessing, natural language processing technology is used to extract structured data, associate and integrate case information, and analyze case quantity trends to review and update data.

4. According to claim 2, a large-scale dynamic data anomaly identification and early warning method for the regulatory field is characterized by: In step S2, firstly, based on business rules and domain knowledge, combined with the actual needs of enterprise operations, key characteristic indicators that can comprehensively reflect the health status of the enterprise are designed and extracted. The key characteristic indicators include financial indicators, equity structure indicators, operating activity indicators, and judicial record indicators. Among them, financial indicators include asset-liability ratio, profit margin, and cash flow status; equity structure indicators include shareholder shareholding ratio and equity concentration; operating activity indicators include sales revenue and market share; and judicial record indicators include the number of lawsuits and judgment results; Then, a multi-dimensional feature vector is constructed based on the extracted key feature indicators, and the extracted key feature indicators are combined and encoded to form a multi-dimensional feature vector, wherein the formation process of the multi-dimensional feature vector is: Different types of data are standardized in different ways, including ratio data, numerical data, and frequency data. For ratio data, the maximum and minimum values ​​are normalized to the interval [0,1]. For numerical data, the Z-score standardization method is used. For frequency data, frequency binning is used before normalization. The judgment results are processed using one-hot encoding; Arrange the standardized indicators in the order of finance, equity structure, business activities, and judicial record indicators to form a multi-dimensional feature vector.

5. The large-scale dynamic data anomaly identification and early warning method for the regulatory field according to claim 4 is characterized by: In step S3, a mapping relationship between the data model and the anomaly identification rule is first established, wherein the model-rule mapping relationship is established by combining automated mapping and manual fine-tuning. Automated mapping uses model-driven technology to automatically parse data model attribute information and match it with preset anomaly identification rules; manual fine-tuning refers to manual adjustment and optimization of the mapping relationship; Then, an anomaly detection model is established, and a machine learning algorithm is used to train and learn the multi-dimensional feature vectors; the anomaly detection model structure adopts a neural network deep learning architecture, which includes an input layer, multiple hidden layers, and an output layer; The input layer inputs a multi-dimensional feature vector; The hidden layer receives data from the input layer. The hidden layer is a multi-layer perceptron structure. Neurons are connected by weights and activated using the ReLU function. Data features are mined through linear transformation and nonlinear activation. The output layer is a single node, and the output value between 0 and 1 represents the probability of abnormality; The back propagation algorithm is used during training to reversely calculate the gradient and update the weights based on the error between the output and the true label.

6. The large-scale dynamic data anomaly identification and early warning method for the regulatory field according to claim 5 is characterized by: In step S4, firstly, according to the actual situation and regulatory requirements of the enterprise, the warning rules and thresholds are preset, where: The asset-liability ratio of financial indicators is set with reference to the industry average and the company's historical data to set the upper threshold of the asset-load ratio; the profit margin is set with reference to the company's profit trend and industry standards to set the lower threshold; the cash flow status is set with reference to the company's cash flow history and business planning to set the lower threshold of the cash flow; The shareholder shareholding ratio of the equity structure indicator is adjusted according to the equity dispersion to set the shareholding ratio change range threshold; the equity concentration is measured by the HHI index, and the HHI index change range threshold is set; The sales revenue of the operating activity indicator sets a decline rate threshold based on seasonality and growth trends; the market share sets a continuous decline threshold based on the decline in two consecutive quarters; The number of lawsuits in the judicial record indicator sets an upper threshold for the average number of lawsuits per year based on the history of the enterprise and industry, and also sets an upper threshold for the loss of a lawsuit; Compare and evaluate the output of the anomaly detection model with the preset warning rules and thresholds in real time to identify potential abnormal behaviors or patterns; Once abnormal behavior or pattern is detected, the early warning mechanism is immediately triggered and an early warning report is generated. The warning report contains abnormal information, estimated risk points and recommended disposal measures.

7. The large-scale dynamic data anomaly identification and early warning method for the regulatory field according to claim 6 is characterized by: In step S5, firstly, a response strategy library is established according to different types of abnormal behaviors and potential risks. The response strategy library contains preset disposal measures for different abnormal types and severity levels. The preset disposal measures include further investigation, risk assessment, and notification of relevant enterprises. Among them, according to the content of the early warning report, the corresponding response strategy is triggered automatically or semi-automatically; the triggered response strategy is tracked and feedback is provided, and at the same time, necessary adjustments and optimizations are made to the response strategy based on the disposal results and actual conditions.

8. A system for executing the large-scale dynamic data anomaly identification and early warning method for the regulatory field as described in any one of claims 1 to 7, characterized in that: The system includes a large-scale data integration platform and anomaly detection and early warning modules, among which: The large-scale data integration platform has built-in enterprise operation status model, equity structure model, financial status model, business activity model and judicial record model. It captures and integrates key data in the corresponding regulatory field through the built-in regulatory data model, and then constructs a multi-dimensional feature vector based on the extracted key feature indicators; The anomaly detection and early warning module includes a mapping relationship establishment model, anomaly detection model, and early warning strategy model. The mapping relationship establishment model uses a combination of automated mapping and manual fine-tuning to establish the model-rule mapping relationship. The anomaly detection model uses a machine learning algorithm to train and learn multi-dimensional feature vectors. The early warning strategy model presets early warning rules and thresholds based on the actual situation and regulatory needs of the enterprise, and compares and evaluates the results output by the anomaly detection model with the preset early warning rules and thresholds in real time to identify potential abnormal behaviors or patterns. Once abnormal behavior or pattern is detected, the early warning mechanism is immediately triggered and an early warning report is generated. The warning report contains abnormal information, estimated risk points and recommended disposal measures.

9. The large-scale dynamic data anomaly identification and early warning system for regulatory fields according to claim 8 is characterized by: The optimization and iteration methods for the system are as follows: Continuously monitor model performance. By continuously monitoring the performance and output results of the anomaly detection model, timely discover problems and deficiencies in the model. At the same time, collect user feedback and opinions to understand the model's effectiveness in actual applications and user needs. Adaptive adjustment and optimization: Adaptive adjustment and optimization of the anomaly detection model based on monitoring results and user feedback, including adjusting model parameters, improving algorithms, and adding feature indicators; Update and improve system functions: According to changes in regulatory policies and market demands, continuously update and improve system functions, including adding new data source interfaces, optimizing data processing procedures, and improving the sensitivity of early warning mechanisms.

Images