Illegal behavior analysis method, device and equipment based on big data model prediction

Abstract

The invention relates to the field of data processing and business management, and particularly discloses an illegal behavior analysis method, device and equipment based on big data model prediction, and the method comprises the steps: obtaining original data from a plurality of preset data sources; extracting, converting and loading the original data, fusing the original data of different data sources to obtain fused data, and storing the fused data in a preset data warehouse; based on the fusion data in the data warehouse, generating a corresponding intelligent anomaly index according to a pre-trained machine learning model to perform illegal behavior analysis, and obtaining an anomaly detection result; the intelligent abnormal indexes comprise abnormal actual receiving information, frequent work order adjustment information and order abnormal information; and feedback information corresponding to the anomaly detection result is acquired, and real-time business demand information and environment change information are acquired, so that the machine learning model is optimized according to the feedback information, the business demand information and the environment change information.

Description

Technical Field

[0001] This application relates to the fields of data processing and business management, and particularly to a method, device, and equipment for analyzing violation behaviors based on big data model prediction. Background Art

[0002] In the digital operation of enterprises, the detection of violation behaviors in sales data (such as fraud and abnormal transactions) is a core link in business risk management. Traditional technical solutions usually have the following problems:

[0003] 1. Fragmented data processing: Existing methods rely on a single data source or simple data aggregation, making it difficult to integrate multi-dimensional data such as work orders, orders, sales amounts, and customer information. Data silos result in incomplete extraction of abnormal behavior features and low detection accuracy.

[0004] 2. Lack of real-time performance: Traditional ETL (Extract-Transform-Load) processes are mainly batch processing-based and cannot meet the requirements of real-time monitoring, resulting in a lag in response to high-frequency and immediate violation behaviors (such as frequent work order adjustments and abnormal actual receipts).

[0005] 3. Rigid anomaly detection mechanism: Dependence on predefined rules and fixed thresholds (such as manually setting amount thresholds and upper limits of modification times), lack of adaptability to changes in business models, difficulty in identifying new violation patterns, and the need for frequent manual rule adjustments, resulting in high operation and maintenance costs.

[0006] 4. Lack of model optimization ability: Failure to establish a feedback-driven dynamic optimization mechanism, inability to continuously improve the performance of the detection model using real-time business data and user feedback, resulting in a decline in long-term detection effects.

[0007] In the prior art, although there are anomaly detection solutions based on simple statistical analysis or single machine learning models, a complete closed loop of "data collection - integration - intelligent analysis - adaptive optimization" has generally not been formed. There are significant defects in key technical links such as multi-source data fusion architecture, combination of models and business rules, and dynamic threshold adjustment, making it difficult to meet the enterprise's requirements for refined and real-time risk control.

[0008] Therefore, there is an urgent need for a method to solve at least one of the above problems. Summary of the Invention

[0009] This application provides a method, device, and equipment for analyzing violation behaviors based on big data model prediction, which is used to solve the problem that although there are abnormal detection solutions based on simple statistical analysis or single machine learning models in the existing technology, a complete closed loop of "data collection - integration - intelligent analysis - adaptive optimization" has generally not been formed. In particular, there are significant defects in key technical links such as multi-source data fusion architecture, combination of models and business rules, and dynamic threshold adjustment, making it difficult to meet the enterprise's requirements for refined and real-time risk control.

[0010] In the first aspect, this application provides a method for analyzing violation behaviors based on big data model prediction, and the method includes:

[0011] Obtain raw data from multiple preset data sources; the preset data sources at least include work order information, order information, sales information, and customer information;

[0012] Extract, transform, and load the raw data, and fuse the raw data from different data sources to obtain fused data, and store the fused data in a preset data warehouse. The data warehouse adopts a star or snowflake architecture and is used to record the source, definition, and change history of the data;

[0013] Based on the fused data in the data warehouse, generate corresponding intelligent anomaly metrics according to a pre-trained machine learning model to perform violation behavior analysis and obtain anomaly detection results; the intelligent anomaly metrics include abnormal actual receipt information, frequent work order adjustment information, and order anomaly information;

[0014] Obtain feedback information corresponding to the anomaly detection results, and obtain real-time business requirement information and environmental change information, so as to optimize the machine learning model according to the feedback information, business requirement information, and environmental change information.

[0015] In some embodiments, the extracting, transforming, and loading the raw data includes: using a preset ETL tool to extract the raw data, and transforming the raw data according to preset business rules, and loading the processed raw data into the corresponding data processing process.

[0016] Exemplarily, the using a preset ETL tool to extract the raw data includes: performing cleaning operations on missing values, duplicate values, and abnormal values in the raw data, and eliminating format differences of the raw data from different data sources through data format normalization processing.

[0017] In some embodiments, the fusion of the raw data from different data sources to obtain the fusion data includes: using a preset big data processing framework to perform field mapping and association on the raw data from work order information, order information, sales information, and customer information based on a preset data standard; generating a comprehensive view covering multi-dimensional business information as the fusion data through data redundancy elimination, conflict value calibration, and semantic consistency verification.

[0018] In some embodiments, the storage of the fusion data in a preset data warehouse includes: storing the fusion data in a preset distributed storage system; constructing a data access interface through a preset SQL query engine, and implementing traceability management of the fusion data by recording the source, field definition, update time, and change history of the fusion data.

[0019] In some embodiments, before storing the fusion data in a preset data warehouse, the method further includes: obtaining business analysis requirements; selecting a star or snowflake data warehouse architecture according to the business analysis requirements, and constructing a real-time data transmission channel based on a preset message queue to configure an index and partition strategy according to the real-time data transmission channel and storage engine characteristics, forming a data warehouse that supports efficient query and real-time update.

[0020] In some embodiments, the machine learning model includes a supervised learning model, an unsupervised learning model, and a time series analysis model; based on the fusion data in the data warehouse, generating corresponding intelligent anomaly metrics according to a pre-trained machine learning model for analyzing violation behaviors and obtaining anomaly detection results, including: training the labeled historical anomaly data using a supervised learning model; identifying anomaly patterns in the unlabeled historical anomaly data through an unsupervised learning model, and combining the time series analysis model to perform trend prediction on time series data to generate intelligent anomaly metrics including an abnormal actual receipt amount threshold, a work order adjustment frequency threshold, and an order status deviation degree; matching the real-time data through the intelligent anomaly metrics and outputting the violation behavior detection result.

[0021] In some embodiments, the optimization of the machine learning model according to the feedback information, business requirement information, and environmental change information includes: performing real-time feature extraction on newly accessed business data through an online learning algorithm and dynamically updating the parameters of the machine learning model; establishing a user feedback interface to collect the manually labeled anomaly detection results as the feedback information, and performing backpropagation optimization on the model prediction error corresponding to the machine learning model; adaptively adjusting the anomaly detection threshold of the machine learning model according to the real-time obtained business requirement information and environmental change information to complete the optimization of the machine learning model.

[0022] Second aspect, the present application provides a violation behavior analysis device based on big data model prediction, which is applied to a computer device. The device includes:

[0023] A data acquisition unit, configured to acquire raw data from multiple preset data sources; the preset data sources at least include work order information, order information, sales information, and customer information;

[0024] A data extraction unit, configured to extract, transform, and load the raw data, fuse the raw data from different data sources, obtain fused data, and store the fused data in a preset data warehouse. The data warehouse adopts a star or snowflake architecture and is used to record the source, definition, and change history of the data;

[0025] An anomaly detection unit, configured to generate corresponding intelligent anomaly metrics for violation behavior analysis based on the fused data in the data warehouse according to a pre-trained machine learning model, and obtain an anomaly detection result; the intelligent anomaly metrics include abnormal actual receipt information, frequent work order adjustment information, and order anomaly information;

[0026] A model optimization unit, configured to obtain feedback information corresponding to the anomaly detection result, and obtain real-time business requirement information and environmental change information, so as to optimize the machine learning model according to the feedback information, business requirement information, and environmental change information.

[0027] Third aspect, the present application provides a computer device, which includes a memory and a processor;

[0028] The memory is used to store a computer program;

[0029] The processor is configured to execute the computer program and implement any one of the violation behavior analysis methods based on big data model prediction provided in the embodiments of the present application when executing the computer program.

[0030] The present application discloses a violation behavior analysis method, device, and equipment based on big data model prediction. The provided method collects raw data through preset data sources (work order, order, sales, customer information), stores it in a star / snowflake data warehouse after ETL processing and cross-source fusion, solves the data island problem, and constructs a full-scale business view; based on the fused data in the data warehouse, uses multiple types of machine learning models such as supervised learning, unsupervised learning, and time series analysis to generate a detection system including dynamic metrics such as abnormal actual receipt, frequent work order adjustment, and order anomaly, breaking through the limitations of traditional fixed rules; by obtaining feedback information, business requirements, and environmental change data in real time, dynamically adjusts the machine learning model parameters and anomaly detection thresholds, and realizes the synchronous evolution of the detection strategy and the actual business scenario.

[0031] By deeply integrating multi-source data fusion technologies (Spark / Flink frameworks), distributed data warehouse architectures (star / snowflake), and machine learning models (random forest, isolation forest, Prophet), an integrated solution of "data-model-application" is formed; through a feedback-driven online learning and threshold adjustment mechanism, the system can automatically adapt to changes in business rules and new violation patterns, significantly improving detection flexibility and long-term effectiveness; based on Kafka message queues and Flink stream processing technologies, real-time data collection and analysis are achieved to meet the immediate risk warning requirements in high-frequency trading scenarios. Through standardized ETL processing and cross-source data fusion, problems such as inconsistent formats and redundant values are eliminated, ensuring data integrity and consistency, and laying a foundation for accurate detection; the intelligent anomaly metric system with multi-model fusion can identify more than 100 violation behaviors and more than 30 anomaly patterns, covering complex scenarios that are difficult to detect by traditional rules (such as abnormal trends in time-series data and multi-dimensional correlation anomalies); the real-time data stream processing architecture reduces the anomaly detection latency to the second level, and combined with a multi-level warning mechanism (message, SMS, phone call), an immediate response to risk events is achieved; the adaptive optimization mechanism enables the model to continuously iterate through historical feedback and real-time data, avoiding the attenuation of detection effects caused by changes in business models and reducing the cost of manual intervention.

[0032] In summary, through the combination of technical architecture innovation and intelligent algorithms, the present invention effectively solves the deficiencies of traditional methods in data integration, detection flexibility, and real-time performance, and provides a more efficient and reliable solution for analyzing violation behaviors for enterprises. Brief Description of the Drawings

[0033] To more clearly illustrate the technical solutions of the embodiments of the present application, the accompanying drawings required for the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0034] Figure 1 It is a step schematic flow chart of a method for analyzing violation behaviors based on big data model prediction provided by an embodiment of the present application;

[0035] Figure 2 It is a schematic block diagram of a device for analyzing violation behaviors based on big data model prediction provided by an embodiment of the present application;

[0036] Figure 3 It is a schematic block diagram of the structure of a computer device provided by an embodiment of the present application. Detailed Embodiments

[0037] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0038] The flowcharts shown in the accompanying drawings are only illustrative examples, and do not necessarily include all contents and operations / steps, nor do they necessarily need to be executed in the described order. For example, some operations / steps can also be decomposed, combined or partially merged, so the actual execution order may change according to the actual situation.

[0039] It should be understood that the terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification of the present application and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include the plural forms.

[0040] It should also be understood that the term "and / or" used in the specification of the present application and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0041] Next, some embodiments of the present application will be described in detail in conjunction with the accompanying drawings. Without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0042] It should be noted that any data involved in the present application is obtained under the permission of the relevant users, complies with the relevant policy regulations, and will not infringe on the user's privacy.

[0043] In the digital operation of enterprises, the detection of irregularities in sales data (such as fraud and abnormal transactions) is the core link of business risk management. The traditional technical solutions usually have the following problems:

[0044] 1. Fragmented data processing: Existing methods rely on a single data source or simple data aggregation, making it difficult to integrate multi-dimensional data such as work orders, orders, sales amounts, and customer information. Data islands result in incomplete extraction of abnormal behavior features and low detection accuracy.

[0045] 2. Lack of real-time performance: The traditional ETL (Extract-Transform-Load) process is mainly batch processing and cannot meet the real-time monitoring requirements, resulting in a lag in response to high-frequency and immediate irregularities (such as frequent work order adjustments and abnormal actual receipts).

[0046] 3. Rigid anomaly detection mechanism: It relies on predefined rules and fixed thresholds (such as manually setting the amount threshold and the upper limit of modification times), lacks adaptability to changes in business models, is difficult to identify new types of violation patterns, and requires frequent manual rule adjustments, resulting in high operation and maintenance costs.

[0047] 4. Lack of model optimization ability: It fails to establish a feedback-driven dynamic optimization mechanism and cannot continuously improve the performance of the detection model by utilizing real-time business data and user feedback, leading to a decline in the long-term detection effect.

[0048] In the prior art, although there are anomaly detection solutions based on simple statistical analysis or single machine learning models, a complete closed-loop of "data collection - integration - intelligent analysis - adaptive optimization" has generally not been formed. In particular, there are significant defects in key technical aspects such as multi-source data fusion architecture, combination of models and business rules, and dynamic threshold adjustment, making it difficult to meet the enterprise's requirements for refined and real-time risk control.

[0049] Therefore, there is an urgent need for a method to solve at least one of the above problems.

[0050] Please refer to Figure 1 , Figure 1 which is a schematic flowchart of a method for analyzing violation behaviors based on big data model prediction provided by an embodiment of the present application. The method is applied to a computer device.

[0051] As Figure 1 shown, the specific steps of the method for analyzing violation behaviors based on big data model prediction include: Step S101 to Step S104.

[0052] S101. Obtain raw data from multiple preset data sources; the preset data sources at least include work order information, order information, sales information, and customer information.

[0053] Specifically, by collecting raw business data from multiple preset data sources inside and outside the enterprise, covering the core business dimensions required for violation behavior analysis, including but not limited to work order information (such as work order creation time, operation records, processing status), order information (such as order amount, commodity category, transaction time), sales information (such as sales amount, discount strategy, payment method), and customer information (such as customer portrait, historical transaction records, credit rating), it provides basic data support for subsequent data integration and analysis.

[0054] Data source adaptation, such as docking heterogeneous data sources through API interfaces (such as RESTful, SOAP), file transfer (CSV / Excel batch import), message queues (Kafka / RabbitMQ), etc., supports data access from relational databases (MySQL / Oracle), non-relational databases (MongoDB / Redis), and business systems (ERP / CRM);

[0055] Data collection strategies include using streaming data collection (Flink / Kafka consumers) for real-time business scenarios (such as high-frequency trading) and scheduled batch collection (Airflow scheduling tasks) for offline historical data to ensure that all data is obtained without omission; by synchronously recording metadata such as data source identifiers (such as data source name, interface version), collection timestamps, and data formats during the data collection phase, it provides a basis for subsequent data traceability and quality assessment.

[0056] By integrating cross-business data such as work orders, orders, sales, and customers, a full-link data view including transaction processes, user behaviors, and business operations is constructed to solve the problem of one-sided feature extraction caused by traditional single data sources, making the characteristics of illegal behaviors more complete (for example, combining work order modification records with order amount fluctuations to identify abnormal operations); standardized collection interfaces and strategies ensure the integrity and real-time nature of the original data, providing high-quality input for subsequent ETL processing and model analysis, and avoiding detection blind spots caused by data missing.

[0057] S102. Extract, transform, and load the original data, fuse the original data from different data sources to obtain fused data, and store the fused data in a preset data warehouse. The data warehouse adopts a star or snowflake architecture and is used to record the source, definition, and change history of the data.

[0058] Specifically, perform extraction (Extract), transformation (Transform), and load (Load) operations on the original data, eliminate data heterogeneity through data cleaning, format unification, cross-source association, etc., generate fused data covering multi-data source information, and store it in a data warehouse that supports a star or snowflake architecture. At the same time, record the data lineage and change history through metadata management.

[0059] ETL processing: Data extraction: Use ETL tools such as Apache Nifi and Talend to extract the original data from each data source, supporting incremental extraction (only obtaining updated data) and full extraction;

[0060] Data conversion includes: cleaning operations: identifying and processing missing values (interpolation filling / removal), duplicate values (deduplication), and outliers (Z-score detection and filtering); format unification: converting time formats (such as "2025 / 06 / 23" and "2025-06-23") and monetary units from different data sources to a unified standard;

[0061] Business rule conversion includes: deriving new fields based on violation detection requirements (such as "work order modification frequency = number of work order modifications / work order creation time interval"); data loading: loading processed data into a temporary storage layer (such as Hadoop HDFS) to provide intermediate data for subsequent fusion.

[0062] Data fusion uses the Spark / Flink big data framework to link data from different data sources based on business primary keys (such as order ID and customer ID) to build a wide table containing the work order-order-customer relationship. It also performs data redundancy elimination (deleting duplicate fields), conflict value calibration (correcting inconsistent data based on authoritative data sources), and semantic consistency verification (ensuring that field definitions for the same business concept are unified, such as "sales" refers to the amount including tax) to generate a fused dataset.

[0063] Data warehouse storage: Architecture selection: Choose a star architecture (simplified query, suitable for fast analysis) or a snowflake architecture (normalized design, suitable for complex dimensional analysis) based on the complexity of business analysis; Storage implementation: Store the integrated data in a distributed storage system (Hive / Amazon Redshift), and use a metadata management tool (Apache Atlas) to record information such as data sources, field definitions, and update logs to support data lineage tracing (such as locating the original data source corresponding to abnormal indicators).

[0064] Through cross-source data association and fusion, a comprehensive business data set is formed that includes work order operation tracks, order transaction details, and customer historical behaviors, enabling violation analysis to be upgraded from "single-dimensional judgment" to "multi-dimensional correlation analysis" (for example, combining customers' historical credit ratings with current order discount anomalies to identify false transactions); standardized cleaning and conversion processes eliminate data noise and format differences, and metadata management achieves data traceability, providing a reliable data foundation for subsequent model analysis and avoiding detection misjudgments due to data inconsistencies; star / snowflake architecture supports efficient multi-dimensional data analysis, and compared with traditional flat data storage, query performance is improved by 30%-50%, meeting the data access speed requirements of real-time detection.

[0065] S103. Based on the fused data in the data warehouse, corresponding intelligent anomaly indicators are generated according to the pre-trained machine learning model to perform violation analysis and obtain anomaly detection results; the intelligent anomaly indicators include abnormal actual collection information, frequent work order adjustment information and order anomaly information.

[0066] Specifically, based on the integrated data in the data warehouse, key features are extracted using pre-trained machine learning models (including supervised learning, unsupervised learning, and time series analysis models), and dynamically adjusted intelligent anomaly metrics (abnormal actual receipt information, frequent work order adjustment information, order anomaly information) are generated. Through metric matching and model inference, detection of violations is achieved, and detection results (such as abnormal transaction levels, violation risk scores) are output.

[0067] Model training and metric definition include: Supervised learning model: Use historically labeled violation data (such as manually marked abnormal work orders, false orders) to train decision tree and random forest models, and learn the feature combinations of known violation patterns (such as "the work order is modified ≥5 times within 30 minutes and the order amount fluctuates >20%"); Unsupervised learning model: Use K-means clustering and isolation forest algorithms to perform anomaly detection on unlabeled data, and identify outliers that do not conform to the normal business distribution (such as a customer's sudden high-frequency cross-regional transactions); Time series model: Use ARIMA and Prophet to model time series data such as sales and work order modification frequencies, predict the normal business trend, and define the deviation between real-time data and predicted values (such as the actual receipt amount being continuously 3 days lower than 80% of the predicted value) as an anomaly metric; Integrate the anomaly signals output by the three types of models to generate a comprehensive detection metric system including "abnormal actual receipt amount threshold", "work order adjustment frequency threshold", and "order status deviation degree".

[0068] Use the Flink stream processing engine to read the incremental integrated data in the data warehouse in real time, trigger anomaly detection tasks according to a preset time window (such as minute level); perform feature engineering on real-time data (such as calculating the number of modifications of the current work order within 24 hours, the deviation ratio of the order discount rate from the historical average), input the trained model for inference, and output detection results of violation behaviors (such as "high-risk work order operations", "abnormal order transactions").

[0069] The multi-model fusion architecture covers the detection of known violation patterns (supervised learning) and unknown anomaly patterns (unsupervised learning). Compared with traditional fixed rule methods, it can identify new types of violation behaviors (such as long-term violations disguised by abnormal time series data trends), and the detection coverage rate is improved; Based on the dynamic thresholds of time series models (such as automatically adjusting the sales anomaly threshold according to the peak and off-peak seasons of the business), it avoids the problems of "undetected in peak seasons and false alarms in off-peak seasons" caused by fixed thresholds; The intelligent anomaly metrics are combined with business scenarios (such as analyzing operation logs corresponding to "frequent work order adjustments" and associating transaction data corresponding to "order anomalies"), realizing the upgrade from "single data point detection" to "full-link monitoring of business processes". For example, operation vulnerabilities are identified through the time series association between work order modification records and order payment times.

[0070] S104. Obtain the feedback information corresponding to the abnormal detection result, and obtain the real-time business requirement information and environmental change information, so as to optimize the machine learning model according to the feedback information, business requirement information and environmental change information.

[0071] Specifically, by collecting the feedback information of the abnormal detection result (such as the misjudgment / missing judgment records of manual review), real-time business requirement information (such as new business rules, adjustment of key risk control points), and environmental change information (such as market policy changes, update of industry violation patterns), the machine learning model is optimized by combining data-driven and rule-driven methods to realize the dynamic iteration of the detection strategy.

[0072] By establishing a manual review interface to receive the annotation of the detection result by the operation personnel (such as "this abnormal order is actually a normal promotion activity"), the annotation data is added to the training set as a new sample, and the model parameters are updated in real time through an online learning algorithm (such as SGD Stochastic Gradient Descent); statistically analyze the historical detection results, calculate the false detection rate and missing detection rate of the model, and adjust the model feature weights or reconstruct the detection logic for high-frequency misjudgment scenarios (such as a certain type of normal business being continuously mislabeled). Connect to the real-time business rules of the enterprise risk control system (such as "the work order modification threshold for the newly launched business is adjusted to 10 times / hour"), and dynamically modify the calculation logic or threshold parameters of the abnormal indicators through the rule engine; monitor external information such as industry violation reports and policy documents, and automatically trigger incremental training of the model when a new type of violation means (such as using the vulnerability of a new payment method) is detected, and supplement the corresponding feature dimensions (such as adding the risk coefficient of the new payment channel).

[0073] Adopt the hot update technology to synchronize the optimized model parameters or rule configurations to the detection service without interrupting the real-time detection, so as to realize the continuous optimization of "learning while deploying"; establish a model performance evaluation system to verify the optimization effect (such as AUC-ROC value, F1 score) regularly (such as daily) to ensure the effectiveness of the optimization direction.

[0074] Through the feedback-driven model iteration, avoid the attenuation of the detection effect caused by the change of the business model in the traditional method, respond in real time to the adjustment of the enterprise risk control strategy (such as relaxing the discount abnormal threshold during the promotion period) and external environmental changes (such as new regulatory regulations increasing compliance detection indicators), so that the system can seamlessly integrate into the business process and reduce the manual intervention cost (the rule adjustment efficiency is improved from "manual hour level" to "system second level"); combine the dual optimization mechanism of data-driven and rule-driven to form a closed loop of "detection - feedback - optimization - re-detection", so that the system has the ability to independently adapt to new types of violation patterns, significantly reduce the dependence on manual experience, and is especially suitable for Internet business scenarios where violation means are rapidly iterated.

[0075] In some embodiments, the extraction, transformation, and loading of the original data include: using a preset ETL tool to extract the original data, transforming the original data according to preset business rules, and loading the processed original data into the corresponding data processing flow.

[0076] Tool selection includes using mature ETL tools such as Apache Nifi, Talend, and Informatica to access heterogeneous data sources by configuring data source connectors (such as JDBC connectors to connect to MySQL and Hive connectors to connect to data warehouses), supporting batch extraction (triggered by scheduled tasks) and incremental extraction (listening to data source change logs and only synchronizing newly added / modified data).

[0077] Exemplarily, the using of a preset ETL tool to extract the original data includes: performing cleaning operations on missing values, duplicate values, and outliers in the original data, and eliminating format differences in the original data from different data sources through data format normalization processing.

[0078] The extraction strategy includes using streaming extraction (real-time pulling of data through Kafka consumers) for data sources with high real-time requirements (such as order transaction systems), and batch extraction (executing full-scale synchronization tasks every day at midnight) for historical data storage systems (such as offline data lakes) to ensure efficient acquisition of different types of data as needed.

[0079] Data cleaning includes: Missing value processing: For key business fields (such as order amount, work order creation time), interpolation methods (mean / median filling) or rule filling (such as "not filled" marking) are used, and missing records are directly removed for non-key fields (such as customer comment information); Duplicate value processing: Deduplication is performed based on the business primary key (such as order ID, work order number), and completely duplicate records are quickly identified through hash algorithms. For records with partial field duplicates but business differences (such as two payment attempts for the same order), the latest valid version is retained; Outlier detection: The Z-score algorithm (data deviating more than 3 times the standard deviation from the mean is considered an outlier) or the IQR method (data outside the interquartile range is marked as an outlier) is used. For example, abnormal data such as negative sales amounts and work order modification times exceeding 5 times the daily average are identified, and invalid records are removed through manual review or rule filtering.

[0080] Format standardization includes: unifying data types: converting time formats from different data sources (such as "2025 / 06 / 23 14:30", "2025-06-23T14:30:00Z") to ISO 8601 standard timestamps, unifying the amount field to numeric type (with the unit standardized to RMB yuan), and unifying enumeration values (such as order status "completed", "finished") to standard business terms ("completed"); cross-source format mapping: establishing a data dictionary mapping table to record the conversion rules for each data source field to the target format (such as "registration time" in customer information corresponding to "first interaction time" in work order information, and achieving cross-source alignment through timestamp conversion functions).

[0081] Business rule conversion includes: deriving calculation fields: generating new metrics according to the needs of violation detection, such as "work order modification frequency = number of work order modifications on the same day / work order creation duration", "order discount rate = (order amount - total original price of goods) / total original price of goods", for input to subsequent anomaly detection models; logical conversion: performing business logic processing on the original data, such as converting "tax-inclusive sales amount" in sales information to "tax-exclusive sales amount" through a tax rate configuration table, and converting "risk level code (A / B / C)" in customer information to a numeric risk score (10 / 5 / 1).

[0082] Data loading includes loading the standardized data after cleaning and conversion into the temporary storage layer of the data processing process (such as the temporary directory of Hadoop HDFS, the staging table of a relational database), providing input data in a unified format for subsequent data fusion, and recording metadata such as data loading time, data volume, and error logs during the loading process for subsequent quality monitoring.

[0083] Through systematic cleaning rules (missing value filling, outlier filtering), the data integrity is improved to the above level, the duplicate data rate is decreased, ensuring that subsequent analysis is based on high-quality data and avoiding detection misjudgments caused by "dirty data"; format standardization eliminates the heterogeneity between data sources (such as inconsistent time formats, unit differences), enabling cross-domain data such as work orders and orders to be directly correlated for analysis (for example, after unifying the "transaction time" of different systems, the time sequence of work order operations and order payments can be accurately matched); by converting business rules, derivative metrics required for the model (such as discount rate, modification frequency) are generated in advance, reducing the data preprocessing workload in the subsequent modeling stage, improving the overall process efficiency, and ensuring that the conversion logic is traceable (metadata records the conversion rules).

[0084] In some embodiments, the fusion of the raw data from different data sources to obtain the fusion data includes: using a preset big data processing framework to perform field mapping and association on the raw data from work order information, order information, sales information, and customer information based on a preset data standard; generating a comprehensive view covering multi-dimensional business information as the fusion data through data redundancy elimination, conflict value calibration, and semantic consistency verification.

[0085] The big data framework application selects Spark (suitable for batch data processing) or Flink (suitable for real-time stream processing) as the underlying framework, uses distributed computing capabilities to process multi-source data on the TB scale, defines the data structure through the DataFrame / Dataset API, and supports parallel computing and distributed storage across data sources.

[0086] Field mapping and association include: unifying the data standard: formulating a global data dictionary, defining the unified caliber of the core fields in each business domain (such as the "customer ID" uses the same coding rule in work orders, orders, and customer information), and establishing a field mapping table (such as mapping the "user identification" in work orders to the "customer ID" in customer information); cross-source association: integrating multi-source data based on the business primary key (such as the "order ID" in order information and the "associated order ID" in work order information) through join operations (inner join, outer join). For example, associating order information with customer information through the "customer ID" to generate an order details table containing the customer portrait; for data without a direct primary key association (such as sales information and work order information), perform fuzzy association through a time window (such as work order operation records within ±1 hour of the order creation time).

[0087] Data redundancy elimination and conflict calibration include: redundancy elimination: identifying duplicate fields in cross-source data (such as both order information and sales information contain "order amount"), retaining the fields from the most authoritative data source (such as taking the "order amount" in the order system as the standard), and deleting redundant fields; unifying different expressions of the same business concept (such as "creation time" and "generation time") to "creation time" and retaining a single field;

[0088] Conflict value calibration includes: establishing a data source priority rule (such as the data in the core business system has a higher priority than the log system). When there is data inconsistency in the same field (such as the order amount recorded in the work order system differs from the order system by more than 5%), trigger a manual review process or automatically correct it based on the high-priority data source. For example, the "credit rating" in customer information is overwritten by the data in the CRM system for conflict values in other systems.

[0089] Semantic consistency verification verifies data semantics through predefined business rules. For example, "When the order status is 'paid', the 'payment time' in the sales information must not be empty" and "When the work order type is 'price adjustment', the corresponding order discount rate must be greater than 0." Real-time verification is achieved through Spark SQL's WHERE conditions or Flink's ProcessFunction. Exception reports are generated for data records that violate the rules and isolated for processing.

[0090] Comprehensive views are generated by integrating multi-source data after association cleaning into wide tables (including 100+ fields such as work order operation records, order details, customer attributes, sales indicators, etc.). For example, a three-dimensional view of "order-work order-customer" is generated. Each record contains information such as order ID, creation time, customer name, historical purchase frequency, number of work order modifications, and final payment amount, supporting subsequent multi-dimensional analysis and model training.

[0091] Through the comprehensive view generated by cross-source association, fragmented work order, order and other data are integrated into a complete business chain including transaction processes, operation records, and user behaviors, enabling the analysis of violation behaviors to be upgraded from "single point detection" to "process tracing" (for example, by associating work order modification records with changes in order amounts, malicious price tampering can be identified); the computing power of the distributed framework supports the real-time integration of billions of data, and the processing efficiency is improved compared to traditional stand-alone ETL tools, making multi-dimensional correlation analysis that was previously impossible to achieve (such as modeling the association between work order operations and historical credit of tens of millions of customers) possible.

[0092] In some embodiments, storing the fused data in a preset data warehouse includes: storing the fused data in a preset distributed storage system; building a data access interface through a preset SQL query engine, and realizing traceability management of the fused data by recording the source, field definition, update time and change history of the fused data.

[0093] Distributed storage system selection: Select a storage system based on data scale and access requirements: For massive historical data at the petabyte level, use Hadoop HDFS (supports high-throughput offline analysis) or Amazon S3 (object storage, suitable for unstructured data); for scenarios with high real-time query requirements, use Google BigQuery (Serverless data warehouse) or Hive (HDFS-based SQL engine), which supports data partitioning (by time and region) and bucketing (to improve query efficiency).

[0094] Deploy SQL query engines such as Hive, Presto, and Impala to build a unified data access interface: Hive is suitable for offline batch queries, Presto supports interactive queries (millisecond-level response) across multiple data sources (HDFS, S3, relational databases), and Impala provides real-time analysis capabilities for Hive data; expose the query service externally through JDBC / ODBC interfaces, and support business systems (such as the risk control platform) to access the integrated data through standard SQL statements.

[0095] Adopt metadata management systems such as Apache Atlas and AWS Glue Catalog to record the complete lineage information of the integrated data: mark the data source where the data was initially collected (such as "work order information comes from ERP system V1.0"), the collection time, and the collection method (API / file transfer);

[0096] Field definitions include the business meaning of each field (such as "work order modification frequency = number of modifications on the same day / number of hours since the work order was created"), data type, and value range (such as "order amount > 0"); record each change to the data (such as at 14:00 on June 23, 2025, the field "discount rate" was changed from retaining 2 decimal places to 3 decimal places due to business rule adjustments), and support data rollback through version control.

[0097] The storage optimization strategy partitions the frequently accessed business tables (such as the "real-time order integration table") by time fields (such as "creation time") to reduce the data scan range during queries; bucketizes the dimension tables (such as the "customer dimension table") to improve the efficiency of join operations; reduces the storage space by 60% through storage format optimization (replacing CSV with Parquet columnar storage), and at the same time increases the query speed by more than 40%.

[0098] The combination of distributed storage and SQL engines supports sub-second queries for billions of data, meeting the stringent requirements of real-time detection for data response speed, and improving the query efficiency compared to traditional centralized databases; the metadata management system realizes full-link tracing from "intelligent anomaly metrics" to "original data source fields" (for example, by backtracking the corresponding sales information collection interface and cleaning rules through the abnormal actual received amount threshold), facilitating the positioning of data quality problems and the detection of logic loopholes.

[0099] In some embodiments, before storing the integrated data in a preset data warehouse, the method further includes: obtaining business analysis requirements; selecting a star or snowflake data warehouse architecture according to the business analysis requirements, constructing a real-time data transmission channel based on a preset message queue, and configuring index and partition strategies according to the real-time data transmission channel and storage engine characteristics to form a data warehouse that supports efficient queries and real-time updates.

[0100] Business analysis requirements are obtained by communicating with business departments (risk control, sales, operations) to clarify the core application scenarios of the data warehouse: high-frequency simple queries (such as daily abnormal order quantity statistics): suitable for star schema (pre-aggregation of dimension tables to reduce join operations); complex dimension analysis (such as comparison of violation behaviors in different regions and different customer levels): suitable for snowflake schema (normalized design to support flexible dimension expansion); collect performance requirements (such as real-time detection requires data update latency < 1 minute, offline analysis allows T+1 update), which serve as the basis for architecture selection and storage strategy.

[0101] The star schema is used to build "fact table (order transaction fact) + dimension tables (customer, time, region)". The dimension tables are highly redundant (such as the customer dimension contains all relevant attributes), simplifying the query logic and being suitable for quickly generating business reports.

[0102] The snowflake schema is used to further split the dimension tables into sub-dimensions (such as splitting the customer dimension into "customer basic information", "customer credit rating", "customer transaction history"). Through foreign key associations, data redundancy is reduced, making it suitable for scenarios that require frequent modification of dimension attributes (such as adding customer tags).

[0103] The real-time data transmission channel is built by using message queues such as Kafka and RabbitMQ to build a real-time data pipeline, achieving low-latency transmission of data from the ETL process to the data warehouse: the data processed by the upstream ETL is written into the Kafka topic, and the data warehouse subscribes to the topic in real time through a consumer group, supporting millisecond-level data synchronization; perform transaction control on real-time data (such as ensuring that each piece of data is only processed once), and improve throughput through message partitioning and parallel processing of consumers (supporting data writes of tens of thousands of TPS).

[0104] Create B-tree indexes (relational storage) or Bloom filters (distributed storage) for fields with high query frequencies (such as "order ID", "customer ID") to accelerate data retrieval; divide storage partitions according to business time characteristics (such as partitioning by "year-month-day") or data attributes (such as partitioning by "region", "business line"). For example, store the order data for June 2025 in an independent partition, and only scan the target partition during query, reducing I / O overhead; for HDFS storage of Hive, adopt the Hive Partition + Bucket combination strategy; for Amazon Redshift, use Sort Key (sorting key) and Dist Key (distribution key) to optimize query performance, shortening the execution time of complex analysis tasks.

[0105] Dynamically select a star / snowflake architecture according to business requirements to avoid "overdesign" or "performance bottlenecks" caused by traditional fixed architectures. For example, during the promotion period of the retail business, a star architecture is adopted to accelerate the generation of real-time reports, and during the regular period, it is switched to a snowflake architecture to support refined dimensional analysis; the real-time channel built by the message queue shortens the latency of data from the completion of ETL to the availability in the data warehouse from the "hour level" of batch processing to the "second level", supporting real-time detection scenarios (such as instant warnings for high-frequency work order adjustments); the partitioning and indexing strategies reduce the execution time of typical queries (such as "query the number of abnormal orders in each region last week") to, and the efficiency of complex association queries (such as "association analysis of customer credit ratings and work order modification frequencies") is increased by more than 5 times, significantly reducing the consumption of computing resources.

[0106] In some embodiments, the machine learning model includes a supervised learning model, an unsupervised learning model, and a time series analysis model; based on the fusion data in the data warehouse, corresponding intelligent anomaly metrics are generated according to the pre-trained machine learning model for analyzing violation behaviors and obtaining anomaly detection results, including: training the supervised learning model with labeled historical anomaly data; identifying anomaly patterns in the unlabeled historical anomaly data through the unsupervised learning model, combining the time series analysis model to perform trend prediction on time series data, and generating intelligent anomaly metrics including the threshold of abnormal received amount, the threshold of work order adjustment frequency, and the deviation degree of order status; matching the real-time data through the intelligent anomaly metrics and outputting the detection results of violation behaviors.

[0107] The supervised learning model trains decision tree, random forest, and support vector machine (SVM) models by using historical labeled anomaly data (such as 100,000 manually marked abnormal work orders and 50,000 false orders). The input features include more than 50 dimensions such as the number of work order modifications, order discount rate, and the number of historical violations of customers. The optimal model parameters (such as the tree depth of the random forest = 15 and the subsample ratio = 0.8) are selected through cross-validation (10-fold).

[0108] The unsupervised learning model applies K-means clustering (presetting 5 normal business clusters) and isolation forest algorithm to the unlabeled historical data (about 1 million normal transaction records), identifies outliers by calculating the "isolation score" of data points (scores > 0.9 are determined as anomalies), and captures unknown violation patterns (such as the first emergence of a cross-regional high-frequency small-amount transaction cluster).

[0109] The time series model models time series data such as sales amount and work order adjustment frequency using ARIMA (automatically identifying the order of differencing) or Prophet (supporting holiday effects), predicts the normal business trends in the next 30 days, and defines the deviation between the real-time data and the predicted value (such as the sales amount being lower than 90% of the predicted value for 3 consecutive days and showing a downward trend) as a time series anomaly signal.

[0110] Intelligent anomaly index generation includes: Anomaly actual received amount threshold: Combining the "high-risk amount range" output by the supervised learning model (such as the actual received amount of a single order < 80% of the cost price) and the "recent sales volume fluctuation threshold" of the time series model (a month-on-month decrease exceeding 25%), a dynamic amount anomaly threshold is generated; Work order adjustment frequency threshold: By identifying the normal work order modification frequency distribution through unsupervised learning (such as an average of ≤ 3 modifications per hour), combined with the characteristics of the business period (the upper limit is 5 times per hour from 10:00 to 16:00 on weekdays, and 2 times per hour in other periods), a period-sensitive frequency threshold is generated; Order status deviation degree: Calculate the transition probability of the actual order status and the expected status (such as the probability that a "paid" order does not change to "shipped" within 24 hours exceeds 1.5 times the historical average), and the deviation score is output through the SVM model (0 - 10 points, and a score of ≥ 7 points is determined as an anomaly).

[0111] The real-time detection process includes: Feature engineering: Extracting immediate features from the real-time inflowing integrated data (such as the number of modifications to the current work order in the past 1 hour, the proportion of the order discount rate deviating from the customer's historical average), and inputting it into the model after normalization; Multi-model fusion decision-making: Using a voting mechanism to integrate the outputs of three types of models (supervised learning determines an anomaly + unsupervised detection identifies an outlier + time series trend anomaly, and if any two of the three are satisfied, an early warning is triggered), reducing misjudgments of a single model and outputting the final detection result (such as "High-risk order: Anomaly in actual received amount + High-frequency work order adjustment, it is recommended for manual review").

[0112] The supervised learning model accurately identifies known violation patterns (such as the historical "low discount and high amount anomaly"), the unsupervised learning captures new unknown anomalies (such as the first occurrence of "multiple accounts collaborating to brush orders"), and the time series model detects trend violations (such as a continuous week of abnormal sales volume decline). The three types of models complement each other to form an all-round detection system; Dynamic threshold adaptability: Dynamic thresholds based on time series and business rules (such as automatically relaxing the discount rate threshold during the promotion period), enabling the detection system to automatically adjust with changes in the business scenario (such as the Double 11 promotion, Spring Festival holiday). Compared with the fixed threshold scheme, the false alarm rate is reduced by 50% and the missed alarm rate is reduced by 40%; Detection accuracy improvement: The multi-model fusion decision-making mechanism improves the comprehensive detection F1 score, especially in complex scenarios (such as violations with both amount anomalies and time series trend anomalies), the detection confidence is increased, and the cost of manual review is reduced.

[0113] In some embodiments, optimizing the machine learning model according to the feedback information, service requirement information, and environmental change information includes: performing real-time feature extraction on newly accessed service data through an online learning algorithm, and dynamically updating the parameters of the machine learning model; establishing a user feedback interface to collect the manually labeled anomaly detection results as the feedback information, and performing backpropagation optimization on the model prediction error corresponding to the machine learning model; and adaptively adjusting the anomaly detection threshold of the machine learning model according to the real-time obtained service requirement information and environmental change information, so as to complete the optimization of the machine learning model.

[0114] Online learning and parameter update include using online learning algorithms such as SGD (Stochastic Gradient Descent) and Adam to perform real-time feature extraction on newly accessed service data (more than 100,000 new detection records are added daily). When the cumulative number of new samples reaches 5,000, the model parameter update is triggered: for the supervised learning model (random forest), the tree node splitting rule is updated through an incremental learning algorithm to adapt to the new features (such as introducing the "payment channel risk coefficient"); for the unsupervised learning model (isolation forest), the feature weights for outlier detection are dynamically adjusted (such as increasing the weight of the "cross-border IP access" feature that frequently appears recently by 30%).

[0115] User feedback-driven optimization includes feedback interface design: providing a manual labeling interface on the risk control platform, where operators can mark the detection results as "false positive", "false negative", or "correct", and the labeled data is synchronously updated to the model training set in real time (including original features, model prediction values, and manual labels); backpropagation optimization: for the samples marked as "false positive" (such as a normal promotion activity being misreported), calculating the model prediction error (such as the classification margin deviation of SVM), and adjusting the feature weights through backpropagation (reducing the influence factor of the "discount rate" feature); for the samples marked as "false negative" (such as a new type of fraud not being detected), increasing the sensitivity of the relevant features (such as increasing the weight of the "IP address change frequency"); feedback statistical analysis: generating a feedback report (top 5 scenarios of false positive rate and false negative rate) every week, and automatically triggering targeted optimization tasks (such as adding the "activity date" feature as an exemption condition for the scenario of "false positive of normal high discount on membership day").

[0116] Business and environment adaptive adjustment includes: Business requirement response: Connect to the enterprise risk control management system to obtain real-time changes in business rules (such as "the modification threshold of new business line work orders is adjusted to 8 times per hour" and "the discount rate threshold for VIP customers is relaxed to 30%"), and directly modify the abnormal index calculation logic through the rule engine (such as adding a "business line" field to the customer dimension table, and calculating the threshold by grouping according to the business line); Perception of environmental changes: Crawl industry violation reports and regulatory new rules (such as new payment and settlement policies), extract key information (such as "virtual currency payments are prohibited") through NLP technology, automatically trigger the addition of a "payment method compliance" detection dimension to the model feature engineering module, and retrain relevant model components (such as updating the classification boundary of SVM).

[0117] The optimization effect is verified by establishing a model evaluation index system (AUC-ROC, F1 score, response latency). After each optimization, the validation set (including 10,000 labeled data) is automatically run. If the indicators do not meet the standards (such as the F1 score drops by more than 2%), it will roll back to the historical version to ensure that the optimization process is controllable.

[0118] In some embodiments, by constructing a data fusion engine with the ability of "intelligent semantic understanding + dynamic rule evolution", problems such as traditional ETL relying on manual configuration and cross-domain data semantic ambiguity are solved, and the upgrade from "rule-driven" to "intelligent-driven" is realized.

[0119] The construction of the business knowledge graph constructs an enterprise-level business knowledge graph by using the Neo4j graph database, including entities (customers, orders, work orders, products), relationships (customers create orders, work orders are associated with orders, products belong to categories), and attributes (order amount data type, work order status enumeration value). Through natural language processing to parse the data dictionary document, more than 100,000 entity relationships are automatically extracted; when a new data source is added, the existing entities are matched through an entity alignment algorithm (such as TransE) (such as determining whether "user ID" and "customer code" are the same entity), and the accuracy rate is over 98%.

[0120] By training a meta - learning model (MAML), input historical data cleaning cases (such as the processing process of the "inconsistent date format" problem of a certain data source), and output personalized cleaning strategies for new data sources (such as automatically generating Python cleaning code snippets); through the semantic constraints of the knowledge graph (such as "the order amount must be a positive number"), logically verify the generated cleaning rules to avoid data loss caused by incorrect rules (such as accidentally deleting valid negative order data). When cross - source data conflicts are detected (such as the "registration time" in customer information and the "first interaction time" in work order information differ by more than 30 days), trigger the graph reasoning engine: first retrieve the business rule knowledge base (such as "the customer registration time should be earlier than the first work order interaction time"), if there is a clear rule, it will be automatically corrected; if there is no clear rule, analyze the context of the conflicting data (such as associated order status, customer historical behavior) through a graph neural network (GNN), and generate conflict resolution suggestions (such as marking as "pending manual review" and attaching a confidence score).

[0121] Exemplarily, the entity alignment algorithm adopts the TransE model, and its core formula is used to calculate the rationality that the head entity (h) is connected to the tail entity (t) through the relationship (r). The formula is as follows:

[0122] ;

[0123] Among them, h is the head entity vector, which is the d - dimensional vector representation of the head entity (such as "user ID", "customer code") in the knowledge graph and is obtained through graph embedding training. It is automatically learned by the TransE model during the training process, with an initial value of a random normal distribution (mean 0, standard deviation 0.1), and the dimension d usually takes 100 - 300 (adjusted according to the scale of the graph).

[0124] r is the relationship vector, which is the d - dimensional vector representation of the relationship (such as "equivalent to", "associated with") between the head entity and the tail entity, reflecting the semantic features of the relationship. It has the same dimension as the entity vector, and the semantic differences of the relationships are learned through training (for example, the relationship vector of "equivalent to" approaches the 0 vector, and the relationship vector of "parent class" reflects the hierarchical difference).

[0125] t is the tail entity vector, which is the d - dimensional vector representation of the tail entity (the entity connected to the head entity through the relationship r) in the knowledge graph. During the training process, it is co - optimized with the head entity and the relationship vector to ensure that the vectors of similar entities are close in space.

[0126] is the distance function, which is used to measure the Euclidean distance between "head entity + relationship vector" and the tail entity vector. The smaller the distance, the higher the credibility of entity alignment. is a non - negative real number. When h + r = t, the distance is 0 (ideal alignment state). In the actual scenario, it is judged through a threshold (such as when the distance < 0.5, it is determined as an aligned entity).

[0127] Please refer to Figure 2 , Figure 2 FIG. is a schematic block diagram of an illegal behavior analysis device based on big data model prediction provided by an embodiment of the present application. The illegal behavior analysis device 200 based on big data model prediction is used to execute the foregoing illegal behavior analysis method based on big data model prediction. Among them, the illegal behavior analysis device based on big data model prediction can be configured in a server or a terminal.

[0128] Among them, the server can be an independent server, a server cluster, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms. The terminal can be an electronic device such as a mobile phone, a tablet computer, a notebook computer, a desktop computer, a user digital assistant, and a wearable device.

[0129] As Figure 2 shown, the illegal behavior analysis device 200 based on big data model prediction includes:

[0130] A data acquisition unit 201, configured to acquire raw data from multiple preset data sources; the preset data sources at least include work order information, order information, sales information, and customer information;

[0131] A data extraction unit 202, configured to extract, transform, and load the raw data, fuse the raw data from different data sources, obtain fused data, and store the fused data in a preset data warehouse. The data warehouse adopts a star or snowflake architecture and is used to record the source, definition, and change history of the data;

[0132] An anomaly detection unit 203, configured to generate corresponding intelligent anomaly metrics for illegal behavior analysis based on the fused data in the data warehouse according to a pre-trained machine learning model, and obtain an anomaly detection result; the intelligent anomaly metrics include abnormal actual receipt information, frequent work order adjustment information, and order anomaly information;

[0133] A model optimization unit 204, configured to obtain feedback information corresponding to the anomaly detection result, and obtain real-time business requirement information and environmental change information, so as to optimize the machine learning model according to the feedback information, business requirement information, and environmental change information.

[0134] In some embodiments, the extraction, transformation, and loading of the original data include: using a preset ETL tool to extract the original data, transforming the original data according to preset business rules, and loading the processed original data into the corresponding data processing flow.

[0135] Exemplarily, the using a preset ETL tool to extract the original data includes: performing cleaning operations on missing values, duplicate values, and outliers in the original data, and eliminating format differences of the original data from different data sources through data format normalization processing.

[0136] In some embodiments, the fusion of the original data from different data sources to obtain fusion data includes: using a preset big data processing framework to perform field mapping and association on the original data from work order information, order information, sales information, and customer information based on a preset data standard; generating a comprehensive view covering multi-dimensional business information as the fusion data through data redundancy elimination, conflict value calibration, and semantic consistency verification.

[0137] In some embodiments, the storing the fusion data in a preset data warehouse includes: storing the fusion data in a preset distributed storage system; constructing a data access interface through a preset SQL query engine, and realizing traceability management of the fusion data by recording the source, field definition, update time, and change history of the fusion data.

[0138] In some embodiments, before storing the fusion data in a preset data warehouse, the method further includes: obtaining business analysis requirements; selecting a star or snowflake data warehouse architecture according to the business analysis requirements, and constructing a real-time data transmission channel based on a preset message queue to configure an index and partition strategy according to the real-time data transmission channel and storage engine characteristics, so as to form a data warehouse supporting efficient query and real-time update.

[0139] In some embodiments, the machine learning model includes a supervised learning model, an unsupervised learning model, and a time series analysis model; based on the fusion data in the data warehouse, generating corresponding intelligent anomaly metrics according to a pre-trained machine learning model for violation behavior analysis and obtaining an anomaly detection result includes: training the labeled historical anomaly data using a supervised learning model; identifying anomaly patterns in the unlabeled historical anomaly data through an unsupervised learning model, combining the time series analysis model to perform trend prediction on time series data, and generating intelligent anomaly metrics including an abnormal actual receipt amount threshold, a work order adjustment frequency threshold, and an order status deviation degree; matching the real-time data through the intelligent anomaly metrics and outputting a violation behavior detection result.

[0140] In some embodiments, optimizing the machine learning model according to the feedback information, service requirement information, and environmental change information includes: performing real-time feature extraction on newly accessed service data through an online learning algorithm to dynamically update the parameters of the machine learning model; establishing a user feedback interface to collect the manually labeled anomaly detection results as the feedback information, and performing backpropagation optimization on the model prediction error corresponding to the machine learning model; and adaptively adjusting the anomaly detection threshold of the machine learning model according to the real-time obtained service requirement information and environmental change information to complete the optimization of the machine learning model.

[0141] It should be noted that those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described model training device and each module can refer to the corresponding processes in the embodiments of the foregoing method for analyzing violation behaviors based on big data model prediction, and will not be elaborated herein.

[0142] The above-described device for analyzing violation behaviors based on big data model prediction can be implemented in the form of a computer program, and this computer program can run on a computer device as shown in Figure 3 Figure.

[0143] Please refer to Figure 3 , Figure 3 which is a schematic block diagram of the structure of a computer device provided by an embodiment of the present application. This computer device can be a server or a terminal.

[0144] Referring to Figure 3 , this computer device includes a processor, a memory, and a network interface connected through a system bus. Among them, the memory can include a storage medium and an internal memory.

[0145] The storage medium can store an operating system and a computer program. This computer program includes program instructions, and when the program instructions are executed, the processor can be made to execute any one of the methods for analyzing violation behaviors based on big data model prediction provided by the embodiments of the present application.

[0146] The processor is used to provide computing and control capabilities to support the operation of the entire computer device.

[0147] The internal memory provides an environment for the operation of the computer program in the storage medium. When this computer program is executed by the processor, the processor can be made to execute any one of the methods for analyzing violation behaviors based on big data model prediction. The storage medium can be non-volatile or volatile.

[0148] This network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art can understand that Figure 3The structure shown is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0149] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0150] Exemplarily, in one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0151] Obtain raw data from multiple preset data sources; the preset data sources at least include work order information, order information, sales information, and customer information;

[0152] Extract, transform, and load the raw data, and fuse the raw data from different data sources to obtain fused data, and store the fused data in a preset data warehouse. The data warehouse adopts a star or snowflake architecture and is used to record the source, definition, and change history of the data;

[0153] Based on the fused data in the data warehouse, generate corresponding intelligent anomaly metrics according to a pre-trained machine learning model for analyzing violation behaviors, and obtain anomaly detection results; the intelligent anomaly metrics include abnormal actual receipt information, frequent work order adjustment information, and order anomaly information;

[0154] Obtain feedback information corresponding to the anomaly detection results, and obtain real-time business requirement information and environmental change information to optimize the machine learning model according to the feedback information, business requirement information, and environmental change information.

[0155] In some embodiments, the extracting, transforming, and loading of the raw data includes: using a preset ETL tool to extract the raw data, and transforming the raw data according to preset business rules, and loading the processed raw data into the corresponding data processing process.

[0156] Exemplarily, the extraction of the raw data by using a preset ETL tool includes: performing a cleaning operation on missing values, duplicate values, and outliers in the raw data, and eliminating the format differences of the raw data from different data sources through data format normalization processing.

[0157] In some embodiments, the fusion of the raw data from different data sources to obtain fusion data includes: using a preset big data processing framework to perform field mapping and association on the raw data from work order information, order information, sales information, and customer information based on a preset data standard; generating a comprehensive view covering multi-dimensional business information as the fusion data through data redundancy elimination, conflict value calibration, and semantic consistency verification.

[0158] In some embodiments, the storing of the fusion data in a preset data warehouse includes: storing the fusion data in a preset distributed storage system; constructing a data access interface through a preset SQL query engine, and realizing the traceability management of the fusion data by recording the source, field definition, update time, and change history of the fusion data.

[0159] In some embodiments, before storing the fusion data in a preset data warehouse, the method further includes: obtaining business analysis requirements; selecting a star or snowflake data warehouse architecture according to the business analysis requirements, and constructing a real-time data transmission channel based on a preset message queue to configure an index and a partitioning strategy according to the real-time data transmission channel and storage engine characteristics, so as to form a data warehouse supporting efficient query and real-time update.

[0160] In some embodiments, the machine learning model includes a supervised learning model, an unsupervised learning model, and a time series analysis model; based on the fusion data in the data warehouse, generating corresponding intelligent anomaly metrics according to a pre-trained machine learning model for analyzing violation behaviors and obtaining anomaly detection results includes: training the supervised learning model on labeled historical anomaly data; identifying anomaly patterns in unlabeled historical anomaly data through the unsupervised learning model, combining the time series analysis model to perform trend prediction on time series data, and generating intelligent anomaly metrics including an abnormal actual receipt amount threshold, a work order adjustment frequency threshold, and an order status deviation degree; matching the real-time data through the intelligent anomaly metrics and outputting a violation behavior detection result.

[0161] In some embodiments, optimizing the machine learning model according to the feedback information, service requirement information, and environmental change information includes: extracting real-time features from newly accessed service data through an online learning algorithm to dynamically update the parameters of the machine learning model; establishing a user feedback interface to collect the manually labeled anomaly detection results as the feedback information, and performing backpropagation optimization on the model prediction error corresponding to the machine learning model; and adaptively adjusting the anomaly detection threshold of the machine learning model according to the real-time obtained service requirement information and environmental change information to complete the optimization of the machine learning model.

[0162] Among them, the computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a SmartMedia Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the computer device.

[0163] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of various equivalent modifications or substitutions, and these modifications or substitutions should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for analyzing illegal behaviors based on big data model prediction, characterized in that, Including: Obtain raw data from multiple preset data sources; The preset data sources at least include work order information, order information, sales information, and customer information; Extract, transform, and load the raw data, fuse the raw data from different data sources, obtain fused data, and store the fused data in a preset data warehouse. The data warehouse adopts a star or snowflake architecture and is used to record the source, definition, and change history of the data; Based on the fused data in the data warehouse, generate corresponding intelligent anomaly metrics according to a pre-trained machine learning model for violation behavior analysis, and obtain anomaly detection results; the intelligent anomaly metrics include abnormal actual receipts information, frequent work order adjustment information, and order anomaly information; Obtain feedback information corresponding to the anomaly detection results, and obtain real-time business requirement information and environmental change information to optimize the machine learning model according to the feedback information, business requirement information, and environmental change information.

2. The method according to claim 1, characterized in that The extracting, transforming, and loading the raw data includes: Adopt a preset ETL tool to extract the raw data, transform the raw data according to preset business rules, and load the processed raw data into the corresponding data processing process.

3. The method according to claim 2, wherein The adopting a preset ETL tool to extract the raw data includes: Execute cleaning operations on missing values, duplicate values, and abnormal values in the raw data, and eliminate the format differences of the raw data from different data sources through data format normalization processing.

4. The method according to claim 1, wherein The fusing the raw data from different data sources to obtain fused data includes: Utilize a preset big data processing framework to perform field mapping and association on the raw data from work order information, order information, sales information, and customer information based on preset data standards; Generate a comprehensive view covering multi-dimensional business information as the fused data through data redundancy elimination, conflict value calibration, and semantic consistency verification.

5. The method according to claim 1, wherein The storing the fused data in a preset data warehouse includes: Store the fused data in a preset distributed storage system; Construct a data access interface through a preset SQL query engine, and realize the traceability management of the fused data by recording the source, field definition, update time, and change history of the fused data.

6. The method according to claim 1, wherein Before the storing the fused data in a preset data warehouse, the method further includes: Obtain business analysis requirements; Select a star or snowflake data warehouse architecture according to business analysis requirements, construct a real-time data transmission channel based on a preset message queue, and configure an index and partition strategy according to the real-time data transmission channel and storage engine characteristics to form a data warehouse supporting efficient query and real-time update.

7. The method according to claim 1, characterized in that The machine learning model includes a supervised learning model, an unsupervised learning model, and a time series analysis model; the generating corresponding intelligent anomaly metrics according to a pre-trained machine learning model for violation behavior analysis based on the fused data in the data warehouse and obtaining anomaly detection results includes: Use a supervised learning model to train the labeled historical anomaly data; Identify abnormal patterns in unlabeled historical abnormal data through an unsupervised learning model, combine with a time series analysis model to predict the trend of time series data, and generate intelligent abnormal indicators including abnormal actual received amount threshold, work order adjustment frequency threshold, and order status deviation degree; Match the real-time data through the intelligent abnormal indicators and output the detection results of violation behaviors.

8. The method according to claim 1, wherein The optimizing the machine learning model according to the feedback information, business requirement information, and environmental change information includes: Extract real-time features from newly accessed business data through an online learning algorithm and dynamically update the parameters of the machine learning model; Establish a user feedback interface to collect the manually labeled abnormal detection results as the feedback information, and perform backpropagation optimization on the model prediction error corresponding to the machine learning model; According to the real-time obtained business requirement information and environmental change information, adaptively adjust the abnormal detection threshold of the machine learning model to complete the optimization of the machine learning model.

9. An illegal behavior analysis device based on big data model prediction, characterized in that, The device includes: A data acquisition unit for acquiring raw data from multiple preset data sources; the preset data sources at least include work order information, order information, sales information, and customer information; A data extraction unit for extracting, transforming, and loading the raw data, fusing the raw data from different data sources to obtain fused data, and storing the fused data in a preset data warehouse. The data warehouse adopts a star or snowflake architecture and is used to record the source, definition, and change history of the data; An abnormal detection unit for generating corresponding intelligent abnormal indicators based on the fused data in the data warehouse according to a pre-trained machine learning model to perform violation behavior analysis and obtain abnormal detection results; the intelligent abnormal indicators include abnormal actual received information, frequent work order adjustment information, and order abnormal information; A model optimization unit for obtaining the feedback information corresponding to the abnormal detection results, and obtaining real-time business requirement information and environmental change information, so as to optimize the machine learning model according to the feedback information, business requirement information, and environmental change information.

10. A computer device, characterized in that, The computer device includes a memory and a processor; The memory is used to store a computer program; The processor is used to execute the computer program and implement the method according to any one of claims 1 to 8 when executing the computer program.

Images