A multi-source data analysis system and method based on artificial intelligence

Abstract

The present invention discloses a multi-source data analysis system and method based on artificial intelligence, which relates to the field of data analysis technology. The method includes the following steps: Step 1, obtaining structured data and metadata from different data sources, and monitoring transmission rate and loss rate; Step 2, preprocessing the metadata and converting it into an integrated data set; Step 3, extracting statistical features, setting thresholds to determine abnormal situations, summarizing abnormal information to establish an indicator system, and generating an abnormality report; Step 4, constructing a decision tree classification model based on multidimensional features, classifying according to abnormality index; incrementally updating model parameters; Step 5, using the classification model to determine abnormalities in new data, setting response rules based on the abnormal classification results, pushing feedback messages, and generating logs. The present invention can effectively improve the lack of directionality in the response and identification of abnormal data in existing technologies.

Description

Technical Field

[0001] The present invention relates to the field of data analysis technology, and in particular to a multi-source data analysis system and method based on artificial intelligence. Background Art

[0002] In today's digital age, the analysis and processing of multi-source data is crucial for decision-making, business optimization, and risk management across various industries. Multi-source data encompasses data from a variety of formats and sources, including databases, formatted text files, and standard APIs. During data transmission, transmission speeds and loss rates can impact data integrity and timeliness. Anomaly detection aims to identify data points that differ significantly from the majority of the data. These anomalies may reflect important information such as system failures, unusual business operations, or data errors. Effective processing of anomaly data enables timely correction, ensuring stable system operations and improving business efficiency.

[0003] Existing technologies for multi-source data analysis have numerous limitations. During the data acquisition phase, access to diverse data sources is complex and lacks a unified, efficient integration approach, resulting in inefficient and error-prone data acquisition. Furthermore, existing technologies lack specificity in responding to abnormal data. When abnormal data is detected, differentiated processing strategies are often not developed based on factors such as the type and severity of the anomaly. Instead, a unified, general approach is often employed, failing to implement the most appropriate measures for each specific anomaly. Summary of the Invention

[0004] The purpose of the present invention is to provide a multi-source data analysis system and method based on artificial intelligence to solve the problems raised in the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a multi-source data analysis method based on artificial intelligence, the method comprising the following steps:

[0006] Step 1: Obtain structured data and metadata from different data sources and monitor the transmission rate and loss rate;

[0007] Step 2: Preprocess the metadata and obtain the integrated data set through conversion;

[0008] Step 3: Extract statistical features, set thresholds to judge abnormal situations, summarize abnormal information to establish an indicator system, and generate abnormality reports;

[0009] Step 4: Build a decision tree classification model based on multidimensional features, classify by anomaly index, and incrementally update model parameters;

[0010] Step 5: Use the classification model to determine the anomaly of the new data, set the response rules based on the anomaly classification results, push the feedback message and generate the log.

[0011] In step 1, for database data: the staff pre-configures a standard database connection protocol (such as JDBC / ODBC), identifies the data table structure based on the data source type based on the standard database connection protocol, and reads the data;

[0012] For formatted text files: Use a file parser (such as parsing rules for CSV, JSON, and XML) to automatically parse the file content and identify the data fields in the file;

[0013] For standard API interfaces: Build API call modules according to the RESTful protocol, automatically initiate requests, and parse the returned JSON / XML data;

[0014] Obtain structured data and metadata in a unified format; the metadata description includes the field name, data type, collection time t, and data volume m; where m is a positive integer representing the number of metadata records;

[0015] The field is denoted as X j ; Where j = 1, 2, ..., n, is used to distinguish different fields; n is a positive integer, indicating the number of metadata description fields;

[0016] Records are represented by x i ; Where i=1,2,…,m, is used to distinguish different records;

[0017] Monitor the transmission rate v(t) and loss rate L:

[0018] Using the sliding window, calculate the average transmission rate: v bar =1 / w·Σ w i=1 v(t i ); where w represents the window length;

[0019] Calculate the packet loss rate, the number of lost packets is expressed as N loss , the total number of packets is expressed as N total , the loss rate calculation formula is: L=N loss / N total ;

[0020] In step 2, for metadata, missing value processing is performed: for field X j The missing values ​​in are filled using the mean filling formula; the data matrix after missing value processing and the filled log are obtained;

[0021] For the data matrix after missing value processing, the Z-score method is used to detect noise: for each field, the mean and standard deviation are calculated, and the Z-score value is further calculated. When the Z-score value exceeds the preset threshold, it is considered to be abnormal noise data. The noise-filtered data matrix and noise anomaly report are obtained.

[0022] Standardize the format of the noise-filtered data matrix: convert the date and time data to a fixed format (such as ISO8601 standard); the original date is represented by d raw , after conversion, it is recorded as d std =f date (d raw );

[0023] Among them, f date Represents the date conversion function corresponding to a fixed format; obtains a data matrix with a unified format;

[0024] Based on the data matrix with unified format, Min-Max normalization is used to convert the numerical data items to the interval [0,1]; the normalized data matrix is ​​obtained;

[0025] For the normalized data matrix, perform data matching and fusion: use the common field timestamp t to perform inner or outer joins to generate the integrated data set;

[0026] In step 3, extract statistical features: for each standardized field X j 'Calculate the mean μ j , standard deviation σ j 、Maximum value max(X j ') and minimum value min(X j '); get the index set I={μ j ,σ j ,max(X j '),min(X j ')};

[0027] Set the threshold to judge abnormality: When the normalized data record x ij 'Not in [μ j -k·σ j ,μ j +k·σ j ] is considered abnormal; where k is a positive integer, representing the preset coefficient;

[0028] For data transmission anomalies: take the transmission rate v(t) as the signal, use mean filtering or median filtering to determine the mutation point in the time series; calculate the first-order difference Δv(t i )=v(t i )-v(t i-1), when |Δv(ti)|>γ, it is considered as a transmission abnormal point; where γ is the set threshold; the transmission abnormal time point set A is obtained trans ;

[0029] For source data anomalies: Isolation forest is used to calculate the anomaly score s(x i ); the abnormal score is determined based on the distance of the data in the feature space. When s(x i )>δ is judged as abnormal; where δ represents the preset threshold; the source data abnormal record set A is obtained source and the corresponding abnormality level;

[0030] For data conversion anomalies: Compare the converted data X' with the expected value X* described by the original metadata and calculate the semantic consistency index: S = 1-||X'-X*|| / ||X*||; when S < λ, it is judged as a data conversion anomaly; where λ is the set minimum alignment; mark and generate a conversion anomaly record set A conv Expected value indicates the expected data state or value range based on the definition, constraints and business rules of the original metadata;

[0031] Summarize various abnormal scores and records and establish an abnormal index system E={A trans ,A source ,A conv Generate anomaly index E(x i ), the calculation formula is: E(x i )=w1·F(x i ∈A trans )+w2·F(x i ∈A source )+w3·F(x i ∈A conv );

[0032] Where F is an indicator function, which takes the value 1 when the condition in the brackets is met, and takes the value 0 otherwise; w1, w2, w3 are the weights of each type of anomaly, 0<w1, w2, w3<1 and w1+w2+w3=1;

[0033] Generate anomaly reports and classification labels.

[0034] In step 4, a classification model based on multidimensional features is constructed: using the branch structure of the decision tree, the abnormal index E(x i ) is compared with the preset category thresholds τ1, τ2:

[0035] E(x i )<τ1, the category is normal; τ1≤E(x i )<τ2, the category is warning; E(xi )≥τ2, the category is abnormal;

[0036] Among them, τ1, τ2 are pre-set category thresholds, τ1<τ2;

[0037] Establish an anomaly classification model M and generate an anomaly index for each new data;

[0038] Compare the prediction results of model M with the actual feedback, adjust the parameters and classification thresholds, and realize incremental model updates.

[0039] In step 5, for new data, after preprocessing, its abnormality level is determined by model M; the response rule is set according to the abnormality classification result: when E(x i )≥τ2, the alarm is automatically triggered and the data is retransmitted; when E(x i )∈[τ1,τ2), it is marked as waiting for manual review; the feedback message is pushed and an automatic processing log is generated.

[0040] An artificial intelligence-based multi-source data analysis system, comprising a data access module, a data preprocessing module, an intelligent analysis module, a pattern recognition module, and a result output module;

[0041] The data access module is used to obtain structured data and metadata from different data sources and monitor the transmission rate and loss rate; the data preprocessing module is used to preprocess the metadata and obtain an integrated data set through conversion; the intelligent analysis module is used to extract statistical features, set thresholds to determine abnormal situations, summarize abnormal information to establish an indicator system, and generate abnormal reports; the pattern recognition module is used to construct a decision tree classification model based on multidimensional features, classify according to the abnormal index, and incrementally update the model parameters; the result output module is used to determine the abnormal situation of new data according to the classification model, set response rules according to the abnormal classification results, push feedback messages and generate logs.

[0042] The data access module includes a database unit, a file parsing unit, an API calling unit and a transmission monitoring unit;

[0043] The database unit is used to connect to the database through the protocol and extract structured data; the file parsing unit is used to parse the file and identify the fields and data content; the API calling unit is used to call the interface according to the RESTful protocol and parse the JSON / XML response; the transmission monitoring unit is used to calculate the transmission rate and loss rate in real time.

[0044] The data preprocessing module includes a missing filling unit, a noise filtering unit, a format standardization unit and a data fusion unit;

[0045] The missing value filling unit is used to process the missing values ​​of the field using the mean interpolation method and record the filling log; the noise filtering unit is used to identify noise data based on the Z-score and generate a noise anomaly report; the format standardization unit is used to unify the date format and normalize the numerical data; the data fusion unit is used to associate multi-source data through timestamps to generate an integrated data set.

[0046] The intelligent analysis module includes a feature extraction unit, an anomaly detection unit and an indicator summary unit;

[0047] The feature extraction unit is used to calculate the field mean, standard deviation and extreme value; the anomaly detection unit is used to identify three types of anomalies through threshold, isolation forest and difference method; the indicator summary unit is used to construct an anomaly indicator system and generate a weighted comprehensive anomaly index.

[0048] The pattern recognition module includes a parameter updating unit and a model building unit;

[0049] The parameter updating unit is used to dynamically adjust the classification threshold according to actual feedback; the model building unit is used to divide the abnormality categories based on the decision tree;

[0050] The result output module includes a classification determination unit, a response rule unit and a log generation unit;

[0051] The classification determination unit is used to output the new data abnormal classification label; the response rule unit is used to trigger the alarm, retransmission or manual review process; and the log generation unit is used to record the abnormal classification results and system response actions.

[0052] Compared with the existing technology, the beneficial effects of the present invention are: the present invention can effectively improve the quality and consistency of data, reduce errors and noise in the data, and provide a reliable data basis for subsequent data analysis and decision-making through preprocessing of multi-source data, including missing value filling, noise filtering, format standardization and normalization, and compared with the traditional single data preprocessing method, it can handle more complex and diverse data problems; the present invention comprehensively considers multiple abnormal situations and establishes a comprehensive abnormal indicator system, making anomaly detection more accurate and comprehensive; it can not only detect anomalies in the data itself, but also discover problems in the data transmission and conversion process. Compared with the existing technology that only focuses on the anomalies of the data itself, it can more timely and accurately discover various abnormal situations of the data in the entire process, which helps to take timely measures to correct and handle them. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a schematic diagram of the steps of a multi-source data analysis method based on artificial intelligence of the present invention;

[0054] Figure 2The figure is a flow chart of a multi-source data analysis system based on artificial intelligence of the present invention. DETAILED DESCRIPTION

[0055] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0056] Example: Figure 1-Figure 2 As shown, the present invention provides a technical solution, a multi-source data analysis method based on artificial intelligence, which includes the following steps:

[0057] Step 1: Obtain structured data and metadata from different data sources and monitor the transmission rate and loss rate;

[0058] Step 2: Preprocess the metadata and obtain the integrated data set through conversion;

[0059] Step 3: Extract statistical features, set thresholds to judge abnormal situations, summarize abnormal information to establish an indicator system, and generate abnormality reports;

[0060] Step 4: Build a decision tree classification model based on multidimensional features, classify by anomaly index, and incrementally update model parameters;

[0061] Step 5: Use the classification model to determine the anomaly of the new data, set the response rules based on the anomaly classification results, push the feedback message and generate the log.

[0062] In step 1, for database data: the staff pre-configures a standard database connection protocol (such as JDBC / ODBC), identifies the data table structure based on the data source type based on the standard database connection protocol, and reads the data;

[0063] For formatted text files: Use a file parser (such as parsing rules for CSV, JSON, and XML) to automatically parse the file content and identify the data fields in the file;

[0064] For standard API interfaces: Build API call modules according to the RESTful protocol, automatically initiate requests, and parse the returned JSON / XML data;

[0065] Obtain structured data and metadata in a unified format; the metadata description includes the field name, data type, collection time t, and data volume m; where m is a positive integer representing the number of metadata records;

[0066] The field is denoted as Xj ; Where j = 1, 2, ..., n, is used to distinguish different fields; n is a positive integer, indicating the number of metadata description fields;

[0067] Records are represented by x i ; Where i=1,2,…,m, is used to distinguish different records;

[0068] Monitor the transmission rate v(t) and loss rate L:

[0069] Using the sliding window, calculate the average transmission rate: v bar =1 / w·Σ w i=1 v(t i ); where w represents the window length;

[0070] Calculate the packet loss rate, the number of lost packets is expressed as N loss , the total number of packets is expressed as N total , the loss rate calculation formula is: L=N loss / N total ;

[0071] In step 2, for metadata, missing value processing is performed: for field X j The missing values ​​in are filled using the mean filling formula; the data matrix after missing value processing and the filled log are obtained;

[0072] For the data matrix after missing value processing, the Z-score method is used to detect noise: for each field, the mean and standard deviation are calculated, and the Z-score value is further calculated. When the Z-score value exceeds the preset threshold, it is considered to be abnormal noise data. The noise-filtered data matrix and noise anomaly report are obtained.

[0073] Standardize the format of the noise-filtered data matrix: convert the date and time data to a fixed format (such as ISO8601 standard); the original date is represented by d raw , after conversion, it is recorded as d std =f date (d raw );

[0074] Among them, f date Represents the date conversion function corresponding to a fixed format; obtains a data matrix with a unified format;

[0075] Based on the data matrix with unified format, Min-Max normalization is used to convert the numerical data items to the interval [0,1]; the normalized data matrix is ​​obtained;

[0076] For the normalized data matrix, perform data matching and fusion: use the common field timestamp t to perform inner or outer joins to generate the integrated data set;

[0077] In step 3, extract statistical features: for each standardized field X j 'Calculate the mean μ j , standard deviation σ j 、Maximum value max(X j ') and minimum value min(X j '); get the index set I={μ j ,σ j ,max(X j '),min(X j ')};

[0078] Set the threshold to judge abnormality: When the normalized data record x ij 'Not in [μ j -k·σ j ,μ j +k·σ j ] is considered abnormal; where k is a positive integer, representing the preset coefficient;

[0079] For data transmission anomalies: take the transmission rate v(t) as the signal, use mean filtering or median filtering to determine the mutation point in the time series; calculate the first-order difference Δv(t i )=v(t i )-v(t i-1 ), when |Δv(ti)|>γ, it is considered as a transmission abnormal point; where γ is the set threshold; the transmission abnormal time point set A is obtained trans ;

[0080] For source data anomalies: Isolation forest is used to calculate the anomaly score s(x i ); the abnormal score is determined based on the distance of the data in the feature space. When s(x i )>δ is judged as abnormal; where δ represents the preset threshold; the source data abnormal record set A is obtained source and the corresponding abnormality level;

[0081] For data conversion anomalies: Compare the converted data X' with the expected value X* described by the original metadata and calculate the semantic consistency index: S = 1-||X'-X*|| / ||X*||; when S < λ, it is judged as a data conversion anomaly; where λ is the set minimum alignment; mark and generate a conversion anomaly record set A conv ;

[0082] Summarize various abnormal scores and records and establish an abnormal index system E={A trans,A source ,A conv Generate anomaly index E(x i ), the calculation formula is: E(x i )=w1·F(x i ∈A trans )+w2·F(x i ∈A source )+w3·F(x i ∈A conv );

[0083] Where F is an indicator function, which takes the value 1 when the condition in the brackets is met, and takes the value 0 otherwise; w1, w2, w3 are the weights of each type of anomaly, 0<w1, w2, w3<1 and w1+w2+w3=1;

[0084] Generate anomaly reports and classification labels.

[0085] In step 4, a classification model based on multidimensional features is constructed: using the branch structure of the decision tree, the abnormal index E(x i ) is compared with the preset category thresholds τ1, τ2:

[0086] E(x i )<τ1, the category is normal; τ1≤E(x i )<τ2, the category is warning; E(x i )≥τ2, the category is abnormal;

[0087] Among them, τ1, τ2 are pre-set category thresholds, τ1<τ2;

[0088] Establish an anomaly classification model M and generate an anomaly index for each new data;

[0089] Compare the prediction results of model M with the actual feedback, adjust the parameters and classification thresholds, and realize incremental model updates.

[0090] In step 5, for new data, after preprocessing, its abnormality level is determined by model M; the response rule is set according to the abnormality classification result: when E(x i )≥τ2, the alarm is automatically triggered and the data is retransmitted; when E(x i )∈[τ1,τ2), it is marked as waiting for manual review; the feedback message is pushed and an automatic processing log is generated.

[0091] An artificial intelligence-based multi-source data analysis system, comprising a data access module, a data preprocessing module, an intelligent analysis module, a pattern recognition module, and a result output module;

[0092] The data access module is used to obtain structured data and metadata from different data sources and monitor the transmission rate and loss rate; the data preprocessing module is used to preprocess the metadata and obtain an integrated data set through conversion; the intelligent analysis module is used to extract statistical features, set thresholds to determine abnormal situations, summarize abnormal information to establish an indicator system, and generate abnormal reports; the pattern recognition module is used to construct a decision tree classification model based on multidimensional features, classify according to the abnormal index, and incrementally update the model parameters; the result output module is used to determine the abnormal situation of new data according to the classification model, set response rules according to the abnormal classification results, push feedback messages and generate logs.

[0093] The data access module includes a database unit, a file parsing unit, an API calling unit and a transmission monitoring unit;

[0094] The database unit is used to connect to the database through the protocol and extract structured data; the file parsing unit is used to parse the file and identify the fields and data content; the API calling unit is used to call the interface according to the RESTful protocol and parse the JSON / XML response; the transmission monitoring unit is used to calculate the transmission rate and loss rate in real time.

[0095] The data preprocessing module includes a missing filling unit, a noise filtering unit, a format standardization unit and a data fusion unit;

[0096] The missing value filling unit is used to process the missing values ​​of the field using the mean interpolation method and record the filling log; the noise filtering unit is used to identify noise data based on the Z-score and generate a noise anomaly report; the format standardization unit is used to unify the date format and normalize the numerical data; the data fusion unit is used to associate multi-source data through timestamps to generate an integrated data set.

[0097] The intelligent analysis module includes a feature extraction unit, an anomaly detection unit and an indicator summary unit;

[0098] The feature extraction unit is used to calculate the field mean, standard deviation and extreme value; the anomaly detection unit is used to identify three types of anomalies through threshold, isolation forest and difference method; the indicator summary unit is used to construct an anomaly indicator system and generate a weighted comprehensive anomaly index.

[0099] The pattern recognition module includes a parameter updating unit and a model building unit;

[0100] The parameter updating unit is used to dynamically adjust the classification threshold according to actual feedback; the model building unit is used to divide the abnormality categories based on the decision tree;

[0101] The result output module includes a classification determination unit, a response rule unit and a log generation unit;

[0102] The classification determination unit is used to output the new data abnormal classification label; the response rule unit is used to trigger the alarm, retransmission or manual review process; and the log generation unit is used to record the abnormal classification results and system response actions.

[0103] In this embodiment, an e-commerce company needs to analyze multi-source data such as sales data and user evaluation data;

[0104] Step 1: Data acquisition and transmission monitoring;

[0105] Database data: Staff pre-configure the JDBC protocol to connect to the company's sales database. Based on this protocol, they identify the sales data table structure and read sales data containing fields such as order number, product ID, sales quantity, sales amount, and order time.

[0106] Formatted text files: Companies regularly receive CSV-formatted logistics and delivery information files from their logistics partners. Using a CSV-specific file parser, the company automatically parses the file content and identifies data fields such as package number, shipping location, destination, and delivery time.

[0107] Standard API interface: Build an API call module according to the RESTful protocol to call the API of a third-party user review platform; automatically initiate a request and parse the returned JSON format data to obtain user review content, ratings, review time, and other information on the product.

[0108] Metadata Generation: While acquiring structured data from the above data sources, metadata in a unified format is generated. For sales data metadata, field names include "Order Number" and "Product ID." Data types, such as "Order Number," are strings. The collection time, t, records the moment each data is acquired, and the data volume, m, counts the number of sales records acquired.

[0109] Transmission monitoring: Use a sliding window to calculate the average transmission rate vbar. The window length w is 10 minutes. The transmission rate v(t) is recorded every 1 minute and the average transmission rate is calculated for these 10 time points. The packet loss rate L is calculated by counting the number of lost packets Nloss and the total number of packets Ntotal to obtain the loss rate.

[0110] Step 2: metadata preprocessing;

[0111] Missing value processing: For the "sales amount" field Xj in the sales data, if there are missing values, calculate the mean of the existing values ​​of the field, use this mean to fill the missing values, and record the filling log.

[0112] Noise detection: Use the Z-score method to detect noise in sales data. Calculate the mean and standard deviation of the "Sales Quantity" field, and further calculate the Z-score value. When the Z-score value exceeds the preset threshold, the corresponding data is considered abnormal noise data, and the noise-filtered data matrix and noise anomaly report are obtained.

[0113] Format standardization: For the date data draw in the order, use the date conversion function fdate corresponding to the ISO8601 standard to convert it to the fixed format dstd, converting "2023 / 10 / 15" to "2023-10-15T00:00:00Z".

[0114] Normalization: For numerical data items such as sales amount, Min-Max normalization is used to convert them to the [0,1] interval.

[0115] Data matching and fusion: Using the common field "order time" t, we inner-join sales data with user review data to generate an integrated dataset, linking sales orders at the same time point with corresponding user reviews.

[0116] Step 3: Statistical feature extraction and anomaly judgment;

[0117] Statistical feature extraction: For the standardized “sales quantity” field Xj’, calculate the mean μj, standard deviation σj, maximum value max(Xj’) and minimum value min(Xj’) to obtain the indicator set I.

[0118] Set a threshold to judge anomalies: when the sales quantity xij' of a sales record after standardization is not within the range of [μj-k・σj,μj+k・σj], it is considered an anomaly; if k=2, μj=5, σj=1, and the sales quantity of a record is 8, it is considered an anomaly.

[0119] Data transmission anomalies: Using the transmission rate v(t) as the signal, apply mean filtering. Calculate the first-order difference Δv(ti) = v(ti) - v(ti-1). When |Δv(ti)| > γ, it is considered a transmission anomaly. For γ = 2, if v(ti) = 12, v(ti-1) = 9, and |Δv(ti)| = 3 > 2, this time point is considered a transmission anomaly, resulting in the set of transmission anomaly time points Atrans.

[0120] Source data anomalies: Isolation forests are used to calculate the anomaly score s(xi) for each record in the user review data. Anomalies are identified when s(xi) > δ. For example, if δ = 0.8 and the anomaly score s(xi) = 0.9 for a review record, this is considered a source data anomaly. The set of source data anomaly records, Asource, and their corresponding anomaly levels are obtained.

[0121] Data conversion anomaly: Compare the converted sales data X' with the expected value X* described by the original metadata and calculate the semantic consistency index S. Assuming λ = 0.9, when S < λ, it is judged as a data conversion anomaly, marked, and the conversion anomaly record set Aconv is generated.

[0122] Summarize anomaly information: Summarize various anomaly scores and records to establish an anomaly index system E. Generate an anomaly index E(xi) for each sales record xi. w1 = 0.3, w2 = 0.4, w3 = 0.3. If a record is a source data anomaly, E(xi) = 0.3 × 0 + 0.4 × 1 + 0.3 × 0 = 0.4.

[0123] Generate exception reports and classification labels: Generate exception reports based on the anomaly index and label the abnormal data with classification labels.

[0124] Step 4: Build and update the decision tree classification model;

[0125] Build a classification model: Using the branching structure of a decision tree, compare the anomaly index E(xi) of the input sales data with the preset classification thresholds τ1 = 0.3 and τ2 = 0.6. When E(xi) < 0.3, the data is classified as normal; when 0.3 ≤ E(xi) < 0.6, the data is classified as warning; and when E(xi) ≥ 0.6, the data is classified as abnormal. This establishes the anomaly classification model M.

[0126] Model update: Compare the prediction results of model M with the actual feedback. If it is found that the actual abnormal data in a certain period of time is misclassified as normal, adjust the model parameters and classification thresholds to achieve incremental model updates.

[0127] Step 5: New data processing and response;

[0128] Abnormality Level Determination: After preprocessing the newly acquired sales data in the same manner as in Step 2, the abnormality level is determined using Model M. Response rules are set: When E(xi) ≥ 0.6, an alarm is automatically triggered and the data is retransmitted; when E(xi) ∈ [0.3, 0.6), the data is marked for manual review. Feedback messages are pushed and an automatic processing log is generated, recording the time of the alarm and the content of the alarm data.

[0129] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. A multi-source data analysis method based on artificial intelligence, characterized by: The method comprises the following steps: Step 1: Obtain structured data and metadata from different data sources and monitor the transmission rate and loss rate; Step 2: Preprocess the metadata and obtain the integrated data set through conversion; Step 3: Extract statistical features, set thresholds to judge abnormal situations, summarize abnormal information to establish an indicator system, and generate abnormality reports; Step 4: Build a decision tree classification model based on multidimensional features, classify according to the anomaly index, and incrementally update the model parameters; Step 5: Use the classification model to determine the anomaly of the new data, set the response rules based on the anomaly classification results, push the feedback message and generate the log; In step 1, the metadata description includes the field name, data type, collection time t, and data volume m; where m is a positive integer representing the number of metadata records; Fields are represented by X j ; Record is represented by x i ; where j = 1, 2, ..., n, is used to distinguish different fields; n is a positive integer, indicating the number of metadata description fields; where i = 1, 2, ..., m, is used to distinguish different records; Monitor the transmission rate v(t) and loss rate L: Use a sliding window to calculate the average transmission rate v bar :v bar =1 / w·Σ w i=1 v(t i ); where w represents the window length; v(t i ) represents the transmission rate corresponding to the acquisition time of the i-th record; Calculate the packet loss rate, the number of lost packets is expressed as N loss , the total number of packets is expressed as N total , the loss rate calculation formula is: L=N loss / N total ; In step 3, extract statistical features: for each standardized field X j 'Calculate the mean μ j , standard deviation σ j 、Maximum value max(X j ') and minimum value min(X j '); get the index set I={μ j ,σ j ,max(X j '),min(X j ')}; Set the threshold to judge abnormality: When the normalized data record x ij 'Not in [μ j -k·σ j ,μ j +k·σ j ] is considered abnormal; where k is a positive integer, representing the preset coefficient; For data transmission anomalies: take the transmission rate v(t) as the signal, use mean filtering or median filtering to determine the mutation point in the time series; calculate the first-order difference Δv(t i )=v(t i )-v(t i-1 ), when |Δv(ti)|>γ, it is considered as a transmission abnormal point; where γ is the set threshold; the transmission abnormal time point set A is obtained trans ; For source data anomalies: Isolation forest is used to calculate the anomaly score s(x i ); the abnormal score is determined based on the distance of the data in the feature space. When s(x i )>δ is judged as abnormal; where δ represents the preset threshold; the source data abnormal record set A is obtained source and the corresponding abnormality level; For data conversion anomalies: Compare the converted data X' with the expected value X* described by the original metadata, and calculate the semantic consistency index S: S=1-||X'-X*|| / ||X*||; when S<λ, it is judged as a data conversion anomaly; where λ is the set minimum alignment; mark and generate the conversion anomaly record set A conv ; Summarize various abnormal scores and records and establish an abnormal index system E={A trans ,A source ,A conv Generate anomaly index E(x i ), the calculation formula is: E(x i )=w1·F(x i ∈A trans )+w2·F(x i ∈A source )+w3·F(x i ∈A conv ); where F is an indicator function, which takes the value 1 when the condition in the brackets is met, and takes the value 0 otherwise; w1, w2, and w3 are the weights of each type of anomaly, 0<w1, w2, w3<1 and w1+w2+w3=1; Generate anomaly reports and classification labels; In step 4, a classification model based on multidimensional features is constructed: using the branch structure of the decision tree, the abnormal index E(x i ) is compared with the preset category threshold τ1, τ2: E(x i )<τ1, the category is normal; τ1≤E(x i )<τ2, the category is warning; E(x i )≥τ2, the category is abnormal; where τ1,τ2 are pre-set category thresholds, τ1<τ2; Establish an anomaly classification model M and generate an anomaly index for each new data; Compare the prediction results of model M with the actual feedback, adjust the parameters and classification thresholds, and realize incremental model updates.

2. The multi-source data analysis method based on artificial intelligence according to claim 1, characterized in that: In step 1, for database data: the staff pre-configures a standard database connection protocol, identifies the data table structure based on the data source type based on the standard database connection protocol, and reads the data; for formatted text files: a file parser is used to automatically parse the file content and identify the data fields in the file; For standard API interfaces: Build API call modules according to the RESTful protocol, automatically initiate requests, and parse the returned JSON / XML data; Get structured data and metadata in a unified format.

3. The multi-source data analysis method based on artificial intelligence according to claim 2, characterized in that: In step 2, for metadata, missing value processing is performed: for field X j The missing values ​​in are filled using the mean filling formula; Get the data matrix after missing value processing and fill log; For the data matrix after missing value processing, the Z-score method is used to detect noise: for each field, the mean and standard deviation are calculated, and the Z-score value is further calculated; when the Z-score value exceeds the preset threshold, it is considered to be abnormal noise data; Get the noise-filtered data matrix and noise anomaly report; Standardize the format of the noise-filtered data matrix: convert the date and time data into a fixed format; the original date is represented by d raw , after conversion, it is recorded as d std :d std =f date (d raw ); where f date Represents the date conversion function corresponding to a fixed format; obtains a data matrix with a unified format; Based on the data matrix with unified format, Min-Max normalization is used to convert the numerical data items to the interval [0,1]; the normalized data matrix is ​​obtained; For the normalized data matrix, data matching and fusion are performed: inner join or outer join is performed using the common field timestamp t; and the integrated data set is generated.

4. The multi-source data analysis method based on artificial intelligence according to claim 3, characterized in that: In step 5, for new data, after preprocessing, it is classified by model M; the response rule is set according to the abnormal classification result: when E(x i )≥τ2, the alarm is automatically triggered and the data is retransmitted; when E(x i )∈[τ1,τ2), it is marked as waiting for manual review; the feedback message is pushed and an automatic processing log is generated.

5. An artificial intelligence-based multi-source data analysis system, applied to the artificial intelligence-based multi-source data analysis method according to any one of claims 1 to 4, characterized in that: The system includes a data access module, a data preprocessing module, an intelligent analysis module, a pattern recognition module and a result output module; The data access module is used to obtain structured data and metadata from different data sources and monitor the transmission rate and loss rate; the data preprocessing module is used to preprocess the metadata and obtain an integrated data set through conversion; the intelligent analysis module is used to extract statistical features, set thresholds to determine abnormal situations, summarize abnormal information to establish an indicator system, and generate abnormal reports; the pattern recognition module is used to construct a decision tree classification model based on multidimensional features, classify according to the abnormal index, and incrementally update the model parameters; the result output module is used to determine the abnormal situation of new data according to the classification model, set response rules according to the abnormal classification results, push feedback messages and generate logs.

6. The artificial intelligence-based multi-source data analysis system according to claim 5, characterized in that: The data access module includes a database unit, a file parsing unit, an API calling unit and a transmission monitoring unit; The database unit is used to connect to the database through the protocol and extract structured data; the file parsing unit is used to parse the file and identify the fields and data content; The API calling unit is used to call the interface according to the RESTful protocol and parse the JSON / XML response; the transmission monitoring unit is used to calculate the transmission rate and loss rate in real time.

7. The artificial intelligence-based multi-source data analysis system according to claim 6, characterized in that: The data preprocessing module includes a missing filling unit, a noise filtering unit, a format standardization unit and a data fusion unit; The missing value filling unit is used to process the missing value of the field using the mean interpolation method and record the filling log; The noise filtering unit is used to identify noise data based on Z-score and generate a noise anomaly report; The format standardization unit is used to unify the date format and normalize the numerical data; the data fusion unit is used to associate multi-source data through timestamps to generate an integrated data set.

8. The artificial intelligence-based multi-source data analysis system according to claim 7, characterized in that: The intelligent analysis module includes a feature extraction unit, an anomaly detection unit and an indicator summary unit; The feature extraction unit is used to calculate the field mean, standard deviation and extreme value; the anomaly detection unit is used to identify three types of anomalies through threshold, isolation forest and difference method; The indicator summary unit is used to construct an abnormal indicator system and generate a weighted comprehensive abnormal index.

9. The artificial intelligence-based multi-source data analysis system according to claim 8, characterized in that: The pattern recognition module includes a parameter updating unit and a model building unit; The parameter updating unit is used to dynamically adjust the classification threshold according to actual feedback; the model building unit is used to divide the abnormality categories based on the decision tree; The result output module includes a classification determination unit, a response rule unit and a log generation unit; The classification determination unit is used to output a new data abnormal classification label; The response rule unit is used to trigger an alarm, retransmission or manual review process; the log generation unit is used to record the abnormal classification results and system response actions.

Images