An intelligent integrated data service device based on

Abstract

The application discloses a kind of based on intelligent integrated data service equipment, the application relates to data processing technical field, the application is by extracting the structured feature of each data source, text and semantic feature, time series feature, behavior feature constitutes feature vector, feature vector is input into trained graph neural network fusion gradient boosting tree GNN-XGBoost Will data source be divided into internal and external data source, the data in internal data source is carried out format compliance check, business logic check, data sensitivity check, risk level determination is carried out to external data source, sensitivity division is carried out to the risk level lower than high, different sensitivity levels use different desensitization method.The application can effectively distinguish data source type, and different processing methods are adopted for different data sources, guarantee data quality and security, improve integration efficiency.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to an intelligent integrated data service device. Background Technology

[0002] As enterprises deepen their digital transformation, the diversity and complexity of internal and external data sources have increased significantly, and traditional data processing technologies have revealed significant limitations in data source management and data governance.

[0003] Existing methods lack a pre-classification mechanism for data sources, failing to establish differentiated processing flows based on the inherent differences between internal and external data sources. This results in a disordered mix of multi-source data during integration. For example, treating standardized internal business data and unstructured external third-party data the same way fails to leverage the format standardization of internal data to improve processing efficiency, and also makes it difficult to accurately control compliance risks of external data. This directly leads to low efficiency in extracting effective data and may even cause business logic conflicts due to misjudgment of data source attributes.

[0004] In addition, traditional data anonymization strategies generally adopt a one-size-fits-all approach, without taking into account the risk level of the data source, the type of sensitive fields, and the usage scenario for fine-grained classification. For low-sensitivity external public data, over-anonymization may be implemented, resulting in a significant decrease in data availability, while for highly sensitive cross-border medical data, simple replacement may be performed, which cannot meet the strong protection requirements for special data.

[0005] Therefore, there is an urgent need for a solution that can intelligently classify data sources and implement differentiated data governance strategies based on the classification results, in order to solve the problems of disorder in data source management and extensive data processing in traditional methods, and improve data integration efficiency. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides an intelligent integrated data service device that solves the problems of disordered data integration and lack of differentiated processing caused by unclassified data sources.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a smart integrated data service device, comprising:

[0008] The data source classification module extracts structural features, text and semantic features, time series features, and behavioral features from each data source to form a feature vector. The feature vector is then input into a trained graph neural network and fused with a gradient boosting tree (GNN-XGBoost) to obtain the data source type.

[0009] The internal data processing module performs format compliance verification, business logic verification, and data sensitivity verification on data from internal data sources.

[0010] The external data processing module assesses the risk level of external data sources based on three aspects: data sensitivity, data source credibility, and usage scenario risk. Data with a risk level below high is classified into different sensitivity levels, and different de-identification methods are used for different sensitivity levels.

[0011] As a further aspect of the present invention, the structured features include authentication methods, data formats, domain name and IP core features, and sensitive field density; the textual and semantic features include domain name compliance, field name compliance, and certificate credibility; and the time series and behavioral features include access frequency, update stability, and periodicity, wherein the access frequency is the average access frequency over the past 7 days, and the update stability is the standard deviation of the data update interval.

[0012] As a further aspect of the present invention, the specific steps for training a graph neural network and fusing a gradient boosting tree (GNN-XGBoost) are as follows:

[0013] Preprocess the specific eigenvalues ​​in the eigenvector;

[0014] Each feature is treated as a graph node. The mutual information value MI between any two features is calculated. Edges with MI > MIth are retained, and the edge weights are normalized to obtain wij, where MIth is the edge weight threshold.

[0015] The preprocessed feature vectors are substituted into the XGBoost classifier to obtain feature importance scores;

[0016] The edge weights of the feature association graph are updated according to the formula wij'=wij×sqrt(Imp(i)×Imp(j)), where sqrt() is the square root function and imp(i) is the XGBoost importance of feature i.

[0017] The output of GNN and XGBoost are concatenated and then passed through a fully connected layer to output the classification probability.

[0018] Every 50 training rounds, the feature mutual information of the current training set is recalculated, and new edge weights are generated.

[0019] The dataset was divided into 5 subsets, and each time a 4-fold training and 1-fold validation were performed. The average F1-Score was calculated.

[0020] Using Bayesian optimization, with the validation set F1-Score as the target, the optimal parameter combination is found through 50 iterations.

[0021] Select the optimal model with three different hyperparameter configurations, obtain the average predicted probability of each model, and take the type corresponding to the largest average as the output result.

[0022] As a further embodiment of the present invention, the initial hyperparameters of GNN include a learning rate of 0.001, a dropout rate of 0.3, and a weight decay of 0.0001. The initial hyperparameters of XGBoost include a tree count of 100, a learning rate of 0.1, a maximum depth of 5, a subsampling rate of 1, a column sampling rate of 1, and a minimum number of sample splits of 20.

[0023] As a further aspect of this invention, the format compliance verification directly calls the enterprise's existing business dictionary and data lineage for verification; the business logic verification combines a real-time business rule engine to perform dynamic logical verification on numerical data; and the data sensitivity verification is based on the enterprise's internal data classification and grading standards, matching sensitive data through preset tags, and only de-identifying data shared across departments. Data between internal core systems is transmitted through the enterprise's private cloud encrypted channel without de-identification.

[0024] As a further aspect of the present invention, the specific steps for classifying the risk levels of external data sources are as follows:

[0025] Construct a 3×3 contribution matrix M, where row 1 represents the data sensitivity, row 2 represents the data source credibility, row 3 represents the use case risk, column 1 represents low risk, column 2 represents medium risk, column 3 represents high risk, and the element value mij represents the contribution weight of dimension i to level j. mij is set based on domain expert knowledge and risk transmission mechanism.

[0026] The original scores of the three dimensions are standardized to the [0,1] interval using the Sigmoid function according to the formula 1 / (1+exp(-k*(x-xth))), where x is the original score, k is the slope, and xth is the threshold parameter.

[0027] Multiply the standardized three-dimensional scoring vector S=[s1,s2,s3] with matrix M to obtain the risk level score vector R. Select the risk level corresponding to max(R) as the final risk level.

[0028] As a further aspect of this invention, data sources with high risk levels are directly intercepted. At the same time, external data fields are compared with existing regulatory requirements, and access is refused if key compliance information is missing.

[0029] As a further aspect of the present invention, the specific rules for classifying sensitive data from external data sources into different levels are as follows:

[0030] Establish a sensitive field type library, classify it into three levels according to the degree of harm, and assign a fixed base score to each type: 5 for high sensitivity, 3 for medium sensitivity, and 1 for weak sensitivity.

[0031] Iterate through all fields of the data source, combine real-time context analysis, use the BERT pre-trained model to perform semantic fuzzy matching, record the sensitivity type of each field, and if there is no match, it is non-sensitive;

[0032] Calculate the total number of sensitive fields C and the total number of data fields N, and count them separately according to their sensitivity levels;

[0033] Calculate the total score for each layer: Sstrong = number of strongly sensitive fields × 5, Smedium = number of medium sensitive fields × 3, Sweak = number of weakly sensitive fields × 1;

[0034] Calculate the sensitivity score = (10*(S strong + S medium + S weak)) / 5*N, and keep the sensitivity score between 0 and 10.

[0035] Data sources with scores of 0-3 are classified as low-sensitivity data sources, those with scores of 4-7 as medium-sensitivity data sources, and those with scores of 8-10 as high-sensitivity data sources.

[0036] As a further aspect of this invention, for low-sensitivity data sources, mild desensitization is employed, specifically by replacing the values ​​of clearly sensitive fields while retaining the complete information of non-sensitive fields; for medium-sensitivity data sources, moderate desensitization is employed, specifically by converting precise values ​​into fuzzy ranges and deleting or replacing indirect identification information; for high-sensitivity data sources, severe desensitization is employed, specifically by adding random noise to numerical data, directly removing high-sensitivity fields, and combining and obfuscating the remaining fields.

[0037] As a further aspect of the present invention, non-sensitive fields missing in external data sources are synthesized and filled in using internal similar data through a generative adversarial network (GAN).

[0038] This invention provides an intelligent integrated data service device, which has the following advantages compared with the prior art:

[0039] (1) This invention achieves differentiated governance of internal and external data sources through intelligent classification, extracts features from the structured metadata, text semantics and time series behavior, and combines the GNN-XGBoost model to integrate feature association and nonlinear modeling capabilities to improve classification accuracy. The internal data source relies on the business dictionary and data lineage to realize format compliance verification. The logical relationship is dynamically verified through the real-time rule engine to avoid the lag of traditional verification, and improve integration efficiency while ensuring data quality.

[0040] (2) This invention constructs a three-dimensional contribution matrix to determine the risk level of external data sources, combines BERT semantic recognition and sensitive field classification system to implement mild, moderate and severe differential desensitization, and uses GAN technology to fill in missing fields based on internal data features, thereby improving the compliance and availability of external data access. Attached Figure Description

[0041] Figure 1 This is a system framework diagram of the present invention. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] like Figure 1 This invention provides a smart integrated data service device, comprising:

[0044] The data source classification module extracts structural features, text and semantic features, time series features, and behavioral features from each data source to form a feature vector. The feature vector is then input into a trained graph neural network and fused with a gradient boosting tree (GNN-XGBoost) to obtain the data source type.

[0045] Structured features include authentication methods, data formats, core domain and IP characteristics, and density of sensitive fields;

[0046] The authentication method is a direct reflection of the legitimacy of the data source identity. For example, data sources that use standard protocols such as OAuth 2.0 and SAML are usually enterprise-level systems with a sound identity management system. Data sources without authentication or that only use simple tokens may come from third-party APIs or edge devices and are at risk of identity forgery.

[0047] Data format affects parsing security and compliance. For example, standard formats facilitate standardized matching of rules for identifying sensitive fields, such as JSON and XML. Custom binary or encrypted formats may hide undeclared sensitive data, such as Base64 encoded fields in logs.

[0048] The authentication method and data format are tagged with symbols. For example, for the authentication method, Kerberos is represented by 0, OAuth2.0 by 1, and API_Key by 2; for the data format, Parquet is represented by 0, JSON by 1, and CSV by 2.

[0049] Domain names and IP addresses are the core carriers of the physical identity of data sources. For example, the ownership of a domain name and the IP range can be directly associated with corporate assets. Overseas IP addresses and dynamic domain names may point to third-party cloud services or suspicious nodes.

[0050] The core characteristics of domain names and IP addresses use binary characteristics. For example, a domain name is represented by 1 if it is a primary domain name and by 0 if it is not. Similarly, an IP address is represented by 1 if it is an internal network IP address and by 0 if it is not.

[0051] The proportion of sensitive fields directly reflects the risk of data content. For example, external illegal data sources may mix sensitive and non-sensitive fields by splicing, forging, etc., and their density distribution may not be consistent with normal business scenarios.

[0052] Textual and semantic features include domain name compliance, field name compliance, and certificate credibility;

[0053] Domain name naming rules reflect the business attributes and compliance of the data source. For example, domain names containing "pay" or "bank" must comply with the compliance requirements of the payment industry, while domain names that illegally use trademarked words may be counterfeit sources.

[0054] Domain compliance is the domain name's enterprise terminology matching score, output by the BERT model, ranging from 0 to 1.

[0055] Field names are semantic tags for data sources. Their compliance directly reflects the qualifications of the data producer and the legality of the processing procedures. For example, field names of legitimate data sources need to pass compliance review and follow unified standards. Illegal data sources often have chaotic field names or conflicting naming rules due to the splicing of data from multiple sources.

[0056] Field name compliance is the field name compliance score, which determines whether it conforms to the enterprise data dictionary. Compliance is 1, and non-compliance is 0.

[0057] Digital certificates are the network identity cards of data sources. Their credibility directly reflects the authenticity of the data source's identity and the security of communication. For example, internal enterprise data sources may use certificates issued by internal CAs, while public services use public CA certificates.

[0058] Certificate credibility is the credibility score of the certificate issuer. A self-built CA by an enterprise receives 1, a public CA receives 0.5, and an unknown CA receives 0.

[0059] Time series and behavioral characteristics include access frequency, update stability, and periodicity;

[0060] Access frequency reflects the business attributes of the data source. For example, the access frequency of an internal data source can be flexibly adjusted according to business needs and is not limited by the external rate, while the access frequency of an external data source is limited by the call quota or protocol agreement of the external interface.

[0061] The access frequency is the average access frequency over the past 7 days, which, after standardization, reflects the activity level of the data source.

[0062] The regularity of data updates reflects the reliability of the data source. For example, for internal data sources, the data generation mechanism is completely controlled by the enterprise, and the update stability is high, with very few interruptions or abnormal fluctuations. However, external data sources rely on the system stability, operation and maintenance capabilities, or cooperative relationships of external parties, and may experience update interruptions or sudden changes in data volume due to external failures or policy changes.

[0063] Update stability is the standard deviation of the data update interval. After standardization, the smaller the standard deviation, the more stable the update.

[0064] Business activities typically have cyclical patterns. For example, the cyclical nature of internal data sources is usually strongly tied to the company's internal business processes, and the cyclical rules are entirely self-defined. In contrast, the cyclical nature of external data sources is determined by external parties, and companies can only passively adapt to it.

[0065] If a periodicity exists, it is recorded as 1; if it does not exist, it is recorded as 0. Internal data sources often have periodicity during working periods, while external ones usually do not.

[0066] For example, the specific parameters of a data source are as follows: the domain name is api.logistics.com, the IP is 52.198.32.12, the authentication method is OAuth2.0, the data format is JSON, and the field examples are tracking_number (logistics tracking number), delivery_status (delivery status), and recipient_email (recipient email, a sensitive field);

[0067] Parse the authentication request headers of the data source, such as the Authorization field in the HTTP request header. If Authorization: Bearer (OAuth 2.0 token) is detected, mark the authentication method _OAuth2.0=1. Read the data file extension or the Content-Type field in the API response header, marking the data format encoding as 1. Resolve the domain name; the domain is logistics.com, marked as 0. Match the IP address range to the enterprise intranet range; the IP is 52.198.32.12, marked as 0. Analyze the proportion of sensitive fields in the statistics fields. The fields tracking_number and delivery_status are non-sensitive, while recipient_email is sensitive, with a sensitive field density of 1 / 3 = 0.33. Use a pre-trained BERT model to determine if the domain name contains enterprise terms. If the domain name does not contain enterprise terms, the score is 0.2. Compare the field names to the enterprise data dictionary; tracking_number does not conform to the enterprise dictionary. Using `LogisticsNo`, the score is recorded as 0.3. Parsing the issuer field of the SSL certificate, the issuer's trustworthiness score is marked as 0.5. The number of API calls within the past 7 days is counted, standardized, and equal to 2. The standard deviation of the time interval between each data update is calculated to reflect stability. Since the update time is not fixed, the standard deviation is 3 hours, which becomes 1.5 after standardization. The Autocorrelation Function (ACF) is used to detect whether the access time is concentrated during working hours. Since the access time is random, periodicity is marked as 0. The resulting feature vector is (1,1,0,0,0.33,0.2,0.3,0.5,2,1.5,0).

[0068] Data source type determination requires simultaneous processing of structured features, text semantic features, and time series features. Traditional models struggle to model different types of features simultaneously, while the combined architecture of GNN-XGBoost can achieve this. GNN excels at processing structured relational data and can model data sources and their interactions as graph structures. XGBoost performs well on tabular data and mixed features, making it suitable for integrating graph features extracted by GNN with other handcrafted features.

[0069] The specific steps for training a graph neural network and fusing a gradient boosting tree (GNN-XGBoost) are as follows:

[0070] The specific feature values ​​in the feature vector are preprocessed. For numerical features, such as the density of sensitive fields, Min-Max normalization is used to [0,1]. For standardized features, such as the mean of access frequency, Z-score standardization is used. Other features remain unchanged.

[0071] Each feature is treated as a graph node. The mutual information value MI between any two features is calculated. Only edges where MI > MIth are retained, and the edge weights are normalized.

[0072] Substitute the preprocessed feature vector into the XGBoost classifier to obtain the feature importance score, and update the edge weights of the feature association graph to the MI value × the geometric mean of the feature importance at both ends. The specific formula is: wij'=wij×sqrt(Imp(i)×Imp(j)), where imp(i) is the XGBoost importance of feature i (normalized to [0,1]), which strengthens the feature association related to the classification task.

[0073] While capturing statistical correlations between features through mutual information, the importance of features to the classification task is not considered. Feature importance scores are obtained through XGBoost training to clarify the contribution of each feature to the classification target, guide the GNN to prioritize key features, and inject the supervision signal of the classification task into the initial structure of the GNN to accelerate convergence and improve accuracy.

[0074] For example, the MI calculated between the authentication method_Kerberos and whether it is an internal network IP is 0.6, and the XGBoost importance is 0.25, so the initial edge weight of GNN is 0.6 × 0.25 = 0.15. The MI calculated between the data format encoding and the standard deviation of the update interval is 0.8, and the XGBoost importance is 0.02, so the initial edge weight of GNN is 0.8 × 0.02 = 0.016. Although the statistical correlation between the data format encoding and the standard deviation of the update interval is high, the initial edge weight of GNN is weakened because the classification importance is low. On the other hand, the combination of authentication method_Kerberos and whether it is an internal network IP has a low statistical correlation but is important in classification, so the edge weight is strengthened.

[0075] The GNN layer uses a 2-layer graph attention network GAT with 4 attention heads per layer, outputting a 64-dimensional embedding vector to capture high-order correlations between features. The XGBoost layer retains the tree model's ability to model non-linear relationships, concatenates the GNN output with the XGBoost output, and outputs the classification probability through a fully connected layer.

[0076] The initial hyperparameters of GNN include a learning rate of 0.001, a dropout rate of 0.3, and a weight decay of 0.0001. The initial hyperparameters of XGBoost include a tree count of 100, a learning rate of 0.1, a maximum depth of 5, a subsampling rate of 1, a column sampling rate of 1, and a minimum number of splits of 20.

[0077] Every 50 training rounds, the feature mutual information of the current training set is recalculated, and new edge weights are generated.

[0078] The dataset is divided into 5 subsets. Each time, 4-fold training and 1-fold validation are performed. The average F1-Score is calculated to evaluate the stability of the model. Bayesian optimization is used to find the optimal parameter combination by iterating 50 times with the validation set F1-Score as the target.

[0079] Three optimal models with different hyperparameter configurations are selected, and the prediction probabilities of each model are combined to obtain the final discrimination result. For example, if the output probabilities of the three models regarding the internal data source are [0.9, 0.85, 0.92], and the average is greater than 0.5, it is determined to be an internal data source.

[0080] The internal data processing module performs format compliance verification, business logic verification, and data sensitivity verification on data from internal data sources.

[0081] The format compliance verification skips the traditional format comparison process and directly calls the enterprise's existing business dictionary and data lineage for verification, significantly improving detection efficiency;

[0082] For example, an e-commerce company's business dictionary specifies that the customer ID format is C + 8 digits. When integrating user management system data, the system automatically verifies the field format. For example, C20250515 passes the verification, while CUS2025 triggers an exception callback. For the business rule that order amount = product unit price × quantity, the system automatically associates data lineage. If an order amount is 1000 yuan, but the unit price is 200 yuan × quantity 3 = 600 yuan, the system does not directly intercept it, but triggers the business system to recalculate through the message queue to avoid blocking the data flow.

[0083] Business logic validation, combined with a real-time business rule engine, performs dynamic logical validation on numerical data, replacing the traditional comparison of historical numerical ranges.

[0084] For example, if a supply chain system has a rule that the inventory quantity is ≥0, when the received inventory data is -5, the system will automatically mark it as abnormal and trigger the inventory count process; if the system rule is set to discount rate ≤50%, but the promotional activity data shows a discount rate of 60%, the system will automatically link the promotional strategy table for secondary verification to avoid profit loss due to data errors.

[0085] Data sensitivity verification is based on the company's internal data classification and grading standards to achieve differentiated desensitization strategies;

[0086] Sensitive data, such as financial data and employee information, can be matched by preset tags without the need for keyword scanning. For example, employee tables exported from the human resources system are automatically associated with sensitive level tags.

[0087] Only data shared across departments is anonymized. For example, when the HR department shares employee data with the sales department, the system automatically hides fields such as salary and attendance. When the finance system transmits cost data to the production system, the unit cost field is kept to two decimal places and ±5% random noise is added.

[0088] Data between internal core systems is transmitted through an encrypted channel in the enterprise's private cloud without any data anonymization, ensuring data integrity. For example, production plan data is transmitted directly after being encrypted using the AES-256 algorithm, and the recipient decrypts it using a digital certificate.

[0089] The external data processing module assesses the risk level of external data sources based on three aspects: data sensitivity, data source credibility, and usage scenario risk.

[0090] The specific steps for classifying the risk level of external data sources are as follows:

[0091] Construct a 3×3 contribution matrix M, where row i corresponds to three risk dimensions: row 1 represents data sensitivity, row 2 represents data source credibility, and row 3 represents use case risk. Column j corresponds to three risk levels: column 1 represents low risk, column 2 represents medium risk, and column 3 represents high risk. The element value mij represents the contribution weight of dimension i to level j, and mij is set based on domain expert knowledge and risk transmission mechanism.

[0092] According to the formula 1 / (1+exp(-k*(x-xth))), the original scores of the three dimensions are standardized to the [0,1] interval by the Sigmoid function, where x is the original score, usually taken as 1-5, k is the slope, which is used to enhance the non-linearity of the entire function, usually taken as 2, and xth is the threshold parameter, usually taken as 3.5, so that the scores not less than 3.5 points quickly approach 1;

[0093] Multiply the standardized three-dimensional scoring vector S=[s1,s2,s3] with matrix M to obtain the risk level score vector R, and select the risk level corresponding to max(R) as the final risk level;

[0094] Data sources with high risk levels are directly blocked. At the same time, external data fields are compared with the requirements of regulations such as the Data Security Law and GDPR. For example, EU data sources must include a GDPR compliance declaration field, and access is refused if key compliance information is missing.

[0095] For data sources with low risk levels, real-time context analysis is combined. For example, when the external data source comes from the medical industry, the recognition weight of fields such as medical records and diagnosis results is automatically enhanced. The BERT pre-trained model is used for semantic fuzzy matching to solve the problem that keyword comparison cannot identify synonyms. For example, ID card, identity card, and ID number are all identified as sensitive fields.

[0096] At the same time, it is also necessary to classify the sensitive data of external data sources into different levels. The specific classification rules are as follows: Establish a sensitive field type library and divide it into three levels according to the degree of harm. Each type is assigned a fixed base score. The base score for high sensitivity is 5, such as ID card number, bank card number, and medical record. The base score for medium sensitivity is 3, such as email address, mobile phone number, and home address. The base score for weak sensitivity is 1, such as device ID, order number, and username.

[0097] Iterate through all fields in the data source and record the sensitivity type of each field. If no match is found, the field is considered non-sensitive.

[0098] Calculate the total number of sensitive fields C and the total number of data fields N, and count them separately according to their sensitivity levels;

[0099] Calculate the total score for each level: Sstrong = number of strongly sensitive fields × 5, Smedium = number of moderately sensitive fields × 3, Sweak = number of weakly sensitive fields × 1. Combine the proportion of sensitive fields to avoid the impact of data volume differences on the score.

[0100] Calculate the sensitivity score = (10*(S strong + S medium + S weak)) / 5*N, and keep the sensitivity score between 0 and 10.

[0101] Data sources with scores of 0-3 are classified as low-sensitivity data sources, those with scores of 4-7 are classified as medium-sensitivity data sources, and those with scores of 8-10 are classified as high-sensitivity data sources.

[0102] For low-sensitivity data sources, since the data has already undergone preliminary desensitization or anonymization, the proportion of sensitive fields is low and it does not involve core privacy or business data, so mild desensitization is adopted. The specific approach is as follows: replace the values ​​of clearly sensitive fields, for example, replace "Zhang San" with "Zhang ×", and retain the complete information of non-sensitive fields, such as age and region, which can be used directly.

[0103] For sensitive data sources containing some sensitive information, such as partners' basic customer information (including names and emails but no ID numbers), where data needs to be shared and collaborated across organizations but a complete trust relationship has not yet been established (e.g., data exchange between upstream and downstream of the supply chain), a moderate level of anonymization is adopted. The specific methods are as follows: convert precise values ​​into fuzzy ranges, for example, convert the specific time "1990-01-01" into "1990s", and the specific location "Chaoyang District, Beijing" into "Beijing". Delete or replace indirect identifying information, such as retaining the email prefix but deleting the suffix "@company.com", or hashing the "customer number".

[0104] For highly sensitive data sources, which contain highly sensitive information such as ID card numbers, medical records, and financial accounts, the data source has low credibility, such as third parties that are accessing the data for the first time, APIs without clear compliance statements, or cross-border transmissions. These data sources need to meet the privacy regulations of different countries and adopt heavy anonymization. The specific approach is as follows: add random noise to the numerical data, such as converting "income of 5,000 yuan" to "5,000 ± 200 yuan", to ensure that the modification of a single data point does not affect the overall statistical results.

[0105] Remove highly sensitive fields directly, such as deleting the entire "ID number" or "bank card number" field, instead of replacing the value;

[0106] Combine and obfuscate the remaining fields, such as combining the hash of "name + mobile number" and converting it into the meaningless string "a1b2c3";

[0107] For non-sensitive fields missing in external data, the missing data is filled in using internal similar data synthesized by a generative adversarial network (GAN). For example, when the external logistics data is missing "delivery area", the missing data is filled in by matching the internal region dictionary based on address keywords.

[0108] Some of the data in the above formulas are numerical calculations with dimensions removed, and the contents not described in detail in this specification are all prior art known to those skilled in the art.

[0109] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A device based on intelligent integrated data services, characterized in that, include: The data source classification module extracts structural features, text and semantic features, time series features, and behavioral features from each data source to form a feature vector. The feature vector is then input into a trained graph neural network and fused with a gradient boosting tree (GNN-XGBoost) to obtain the data source type. The internal data processing module performs format compliance verification, business logic verification, and data sensitivity verification on data from internal data sources. The external data processing module assesses the risk level of external data sources based on three aspects: data sensitivity, data source credibility, and usage scenario risk. Data with a risk level below high is classified into different sensitivity levels, and different de-identification methods are used for different sensitivity levels.

2. The intelligent integrated data service device according to claim 1, characterized in that, Structured features include authentication methods, data formats, core domain and IP features, and density of sensitive fields. Textual and semantic features include domain compliance, field name compliance, and certificate credibility. Time series and behavioral features include access frequency, update stability, and periodicity. The access frequency is the average access frequency over the past 7 days, and the update stability is the standard deviation of the data update interval.

3. The intelligent integrated data service device according to claim 1, characterized in that, The specific steps for training a graph neural network and fusing a gradient boosting tree (GNN-XGBoost) are as follows: Preprocess the specific eigenvalues ​​in the eigenvector; Each feature is treated as a graph node. The mutual information value MI between any two features is calculated. Edges with MI > MIth are retained, and the edge weights are normalized to obtain wij, where MIth is the edge weight threshold. The preprocessed feature vectors are substituted into the XGBoost classifier to obtain feature importance scores; The edge weights of the feature association graph are updated according to the formula wij'=wij×sqrt(Imp(i)×Imp(j)), where sqrt() is the square root function and imp(i) is the XGBoost importance of feature i. The output of GNN and XGBoost are concatenated and then passed through a fully connected layer to output the classification probability. Every 50 training rounds, the feature mutual information of the current training set is recalculated, and new edge weights are generated. The dataset was divided into 5 subsets, and each time a 4-fold training and 1-fold validation were performed. The average F1-Score was calculated. Using Bayesian optimization, with the validation set F1-Score as the target, the optimal parameter combination is found through 50 iterations. Select the optimal model with three different hyperparameter configurations, obtain the average predicted probability of each model, and take the type corresponding to the largest average as the output result.

4. The intelligent integrated data service device according to claim 1, characterized in that, The initial hyperparameters of GNN include a learning rate of 0.001, a dropout rate of 0.3, and a weight decay of 0.0001. The initial hyperparameters of XGBoost include a tree count of 100, a learning rate of 0.1, a maximum depth of 5, a subsampling rate of 1, a column sampling rate of 1, and a minimum number of splits of 20.

5. The intelligent integrated data service device according to claim 1, characterized in that, Format compliance verification directly calls the company's existing business dictionary and data lineage for verification. Business logic verification combines a real-time business rule engine to perform dynamic logical verification on numerical data. Data sensitivity verification is based on the company's internal data classification and grading standards, and sensitive data is matched by preset tags. Only data shared across departments is anonymized. Data between internal core systems is transmitted through the company's private cloud encrypted channel without anonymization.

6. The intelligent integrated data service device according to claim 1, characterized in that, The specific steps for classifying the risk level of external data sources are as follows: Construct a 3×3 contribution matrix M, where row 1 represents the data sensitivity, row 2 represents the data source credibility, row 3 represents the use case risk, column 1 represents low risk, column 2 represents medium risk, column 3 represents high risk, and the element value mij represents the contribution weight of dimension i to level j. mij is set based on domain expert knowledge and risk transmission mechanism. The original scores of the three dimensions are standardized to the [0,1] interval using the Sigmoid function according to the formula 1 / (1+exp(-k*(x-xth))), where x is the original score, k is the slope, and xth is the threshold parameter. Multiply the standardized three-dimensional scoring vector S=[s1,s2,s3] with matrix M to obtain the risk level score vector R. Select the risk level corresponding to max(R) as the final risk level.

7. The intelligent integrated data service device according to claim 1, characterized in that, Data sources with high risk levels will be directly blocked. At the same time, external data fields will be compared with existing regulatory requirements, and access will be refused if key compliance information is missing.

8. The intelligent integrated data service device according to claim 1, characterized in that, The specific rules for classifying sensitive data from external data sources are as follows: Establish a sensitive field type library, classify it into three levels according to the degree of harm, and assign a fixed base score to each type: 5 for high sensitivity, 3 for medium sensitivity, and 1 for weak sensitivity. Iterate through all fields of the data source, combine real-time context analysis, use the BERT pre-trained model to perform semantic fuzzy matching, record the sensitivity type of each field, and if there is no match, it is non-sensitive; Calculate the total number of sensitive fields C and the total number of data fields N, and count them separately according to their sensitivity levels; Calculate the total score for each layer: Sstrong = number of strongly sensitive fields × 5, Smedium = number of medium sensitive fields × 3, Sweak = number of weakly sensitive fields × 1; Calculate the sensitivity score = (10*(S strong + S medium + S weak)) / 5*N, and keep the sensitivity score between 0 and 10. Data sources with scores of 0-3 are classified as low-sensitivity data sources, those with scores of 4-7 as medium-sensitivity data sources, and those with scores of 8-10 as high-sensitivity data sources.

9. A smart integrated data service device according to claim 1, characterized in that, For low-sensitivity data sources, mild desensitization is adopted. Specifically, this includes replacing the values ​​of clearly sensitive fields while retaining the complete information of non-sensitive fields. For moderately sensitive data sources, moderate desensitization is adopted, which includes converting precise values ​​into fuzzy ranges and deleting or replacing indirect identification information. For highly sensitive data sources, heavy desensitization is adopted, which includes adding random noise to numerical data, directly removing highly sensitive fields, and combining and obfuscating the remaining fields.

10. A smart integrated data service device according to claim 1, characterized in that, For non-sensitive fields missing in external data sources, they are synthesized and filled in using internal similar data through a generative adversarial network (GAN).

Images