Multi-source data automatic processing and service integration system based on artificial intelligence driving

Abstract

The invention relates to the technical field of electric digital data processing, and discloses a multi-source data automatic processing and service integration system based on artificial intelligence driving. Comprising the steps that a multi-source data access module achieves concurrent data access and caching, an AI driving data preprocessing module completes field automatic mapping through a data isomerism adaptation algorithm, and an intelligent data fusion module calculates a multi-source data fusion weight through a dynamic weight distribution algorithm; the service integration module realizes standardization of fusion data, and the monitoring scheduling module monitors each module in real time. High-concurrency multi-protocol data access is achieved through multi-source data automatic processing and service integration and a distributed Kafka cluster, automatic data mapping and precise data cleaning are completed through an improved algorithm depending on an AI driving module, the data fusion precision is improved in combination with a dynamic weight algorithm, the service integration flexibility is enhanced through multi-protocol API packaging and load balancing, and the service integration efficiency is improved. And stable system operation and maintenance are guaranteed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electric digital data processing, in particular to a multi-source data automatic processing and service integration system based on artificial intelligence driving, which is suitable for intelligent government affairs, intelligent manufacturing, financial risk control and other scenes requiring unified processing and service of multi-source heterogeneous data. BACKGROUND

[0002] At present, multi-source data processing and service integration has become a core supporting technology in the field of digital economy. Its core goal is to standardize the information of heterogeneous data sources such as Internet of Things devices, relational databases, unstructured files (such as PDF, JSON), third-party APIs, and encapsulate them as reusable services for upper-layer application calling. In the prior art, multi-source data processing relies on traditional ETL tools to complete data extraction and format conversion, and service integration is realized through API gateway or enterprise service bus (ESB).

[0003] The existing technology has the following problems: 1. The degree of automation of multi-source data adaptation is low. The field definition, data type (such as integer type, character type, timestamp), and communication protocol differences of different data sources need to be manually written mapping rules. When the number of data sources exceeds 15, the rule maintenance workload increases by more than 3 times, and the manual configuration error rate is as high as 8%-12%; 2. The data fusion accuracy is limited by static weights. The traditional scheme uses pre-set fixed weights (such as assigning weights according to data source priority) to fuse multi-source data. When the data source quality fluctuates (such as temporary sensor failure leading to data missing), the fusion result error will rise to more than 15%; 3. AI model and service integration are disconnected. In the existing system, AI models such as data classification and anomaly detection need to be deployed independently, and manual development interfaces are needed to realize linkage with data processing modules. The model iteration period is as long as 7-10 days, which cannot adapt to the rapid changes of business requirements. SUMMARY

[0004] The purpose of the present application is to provide a multi-source data automatic processing and service integration system based on artificial intelligence driving to solve the problems raised in the background art.

[0005] To solve the above technical problems, the present application is realized by the following technical scheme: The present application is a multi-source data automatic processing and service integration system based on artificial intelligence driving, which includes a multi-source data access module, an AI-driven data preprocessing module, an intelligent data fusion module, a service integration module, and a monitoring and scheduling module. The multi-source data access module adopts a distributed Kafka cluster, supports HTTP, MQTT, FTP, and JDBC or ODBC protocols, and realizes concurrent data access and caching.

[0006] The AI-driven data preprocessing module completes automatic field mapping through a data heterogeneity adaptation algorithm. The algorithm calculates feature similarity based on an improved cosine similarity, and the formula is: wherein, is a feature vector, is a feature weight, is a similarity.

[0007] The intelligent data fusion module calculates fusion weights through a dynamic weight distribution algorithm, and the formula is: wherein is a weight, is a model accuracy, is an entropy weight, is a balance coefficient.

[0008] The service integration module supports RESTful or gRPC protocol encapsulation API.

[0009] The monitoring and scheduling module realizes early warning and resource scheduling based on the 3 principle.

[0010] Further, the Kafka cluster of the multi-source data access module supports concurrent data access of not less than 2000 per second, and the cache data retention time can be configured to 12-72 hours.

[0011] Further, the data cleaning process in the AI-driven data preprocessing module uses a K-nearest neighbor (K=3-7) interpolation algorithm to fill in missing values, and an isolation forest algorithm (abnormal rate threshold 0.01-0.05) to eliminate abnormal values.

[0012] Further, the feature weight in the data heterogeneity adaptation algorithm is obtained through a random forest model, the model takes the field mapping accuracy as the objective function, and the iteration number is set to 100-200 times.

[0013] Further, the entropy weight in the intelligent data fusion module is calculated by the formula , wherein is the data category appearance probability, is the number of data categories.

[0014] Further, the balance coefficient in the dynamic weight distribution algorithm has a value range of 0.5-0.7, and is determined by grid search optimization.

[0015] Further, the service integration module is built-in API gateway, adopts minimum connection number algorithm to realize load balancing, supports service registration, discovery and interface permission control.

[0016] Further, the monitoring and scheduling module adopts Prometheus+Grafana architecture, and the monitoring indicators include CPU occupancy (threshold value 80%), memory usage (threshold value 85%), and data throughput (threshold value 1000 pieces / second).

[0017] Further, the AI model management module is further included, a federal learning framework is supported, multi-agency data collaborative training is realized, and a Docker+K8s containerization architecture is adopted for model deployment.

[0018] Further, the modules adopt micro-service architecture design, asynchronous communication is realized through RabbitMQ message queue, and independent deployment and horizontal expansion of the modules are supported.

[0019] The application has the following beneficial effects: (1) The application builds a high-concurrency data access channel through a distributed Kafka cluster, supports multiple mainstream protocols such as HTTP, MQTT, FTP, JDBC and ODBC, can realize a concurrent data processing capacity of not less than 2000 pieces per second, effectively solves the protocol adaptation problem of traditional systems in multi-source data access, at the same time, the cache data retention time length supports flexible configuration of 12-72 hours, meets the real-time data processing demand, provides sufficient space for data backtracking and verification, greatly reduces the data access bottleneck caused by scattered data sources and various formats, and lays a foundation for efficient and stable subsequent data processing links.

[0020] (2) The AI-driven data preprocessing module of the application adopts an improved cosine similarity algorithm to complete field automatic mapping, combines a random forest model to train feature weights (iteration is 100-200 times to achieve field mapping accuracy), replaces the cumbersome process of traditional manual mapping, in the data cleaning link, K-nearest neighbor (K=3-7) interpolation algorithm is used to accurately fill in missing values, and an isolation forest algorithm (abnormal rate threshold value 0.01-0.05) is used to efficiently eliminate abnormal values, so that the cost of manual intervention is significantly reduced, and high-quality data support is provided for subsequent data analysis (3) The application adopts a dynamic weight distribution algorithm, accurately fuses model accuracy and entropy weight (based on data category probability calculation), and introduces a balance coefficient of 0.5-0.7 (optimized by grid search), so as to realize scientific distribution of fusion weight, compared with a traditional fixed weight fusion mode, the algorithm can dynamically adjust the weight proportion according to the data characteristics, so that the fusion result is more suitable for the actual business scene, for example, in multi-source monitoring data fusion, the influence of single data source deviation on the result can be significantly reduced, the fusion precision is improved by more than 30%, a more reliable data basis is provided for subsequent business decision, and the application is especially suitable for complex decision-making scenes that need multi-dimensional data support.

[0021] (4) The application supports RESTful and gRPC two mainstream protocol encapsulated API through a service integration module, realizes service registration, discovery and interface permission control in cooperation with a built-in API gateway, and realizes load balancing by using a minimum connection number algorithm. This design enables the system to not only meet the efficient communication between internal modules, but also conveniently connect external business systems, and solves the problems of single interface protocol and high integration difficulty of traditional systems.

[0022] (5) The monitoring and scheduling module of the application constructs a comprehensive monitoring system based on the Prometheus+Grafana architecture, adopts the 3σ principle to realize abnormal early warning for key indicators such as CPU occupancy (threshold value 80%), memory usage (threshold value 85%), and data throughput (threshold value 1000 pieces / second), and when the indicators exceed the threshold value, the system can automatically trigger the resource scheduling mechanism to avoid service interruption caused by insufficient resources. This active monitoring and intelligent scheduling mode shortens the system fault discovery time to the minute level, improves the operation and maintenance response efficiency by more than 50%, reduces the artificial inspection cost, and ensures the stable operation of the system in the high concurrency and large data volume scene.

[0023] (6) The application adopts a micro-service architecture design, each module realizes asynchronous communication through RabbitMQ, supports independent deployment and horizontal expansion, can flexibly increase and decrease node resources according to business needs, and at the same time, the AI model management module supports a federated learning framework, can complete model training without sharing original data in multi-institutional data collaborative training, effectively protects data privacy, cooperates with Docker+K8s containerized deployment, and the system can be quickly migrated and deployed in different hardware environments, and the deployment efficiency is improved by more than 40%. This architecture design not only meets the expansion needs of large distributed systems, but also provides reliable protection for data security in multi-institutional collaboration scenarios, and is suitable for a range of applications from single applications to cross-institutional collaborative platforms.

[0024] Of course, any product implementing the application does not necessarily need to achieve all the advantages described above. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed for the description of the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor based on these drawings.

[0026] Figure 1 The system flowchart of the present application. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings of the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0028] Please refer to Figure 1 The present application is an artificial intelligence driven multi-source data automatic processing and service integration system, which includes a multi-source data access module, an AI driven data preprocessing module, an intelligent data fusion module, a service integration module and a monitoring and scheduling module. The multi-source data access module adopts a distributed Kafka cluster, supports HTTP, MQTT, FTP and JDBC or ODBC protocols, realizes concurrent data access and caching, wherein the Kafka cluster of the multi-source data access module supports concurrent data access of not less than 2000 per second, and the caching data retention time can be configured to 12-72 hours.

[0029] The AI driven data preprocessing module completes field automatic mapping through a data heterogeneity adaptation algorithm, which calculates feature similarity based on improved cosine similarity, and the formula is as follows: ; Wherein, is a feature vector, is a feature weight, is a similarity, wherein the data cleaning process in the AI driven data preprocessing module fills in missing values by using K nearest neighbor interpolation algorithm (K=3-7), and removes outliers by using isolated forest algorithm (abnormal rate threshold 0.01-0.05).

[0030] The intelligent data fusion module calculates the fusion weight by using a dynamic weight distribution algorithm, and the formula is as follows: Wherein is a weight, is a model accuracy, is an entropy weight, is a balance coefficient. The service integration module supports API encapsulation using RESTful or gRPC protocols. The monitoring and scheduling module is based on 3 The principle is to achieve early warning and resource allocation.

[0031] In this implementation scheme, the data heterogeneity adaptation algorithm improves the cosine similarity calculation to achieve automatic field mapping by calculating data feature similarity. Let the feature vector of the data source to be adapted be the i-th feature value (e.g., field length, data type encoding), and the feature importance weight (obtained through training the RandomForest model, with the training objective being field mapping accuracy). Then the similarity calculation formula is: ,when When, the system automatically establishes field mapping; when When manual review is triggered, When this occurs, it is marked as an adaptation failure.

[0032] In addition, the dynamic weight allocation algorithm combines AI model accuracy and data entropy weights to dynamically adjust the fusion weights: assuming system deployment... One AI model (corresponding to) (the first data source), the first The processing accuracy of each model is (Verified using the test set, range) , No. The entropy weight of each data source is (Reflecting data uncertainty), balance coefficient (Value) (determined through grid search optimization), then the first... The fusion weight of each data source The calculation formula is: , ,in, For the first The first data source The probability of occurrence of class data Number of data categories; satisfy This ensures that the fusion results are normalized.

[0033] Based on the above scheme, taking the multi-source data processing scenario of smart government affairs as an example, the implementation process of this system can be divided into the following embodiments.

[0034] Example 1 Data source access: The multi-source data access module connects to three types of government data through public security population data, civil affairs marriage registration data, and social security participation data. Public security population data: Connect MySQL database using JDBC protocol, synchronization period 1 hour; Civil Affairs Marriage Registration Data: Obtain daily CSV files through FTP protocol, synchronization period 24 hours; Social security data: Receive real-time data transmitted by Internet of Things gateway through MQTT protocol, synchronization frequency 1 minute / time; All data is stored in Kafka cluster (3 node deployment, single node memory 16GB) after access, cache retention time is set to 24 hours.

[0035] Example 2 Data preprocessing, data cleaning, feature extraction and field mapping: Data cleaning: Fill in the missing insurance time in social security data using KNN algorithm (K=5), and remove invalid ID numbers (such as bit number ≠ 18 bits) in public security data through isolation forest algorithm (abnormal rate threshold 0.01); Feature extraction: Use CNN model (input dimension 64x64, convolution kernel 3x3) to extract text features (such as marriage status description) in civil affairs CSV file, and use LSTM model (hidden layer dimension 128) to extract trend features of social security time series data; Field mapping: The feature weight (ID number weight 0.9, name weight 0.8, address weight 0.6) is obtained by training the RandomForest model, and the similarity S=0.92 between public security "ID number" and social security "insurance certificate number" is calculated by formula 1. Automatic mapping rule is established.

[0036] Example 3 Data fusion, entropy weight calculation, weight allocation and data fusion: Entropy weight calculation: Public security data class distribution is uniform , formula 2 is substituted to get H1=1.609, civil affairs data class is concentrated , get H2=1.213, social security data class fluctuates greatly , get H3=1.522; Weight allocation: AI model accuracy A1=96% (public security), A2=93% (civil affairs), A3=91% (social security), α=0.6, formula 3 is substituted to get W1=0.35, W2=0.28, W3=0.37; Data fusion: The multi-source data of the same user is weighted and summed according to the weight, such as "household address" field fusion result=0.35x public security address+0.28x civil affairs address+0.37x social security address.

[0037] Example 4 Service integration and monitoring, service packaging, load balancing and monitoring and warning: Service encapsulation: encapsulate the fused "personal government data" as RESTful API, interface path / api / v1 / govern / data, support GET / POST request; Load balancing: API gateway uses the minimum connection number algorithm to distribute requests to 3 service nodes, and single node QPS supports 500; Monitoring and early warning: when the Kafka throughput is less than 1000 per second, the monitoring module triggers the K8s container expansion, increases the Kafka node from 3 to 5, and restores the throughput to more than 2000 per second.

[0038] The preferred embodiments of the application disclosed above are only used to help explain the application. The preferred embodiments do not describe all the details and limit the application to the specific embodiments described. Obviously, many modifications and changes can be made according to the content of the specification. The specification selects and describes these embodiments in order to better explain the principles and practical applications of the application, so that those skilled in the art can well understand and use the application. The application is limited by the claims and their entire scope and equivalents.

Claims

1. A system for automated processing and service integration of multi-source data driven by artificial intelligence, characterized in that, It includes a multi-source data access module, an AI-driven data preprocessing module, an intelligent data fusion module, a service integration module, and a monitoring and scheduling module; The multi-source data access module adopts a distributed Kafka cluster and supports HTTP, MQTT, FTP and JDBC or ODBC protocols to achieve concurrent data access and caching. The AI-driven data preprocessing module automatically maps fields using a data heterogeneity adaptation algorithm. This algorithm calculates feature similarity based on improved cosine similarity, using the following formula: ,in, For feature vectors, For feature weights, Similarity; The intelligent data fusion module calculates the fusion weight using a dynamic weight allocation algorithm, the formula of which is: ,in As weight, For model accuracy, For entropy weight, This is the balance coefficient; The service integration module supports API encapsulation using RESTful or gRPC protocols. The monitoring and scheduling module is based on 3 The principle is to achieve early warning and resource allocation.

2. The system according to claim 1, characterized in that, The Kafka cluster of the multi-source data access module supports concurrent data access of no less than 2,000 records per second, and the cache data retention period can be configured to 12-72 hours.

3. The system according to claim 1, characterized in that, The data cleaning process in the AI-driven data preprocessing module uses the K-nearest neighbor (K=3-7) interpolation algorithm to fill missing values ​​and the isolated forest algorithm (anomaly rate threshold of 0.01-0.05) to remove outliers.

4. The system according to claim 1, characterized in that, Feature weights in the data heterogeneity adaptation algorithm The model was trained using a random forest model, with the field mapping accuracy as the objective function and the number of iterations set to 100-200.

5. The system according to claim 1, characterized in that, The entropy weight in the intelligent data fusion module Through formula Calculation, where For the probability of data categories appearing, This represents the number of data categories.

6. The system according to claim 1, characterized in that, The balance coefficient in the dynamic weight allocation algorithm The value ranges from 0.5 to 0.7 and was determined through optimization using a grid search method.

7. The system according to claim 1, characterized in that, The service integration module has a built-in API gateway, which uses the least connection number algorithm to achieve load balancing and supports service registration, discovery, and interface permission control.

8. The system according to claim 1, characterized in that, The monitoring and scheduling module adopts a Prometheus+Grafana architecture, and the monitoring metrics include CPU utilization (threshold 80%), memory utilization (threshold 85%), and data throughput (threshold 1000 records / second).

9. The system according to claim 1, characterized in that, It also includes an AI model management module, supports federated learning frameworks, enables collaborative training of data from multiple institutions, and the model deployment adopts a Docker+K8s containerized architecture.

10. The system according to claim 1, characterized in that, Each module adopts a microservice architecture design, uses RabbitMQ message queues to achieve asynchronous communication, and supports independent deployment and horizontal scaling of modules.

Images