Multi-source data processing method and device, equipment and storage medium

Abstract

The invention discloses a multi-source data processing method, and relates to the technical field of data processing. When the method is executed, multi-source data to be processed, including real-time data and non-real-time data, is obtained firstly; performing data preprocessing on the multi-source data according to data types; processing the data subjected to data preprocessing by adopting a corresponding preset optimization strategy based on the data type; and finally, storing the optimized data. Therefore, the data is preprocessed according to the data types, the data can be preprocessed according to the characteristics and requirements of different data, the data quality is effectively improved, and a solid foundation is laid for subsequent processing; corresponding preset optimization strategies are adopted for processing based on the data types, the advantages of different strategies in different data scenes can be fully played, and therefore the value of various types of data is fully mined; and finally, the optimized data is stored, and high-quality and structured data resources are provided for subsequent data analysis, application and the like.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a multi-source data processing method, apparatus, device, and storage medium. Background Art

[0002] With the continuous development of information technology, data has become a core asset for enterprises and organizations, and the collection and use of data has received much attention from enterprises. To effectively manage and utilize data resources, it is necessary to integrate, clean, and process data from different sources.

[0003] Existing multi-source data fusion technology systems and applications based on big data mining leverage big data to improve information resource utilization, provide more accurate information for decision-making, and promote social development and innovation. However, when processing large amounts of raw data, these solutions must consider system stability, scalability, and maintainability. While solely employing a single algorithm or model may be effective in specific scenarios, it struggles to balance multiple aspects of system performance, lacks flexibility, and struggles to adapt to changing data characteristics and application scenarios.

[0004] Therefore, there is an urgent need for a multi-source data processing technology to meet the challenges of massive multi-source data. Summary of the Invention

[0005] To address the above issues, this application provides a multi-source data processing method, which includes the following contents:

[0006] In a first aspect, the present application provides a multi-source data processing method, the method comprising:

[0007] Acquire multi-source data to be processed, wherein the multi-source data includes real-time data and non-real-time data;

[0008] Performing data preprocessing on the multi-source data according to data types;

[0009] For the data that has been pre-processed, the corresponding preset optimization strategy is used to process it based on the data type;

[0010] The optimized data is stored.

[0011] Optionally, obtaining the multi-source data to be processed includes:

[0012] Non-real-time data collection based on database or interface;

[0013] Based on the real-time stream data processing engine, it receives real-time data from business systems and sensor channels.

[0014] Optionally, the performing data preprocessing on the multi-source data according to data types includes:

[0015] Use pre-trained data processing models to clean and standardize invalid, duplicate records, missing values, and outliers in real-time data according to preset real-time data processing strategies;

[0016] Use pre-trained data processing models to clean and standardize invalid, duplicate records, missing values ​​and outliers in non-real-time data according to preset non-real-time data processing strategies.

[0017] Optionally, the process of processing the real-time data based on the optimization strategy corresponding to the real-time data includes:

[0018] First data is obtained by performing hybrid computing based on static data in a data warehouse and preprocessed real-time data; the static data includes fact tables and dimension tables stored in the data warehouse, the fact tables storing quantitative data related to business processes, and the dimension tables storing descriptive information. The hybrid computing supports the definition of external tables to achieve hybrid computing of offline data and real-time streaming programs, and employs refined processing of incremental tables, which contain new or changed data over a period of time, and is processed using technologies such as timestamps and log serialization.

[0019] Optimizing management and resource allocation of the first data using a preset reinforcement learning method to obtain second data; the reinforcement learning method dynamically adjusts the real-time data pipeline based on the amount, structure, and importance of the input data to optimize data flow and resource allocation;

[0020] Performing online calculations on the real-time data in the second data using stream processing technology, aggregating the real-time data in the incremental table to generate statistical indicators; and building a data analysis model based on the statistical indicators and historical data.

[0021] The graph neural network model and the data analysis model are used to perform dynamic analysis and anomaly detection of graph structures.

[0022] Optionally, the non-real-time data includes structured data, semi-structured data, and unstructured data, and the process of processing the non-real-time data based on the optimization strategy corresponding to the non-real-time data includes:

[0023] Use SQL queries or data extraction tools to unify the structure of structured data;

[0024] Process semi-structured data into data dictionaries or tables and standardize label naming;

[0025] Perform feature extraction on unstructured data and convert the extracted features into TF-IDF or visual feature vectors.

[0026] Optional, real-time data storage includes:

[0027] The data obtained after processing the real-time data based on the optimization strategy corresponding to the real-time data is integrated with the fact table and dimension table information in the data warehouse to construct a wide table; by extracting data related to business activity metrics from the fact table, using the foreign key relationship between the fact table and the dimension table, the descriptive attribute information in the dimension table is added to the fact table to form a wide table containing the attributes to be analyzed, and the wide table is persistently stored to provide data support for subsequent data analysis and report generation;

[0028] Use data visualization tools to display real-time analysis results and enhance data comprehensibility and analytical capabilities.

[0029] Optional, non-real-time data storage includes:

[0030] The non-real-time data is stored separately according to structured, semi-structured and unstructured data based on the Hadoop framework.

[0031] In a second aspect, the present application provides a multi-source data processing device, the device comprising:

[0032] An acquisition unit, configured to acquire multi-source data to be processed, wherein the multi-source data includes real-time data and non-real-time data;

[0033] A first processing unit is configured to perform data preprocessing on the multi-source data according to data types;

[0034] The second processing unit is used to process the data after data preprocessing by adopting a corresponding optimization strategy based on the data type;

[0035] The storage unit is used to store the data after optimization processing.

[0036] Optionally, the acquiring unit acquiring the multi-source data to be processed includes:

[0037] Non-real-time data collection based on database or interface;

[0038] Based on the real-time stream data processing engine, it receives real-time data from business systems and sensor channels.

[0039] Optionally, the first processing unit preprocessing the multi-source data according to data types includes:

[0040] Use pre-trained data processing models to clean and standardize invalid, duplicate records, missing values, and outliers in real-time data according to preset real-time data processing strategies;

[0041] Use pre-trained data processing models to clean and standardize invalid, duplicate records, missing values ​​and outliers in non-real-time data according to preset non-real-time data processing strategies.

[0042] Optionally, the second processing unit processes the real-time data based on the optimization strategy corresponding to the real-time data, including:

[0043] First data is obtained by performing hybrid computing based on static data in a data warehouse and preprocessed real-time data; the static data includes fact tables and dimension tables stored in the data warehouse, the fact tables storing quantitative data related to business processes, and the dimension tables storing descriptive information. The hybrid computing supports the definition of external tables to achieve hybrid computing of offline data and real-time streaming programs, and employs refined processing of incremental tables, which contain new or changed data over a period of time, and is processed using technologies such as timestamps and log serialization.

[0044] Optimizing management and resource allocation of the first data using a preset reinforcement learning method to obtain second data; the reinforcement learning method dynamically adjusts the real-time data pipeline based on the amount, structure, and importance of the input data to optimize data flow and resource allocation;

[0045] Performing online calculations on the real-time data in the second data using stream processing technology, aggregating the real-time data in the incremental table to generate statistical indicators; and building a data analysis model based on the statistical indicators and historical data.

[0046] The graph neural network model and the data analysis model are used to perform dynamic analysis and anomaly detection of graph structures.

[0047] Optionally, the non-real-time data includes structured data, semi-structured data and unstructured data, and the processing process of the non-real-time data by the second processing unit based on the optimization strategy corresponding to the non-real-time data includes: using SQL query or data extraction tools to uniformly process the structure of the structured data; processing the semi-structured data into the form of a data dictionary or table, and standardizing label naming; performing feature extraction on the unstructured data, and converting the extracted features into TF-IDF or visual feature vectors.

[0048] Optionally, the storage unit storing the real-time data includes: integrating data obtained after processing the real-time data based on the optimization strategy corresponding to the real-time data with information in the fact table and dimension tables in the data warehouse to construct a wide table; extracting data related to business activity metrics from the fact table, using foreign key associations between the fact table and the dimension tables, adding descriptive attribute information in the dimension table to the fact table to form a wide table containing attributes to be analyzed, and persistently storing the wide table to provide data support for subsequent data analysis and report generation;

[0049] Use data visualization tools to display real-time analysis results and enhance data comprehensibility and analytical capabilities.

[0050] Optionally, the storage unit storing the non-real-time data includes: storing the non-real-time data separately as structured, semi-structured and unstructured data based on the Hadoop framework.

[0051] In a third aspect, the present application provides a device comprising a memory and a processor, wherein the memory is used to store instructions or codes, and the processor is used to execute the instructions or codes so that the device performs the multi-source data processing method introduced in any implementation of the first aspect.

[0052] In a fourth aspect, the present application provides a computer-readable storage medium storing a code. When the code is executed, the device executing the code implements the multi-source data processing method described in any implementation of the first aspect.

[0053] The present application provides a multi-source data processing method. When executing the method, first, multi-source data to be processed is obtained, and the multi-source data includes real-time data and non-real-time data; then, the multi-source data is pre-processed according to the data type; then, the data after data pre-processing is processed using a corresponding preset optimization strategy based on the data type; finally, the data after optimization processing is stored. In this way, by pre-processing according to the data type, it is possible to accurately remove noise, correct errors, unify formats, etc. according to the characteristics and needs of different data, effectively improve data quality, and lay a solid foundation for subsequent processing; processing based on the data type using a corresponding preset optimization strategy can give full play to the advantages of different strategies in different data scenarios, thereby fully exploring the value of various types of data; finally, the optimized data is stored, providing high-quality, structured data resources for subsequent data analysis, application, etc. The effect of improving data processing efficiency and enhancing data availability and value is achieved. In this way, when faced with complex and diverse multi-source data, valuable information can be extracted from it more efficiently and accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in this embodiment or the prior art, the following briefly introduces the drawings required for use in the embodiment or the prior art description. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0055] Figure 1 A flowchart of a multi-source data processing method provided in an embodiment of the present application;

[0056] Figure 2 A pseudo code of the GraphSAGE algorithm provided in an embodiment of the present application;

[0057] Figure 3 A schematic diagram of the structure of a multi-source data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0059] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0060] Figure 1 A flowchart of a multi-source data processing method provided in an embodiment of the present application. Figure 1 As shown, the multi-source data processing method provided in the embodiment of the present application may include:

[0061] S101 : Acquire multi-source data to be processed, where the multi-source data includes real-time data and non-real-time data.

[0062] The multi-source data includes real-time data and non-real-time data, specifically covering operation and maintenance data in different fields such as Infrastructure as a Service (IaaS), Platform as a Service (PaaS), Data as a Service (DaaS), and Software as a Service (SaaS).

[0063] When acquiring real-time data, we use real-time stream data processing engines such as Apache Kafka and Apache Flink to receive real-time data streams from business systems, sensors, user behavior, and other channels. When acquiring non-real-time data, we use database connections such as MySQL, Hive, DB2, Teradata, and MongoDB to collect structured data; and use API interfaces, file servers (FTP / SFTP), ES, and other middleware to collect semi-structured and unstructured data.

[0064] Specifically, when acquiring non-real-time data, there are two situations: database-based collection and interface-based collection. For database-based collection, multi-source data collection capabilities and various control components such as script components, terminal command components, and control components are used to achieve rapid exchange of data between different platforms. It supports the configuration of acquisition of source information, libraries, tables, and field information, and supports full and incremental extraction of data. In specific implementation, on-site / off-site collection capabilities can be provided based on database characteristics. The on-site collection method exports database tables to the file server through native commands; the off-site collection method completes data exchange based on memory.

[0065] For interface-based collection, it manages data collection tasks for relational databases such as MySQL, DB2, Teradata, and non-relational databases such as MongoDB, and has the ability to adapt to multiple data formats including structured and unstructured data, such as JSON, XML files, and binary data. By flexibly configuring extraction modes and transmission protocols, unified access and efficient processing of multi-source heterogeneous data can be achieved. When collecting based on interfaces, data extraction is adopted. The specific extraction process includes timed scheduling extraction, rule-based file extraction, real-time incremental data synchronization, streaming message triggering, etc. It adapts to multiple transmission protocols (FTP, SFTP) and supports multi-concurrent collection. The specific descriptions of various extraction modes are as follows:

[0066] Scheduled extraction: This allows you to set up automated scheduled tasks to capture data from a specified data source based on preset frequency and rules. This allows you to trigger tasks based on time, initiate processes and execute tasks according to the configured cycle and time, ultimately achieving automated data updates and integration.

[0067] Rule-based file extraction: Collect files according to certain business rules, such as collecting files in the order of the time contained in the file name (time is generated based on variables); collecting files based on row / column delimiters and user-defined delimiters.

[0068] Real-time incremental data synchronization: Real-time incremental extraction is achieved through direct monitoring and identification of database logs.

[0069] Streaming message triggering: Visualized streaming message triggering task orchestration shields underlying technical details, abstracts real-time data development forms, provides graphical data orchestration services, and triggers related extraction operations by monitoring and processing real-time streaming messages.

[0070] After the extraction of the non-real-time data is completed, the extracted files need to be corrected and adapted, including correction and adaptation when collecting files and correction and adaptation after collecting files. When collecting files, the content of the file header is filtered; adapt to diversified source file types, including csv, txt, compressed format files gz and other formats; verification strategies for empty files, no files, and the number of files, the strategies mainly include: waiting, failure, success, etc.; verification file content includes file name, file size, number of records, verification file name containing variables and wildcards, etc.; verification strategy for collection based on verification files, verification file content organization supports row mode (one row of data shows the data of a verification file) and column mode (one column of data shows the data of a verification file) verification.

[0071] S102: Preprocess the multi-source data according to their data types.

[0072] Use pre-trained data processing models to clean and standardize invalid, duplicate records, missing values ​​and outliers in real-time data according to preset real-time data processing strategies; use pre-trained data processing models to clean and standardize invalid, duplicate records, missing values ​​and outliers in non-real-time data according to preset non-real-time data processing strategies.

[0073] Clean real-time data using pre-trained data processing models and real-time data processing strategies to clean and standardize invalid, duplicate records, missing values, and outliers. The specific steps are as follows:

[0074] Data cleaning: Remove invalid or duplicate records from real-time data streams and handle missing and outliers. For example, thresholds can be set to identify and remove abnormally high values ​​or obviously erroneous data points.

[0075] Standardization: Convert data into a unified format, such as converting timestamps into a standard time format, to ensure data consistency and comparability.

[0076] Dynamically adjust strategies: Machine learning models—pre-trained data processing models—are introduced to dynamically adjust data processing strategies based on changing characteristics of real-time data streams. For example, during peak traffic periods, more precise data cleaning and aggregation algorithms are automatically employed to improve processing efficiency and accuracy. For example, in real-time user experience monitoring in the telecommunications industry, traffic data, signal strength, network latency, and other metrics are collected from each base station in real time. Using stream processing frameworks such as Kafka, data is transmitted to the processing engine in real time. During the preprocessing phase, invalid or duplicate data records are automatically removed, missing values ​​and outliers are handled, and all timestamps are converted to a standard time format for subsequent analysis.

[0077] For non-real-time data, we use automated machine learning (AutoML) technology to adaptively select and optimize preprocessing strategies to adapt to structured, semi-structured, and unstructured data from different data sources. The specific steps are as follows:

[0078] Data source analysis: AutoML first conducts in-depth analysis of data from different sources (such as databases, APIs, and file systems) to identify the structure and characteristics of the data.

[0079] Strategy Selection and Optimization: Based on data analysis results, AutoML automatically selects the optimal preprocessing strategy, including missing value filling, data cleaning, and outlier detection. It also evaluates the impact of each preprocessing method on downstream model performance in real time and optimizes the preprocessing pipeline based on the actual results.

[0080] Large-Scale Data Processing: This system supports large-scale data processing through distributed computing frameworks (such as Hadoop and Spark), further improving processing performance. For example, in user behavior analysis in the telecommunications industry, the system needs to process large amounts of non-real-time user behavior data, including application usage time, data traffic, and user location. Using AutoML technology, the system automatically analyzes the structure and characteristics of this data and selects appropriate preprocessing strategies. For example, statistical methods are used to fill in fields with a high number of missing values, while model-based methods are used to detect and correct outliers.

[0081] Whether it is real-time data or non-real-time data, the built-in intelligent data feature extraction mechanism, i.e., pre-trained data processing model, automatically detects and removes data with errors, missing values, and outliers, and extracts data features that meet the requirements. The specific mechanism is as follows:

[0082] Training data screening and preparation: Define basic specifications for each data source, including data name, source address, content, and access permissions. Sequentially number the data dimensions within each data source or dataset. From all descriptive information, select the basic source data that meets the specifications as the raw data to form the training set for training the data feature extraction algorithm.

[0083] Feature extraction algorithm training: Initial features are set, including time series features, such as extracting the hour, week, and month from timestamps; usage statistics, such as calculating total usage, average usage, and peak usage; and behavioral pattern features, such as identifying usage patterns, including periodic and bursty usage. Based on the test results, a data feature extraction algorithm is selected. In this embodiment, a VAE (Vertical Autoencoder) is used for feature extraction. The VAE structure consists of an input layer, an encoding layer, an intermediate layer, a decoding layer, and an output layer. Specific parameters are adjusted based on the number of data features.

[0084] Assume that a 5G telecom product dataset has 20 features. The following parameters can be set: 20 neurons in the input layer, 10 neurons in the encoding layer, 5 neurons in the intermediate layer, 10 neurons in the decoding layer, and 20 neurons in the output layer. Use ReLU as the activation function, mean squared error as the loss function, and the Adam optimizer. Set the learning rate to 0.001, the batch size to 64, and the number of training epochs to 10 (or until convergence). Use cross-validation to evaluate the performance of the feature extraction algorithm. This involves repeating the training process with a randomly selected training set from the original data and comparing the results of the two training runs to ensure consistency.

[0085] Feature Extraction and Application: After training, the encoding layer of the autoencoder outputs a new feature representation of the data. For new data, the autoencoder generates a feature vector through the forward propagation process. The data's eligibility is determined based on the reconstruction error of the feature vector. Only when the reconstruction error falls below a set threshold is the data considered qualified and used for subsequent analysis.

[0086] For example, using telecom customers' 5G product usage data as an example, the system first selects data sources that meet basic specifications as a training set. The autoencoder model is trained by specifying time series features such as hour, week, and month; usage statistics such as total usage, average usage, and peak usage; and behavioral pattern features such as periodicity and bursts. After training, the model is used to extract features from new 5G product usage data and determine the data's eligibility based on reconstruction error, ensuring the accuracy and reliability of subsequent analysis.

[0087] S103: Process the pre-processed data using a corresponding preset optimization strategy based on the data type.

[0088] The processing of the real-time data based on the optimization strategy corresponding to the real-time data includes: performing hybrid calculations based on static data in a data warehouse and pre-processed real-time data to obtain first data; the static data includes fact tables and dimension tables stored in the data warehouse, the fact tables storing quantitative data related to business processes, and the dimension tables storing descriptive information; the hybrid calculation realizes hybrid calculations of offline data and real-time stream programs by supporting the definition of external tables, and adopts refined processing of incremental tables, the incremental tables containing new data or changed data within a period of time, and processing of incremental tables is realized through technologies such as timestamps and log serialization; using a preset reinforcement learning method to optimize management and resource allocation of the first data to obtain second data; the reinforcement learning method dynamically adjusts the real-time data pipeline according to the amount, structure and importance of the input data to optimize data flow and resource allocation; using stream processing technology to perform online calculations on the real-time data in the second data, aggregating the real-time data in the incremental tables to generate statistical indicators; constructing a data analysis model based on the statistical indicators and historical data; and using a graph neural network model and the data analysis model to perform dynamic analysis of graph structures and anomaly detection.

[0089] The hybrid computation based on static data in the data warehouse and pre-processed real-time data to obtain the first data includes enabling hybrid computation of offline data with real-time streaming programs by supporting the definition of external tables. Hybrid computation of static and real-time data requires refined processing of incremental tables. Static data is typically stored in fact tables or dimension tables in the data warehouse. Fact tables store data related to business processes, such as sales and transactions. Fact tables typically contain a large number of records and grow over time, used to measure business activity. Dimension tables store descriptive information, such as customer, product, and time dimensions, and typically do not change significantly. Real-time data is typically stored in operational databases or dedicated real-time data processing systems, such as Apache Kafka and Apache Storm. Fact tables are linked to dimension tables via foreign keys, forming a star schema or snowflake schema. Constructing a wide table involves merging multiple tables through joins into a single large table containing all relevant fields. Incremental tables only contain new or changed data within a certain period of time. Processing of incremental tables can be achieved using technologies such as timestamps and log serialization.

[0090] Specific processing flow: First, clarify whether the table involved in the statistical analysis is a fact table or a dimension table. Preprocess the incremental table to reduce invalid business operation data and table association data, thereby reducing the data volume. Dimensional modeling is performed based on the processed fact table to construct a wide table. Simultaneously, the impact of incremental data on the full data in the fact table and dimension tables is analyzed. In this way, combined with changes in dimension and fact tables, a precise understanding of the actual impact of incremental data on the full data can be obtained. Fact tables and dimension tables are two core concepts. The following is the processing flow for static and real-time data, the relationship between each table, and its construction logic:

[0091] Fact tables primarily store quantitative data about business activities, such as sales figures and transaction counts. In practice, fact tables typically contain both historical and incremental data. When processing incremental data, newly arrived data must be merged into the fact table while removing duplicates and cleaning invalid data. Dimension tables store attribute information related to the fact table, such as customer information, product information, and time dimensions. Dimension tables are generally static and change infrequently. In data modeling, dimension tables enrich the context of the fact table through descriptive attributes. Delta tables store newly arrived data and typically contain records added or updated since the last processing. There are two main processing steps: cleaning, which removes invalid business operation data to ensure data quality; and filtering, which reduces data volume by retaining only necessary data relevant to the fact and dimension tables. After preprocessing, the incremental data is merged into the fact table and dimensionally modeled based on the information in the dimension tables. Wide tables join the fact table with dimension tables to form a single table containing all necessary attributes for subsequent analysis. Wide tables extract metric data from fact tables and add attributes from dimension tables to the fact tables through foreign key joins. After processing, wide tables are generated for analysis. Fact tables are linked to dimension tables through foreign keys to ensure data integrity. For example, a sales fact table can be linked to a customer dimension table using customer ID to obtain detailed customer information. By updating incremental tables, the impact of incremental data on the full data set is regularly calculated to assess business trends and changes. Impact analysis can be performed by comparing incremental data with historical data, for example, by calculating the ratio of new transactions in the incremental table to total transaction volume.

[0092] Real-time data is output to data storage to support offline computing. By writing to a database partition table or pushing directory switch events to a message queue, it drives the DACP scheduling task for subsequent batch processing tasks. Specifically, to support offline computing needs, the data storage device will periodically perform scheduled data transfers. The steps are as follows:

[0093] 1. Determine whether the operating capacity indicators of the data storage device meet the conditions for switching the storage database;

[0094] 2. If the operational capability indicator meets the conditions for switching the storage database, wait for the completion of a scheduled data transfer process, and after completion of the scheduled data transfer process, determine that all data in the initial database is transferred to the target database, wherein the scheduled data transfer process is a process of periodically transferring newly generated data from the initial database to the target database;

[0095] 3. Switch the database for reading and writing data from the initial database to the target database.

[0096] Since the data storage device originally performs a scheduled data migration process periodically, when the operating capability indicator reaches the switching condition, only the scheduled data migration process that has not yet been performed needs to be transferred.

[0097] The first data is optimized and resource allocated using a pre-defined reinforcement learning method. An AI-driven data orchestration framework is introduced to obtain the second data, dynamically adjusting the real-time data pipeline based on the volume, structure, and importance of the input data. This AI model (reinforcement learning) can optimize data flow and resource allocation based on historical patterns and predictions of peak data times. The steps are as follows:

[0098] 1. Use reinforcement learning to optimize real-time data ingestion based on Kafka and Flume. The model can dynamically adjust the pipeline by learning the optimal ingestion and processing strategies under different data loads.

[0099] ② The orchestration model can learn to dynamically scale the pipeline to meet real-time demands by adjusting memory allocation, number of processing threads, and stream partitioning.

[0100] This application makes full use of stream processing technology to process and analyze the real-time data in the second data. Specifically, with the help of stream processing technology, real-time data is calculated online, such as calculating key indicators such as real-time sales and the number of active users. At the same time, aggregation operations are performed on real-time data in the incremental table to generate various statistical indicators, such as hourly sales, user activity, etc. These statistical indicators can intuitively reflect the real-time dynamics of the business. In order to more deeply explore the value of data, this application introduces and develops a hybrid model. This model organically combines multi-source real-time data with historical data for analysis. In this way, on the one hand, it can effectively identify changes in business trends, and on the other hand, it can keenly detect abnormal behaviors. Real-time data has the characteristics of strong liquidity, while historical data has stability. By utilizing the difference in characteristics between the two, a more accurate prediction model can be constructed to provide strong support for business decision-making.

[0101] Graph neural network models and the aforementioned data analysis model are used to dynamically analyze graph structures and detect anomalies. Real-time data processing incorporates graph neural network (GNN) models to enable dynamic analysis of graph structures within real-time data. This not only supports conventional data processing but also enables real-time relationship analysis and anomaly detection for real-time data from social networks and IoT devices. The steps are as follows: ① Adopt an architecture that combines static graph data with real-time data (such as user behavior and transactions). GNN models are trained to detect changes in relationships or unusual dynamics within groups. ② GraphSAGE or GAT (graph attention network) models are used to predict changes in important nodes or links within the real-time graph.

[0102] The pseudo code of the algorithm is as follows Figure 2 As shown, Figure 2 This is pseudocode for the GraphSAGE algorithm provided in the embodiments of this application. The GraphSAGE algorithm aims to generate a vector representation (embedding) for each node in a graph. These vectors capture the node's characteristics and its structural information within the graph, and can be used for subsequent tasks such as node classification and link prediction. The following explains the pseudocode.

[0103] Input: Graph Refers to the input graph structure, where V is a set of nodes, Is the edge set. Input features : The initial feature vector of each node v; Depth K: The propagation depth of the algorithm, which determines the scope of the aggregated information; Weight matrix W k , : Parameter matrix used to linearly transform the aggregated features in each layer of propagation; nonlinear function : Used to introduce nonlinearity, such as ReLU function; differentiable aggregation function AGGREGATE k , : Function used to aggregate node neighbor information, such as mean aggregation, pooling aggregation, etc.; neighbor function N:v→2 V , defines how to sample the set of neighbor nodes for each node v.

[0104] Output: vector representation z v for all v∈V: the final vector representation generated for each node in the graph.

[0105] Algorithm steps: Initialization, for each node v∈V, its initial feature vector X v Assign to ,Right now ←X v .here Represents the feature representation of node v at layer 0 (initial layer).

[0106] Multi-layer propagation: For each layer k=1,…,K, for each node v∈V, use the aggregation function AGGREGATE k The feature representation of the neighbor node u∈N(v) of the aggregation node v in the previous layer (k-1 layer) , get the aggregated neighbor information ,Right now ←AGGREGATE k ({ ,∀u∈N(v)}). Feature update, the previous layer feature of node v itself is represented and aggregated neighbor information Perform concatenation (CONCAT operation) and then pass the weight matrix W k Perform linear transformation and apply nonlinear function σ to obtain the feature representation of node v at the kth layer ,Right now )). Feature representation of all nodes v∈V Perform normalization processing, that is ← / 2, , to ensure that the length of the feature vector is unit length and improve the stability of training.

[0107] Output result: After propagation through K layers, the feature representation of each node in the K layer is As its final vector representation z v , that is, z v ← , .

[0108] Assuming K aggregations are performed, this can be considered K layers. Each aggregation step aggregates the features of each node from the previous layer, assuming the node's own features from the previous layer, to obtain the features of that layer. This aggregation is repeated K times to obtain the final features of the node. The node features of the bottom layer are the input node features. The final output is the node relationships and corresponding classifications. This real-time updated node relationship graph can be used to predict future changes in key nodes or links (such as the risk of base station failures), and to generate early warnings (such as identifying areas with high call dropout rates) based on anomaly detection in the dynamic graph structure.

[0109] Non-real-time data is processed based on the optimization strategy corresponding to non-real-time data. The processing process covers the standardization of different types of data. The ultimate goal is to convert all types of data into numerical feature vectors. The specific processing process is as follows:

[0110] For structured data, use SQL queries or data extraction tools to unify the structure of the structured data. For data in relational databases, ensure that the data follows the same format and data type. Use SQL queries or data extraction tools to standardize the data.

[0111] For semi-structured data, process it into a data dictionary or table format and standardize tag naming. For XML or JSON files, use a parser to convert the data into a more manageable format, such as a dictionary or table. Also, standardize the data structure, such as using a unified naming convention for tags or keys.

[0112] For unstructured data, we perform feature extraction and transform the extracted features. For text data, we perform preprocessing steps such as word segmentation, stop word removal, stemming, or lemmatization. For image and video data, we use computer vision techniques to extract features such as edges, color, texture, and object recognition.

[0113] Through the above process, the ultimate goal is to convert all types of data into numerical feature vectors. These feature vectors include:

[0114] Normalized numerical data; One-Hot encoded categorical data; Bag-of-Words models or TF-IDF vectors extracted from text; Visual feature vectors extracted from images and videos.

[0115] S104: Storing the optimized data.

[0116] For the storage process of real-time data, the data obtained after processing the real-time data based on the optimization strategy corresponding to the real-time data is first integrated with the fact table and dimension table information in the data warehouse to build a wide table; by extracting data related to business activity measurements from the fact table, using the foreign key relationship between the fact table and the dimension table, the descriptive attribute information in the dimension table is added to the fact table to form a wide table containing the attributes to be analyzed, and the wide table is persistently stored to provide data support for subsequent data analysis and report generation; the real-time analysis results are displayed through data visualization tools to enhance the comprehensibility and analysis capabilities of the data.

[0117] First, the data generated after processing the optimization strategy based on real-time data is integrated with the fact table and dimension table information in the data warehouse. The fact table stores data closely related to business activity metrics, while the dimension table contains descriptive attribute information. The two are linked by foreign keys. Data related to business activity metrics is extracted from the fact table. Using foreign key relationships, the descriptive attribute information in the dimension table is added to the fact table, thereby constructing a wide table containing the attributes to be analyzed. Once the wide table is constructed, it is persisted. This wide table provides solid data support for subsequent data analysis and report generation, enabling the analysis process to be based on comprehensive and structured data.

[0118] To enhance data comprehensibility and analytical capabilities, real-time analysis results are visualized. This solution uses Echarts combined with PowerBI for display. This combination provides intuitive decision support for mid- and senior-level managers. Furthermore, the integration of natural language processing (NLP) technology enables users to query real-time data using natural language, greatly enhancing the interactive experience between users and data and allowing non-technical personnel to easily access the information they need.

[0119] In terms of storage, the system also incorporates graph databases and graph neural networks (GNNs). By integrating real-time data with the graph database and constructing a dynamic graph, it enables real-time updates of nodes and edges, thus supporting dynamic monitoring of relationship changes. Leveraging GNN models, it is possible to classify nodes in the graph database, predict the future generation of key nodes and edges, and detect anomalous relationships early on. Leveraging graph models, the system visualizes complex multi-source data relationships, further enhancing data comprehensibility and analysis capabilities, and helping users gain deeper insight into the connections and patterns underlying the data.

[0120] Through the above methods, real-time data can be effectively stored, analyzed and displayed, providing strong support for corporate decision-making and business development.

[0121] In one implementation of the embodiment of the present application, the storing of the non-real-time data includes: storing the non-real-time data separately as structured, semi-structured and unstructured data based on the Hadoop framework.

[0122] To meet the demand for high-throughput data import, all data is first obtained from the data source with ID identification, arranged in alphabetical or numerical order, and the data is retrieved and grouped according to characteristic characters. By building a data queue, the data is ensured to be orderly and easy to process. When the data volume reaches the preset threshold, the data group is aggregated into data blocks. At the same time, considering the frequency of database writes, a multi-layer data cache layer is established, and the number of data blocks in each layer is dynamically adjusted according to the number of common nodes of the data blocks in the adjacent layers to achieve balanced data storage. For unstructured data, such as images and videos, spatiotemporal segmentation, feature extraction and object recognition technologies are used to convert them into understandable information.

[0123] The above are some specific implementations of a multi-source data processing method provided in the embodiment of the present application. Based on this, the present application also provides a corresponding device. The device provided in the embodiment of the present application will be introduced from the perspective of functional modularization.

[0124] Figure 3 A schematic diagram of the structure of a multi-source data processing device provided in an embodiment of the present application. Figure 3 As shown, the multi-source data processing device 300 provided in this embodiment of the present application includes:

[0125] An acquisition unit 310 is configured to acquire multi-source data to be processed, wherein the multi-source data includes real-time data and non-real-time data;

[0126] A first processing unit 320 is configured to perform data preprocessing on the multi-source data according to data types;

[0127] The second processing unit 330 is used to process the data after data preprocessing using a corresponding optimization strategy based on the data type;

[0128] The storage unit 340 is used to store the data after the optimization process.

[0129] In one implementation of the embodiment of the present application, the acquiring unit acquiring the multi-source data to be processed includes:

[0130] Non-real-time data collection based on database or interface;

[0131] Based on the real-time stream data processing engine, it receives real-time data from business systems and sensor channels.

[0132] In one implementation of the embodiment of the present application, the first processing unit preprocesses the multi-source data according to data type, including:

[0133] Use pre-trained data processing models to clean and standardize invalid, duplicate records, missing values, and outliers in real-time data according to preset real-time data processing strategies;

[0134] Use pre-trained data processing models to clean and standardize invalid, duplicate records, missing values ​​and outliers in non-real-time data according to preset non-real-time data processing strategies.

[0135] In one implementation of the embodiment of the present application, the second processing unit processes the real-time data based on the optimization strategy corresponding to the real-time data, including:

[0136] First data is obtained by performing hybrid computing based on static data in a data warehouse and preprocessed real-time data; the static data includes fact tables and dimension tables stored in the data warehouse, the fact tables storing quantitative data related to business processes, and the dimension tables storing descriptive information. The hybrid computing supports the definition of external tables to achieve hybrid computing of offline data and real-time streaming programs, and employs refined processing of incremental tables, which contain new or changed data over a period of time, and is processed using technologies such as timestamps and log serialization.

[0137] Optimizing management and resource allocation of the first data using a preset reinforcement learning method to obtain second data; the reinforcement learning method dynamically adjusts the real-time data pipeline based on the amount, structure, and importance of the input data to optimize data flow and resource allocation;

[0138] Performing online calculations on the real-time data in the second data using stream processing technology, aggregating the real-time data in the incremental table to generate statistical indicators; and building a data analysis model based on the statistical indicators and historical data.

[0139] The graph neural network model and the data analysis model are used to perform dynamic analysis and anomaly detection of graph structures.

[0140] In one implementation of an embodiment of the present application, the non-real-time data includes structured data, semi-structured data and unstructured data. The processing process of the non-real-time data processed by the second processing unit based on the optimization strategy corresponding to the non-real-time data includes: using SQL query or data extraction tools to uniformly process the structure of the structured data; processing the semi-structured data into the form of a data dictionary or table, and standardizing label naming; performing feature extraction on the unstructured data, and converting the extracted features into TF-IDF or visual feature vectors.

[0141] In one implementation of the embodiment of the present application, the storage unit stores real-time data including: integrating data obtained after processing the real-time data based on the optimization strategy corresponding to the real-time data with fact table and dimension table information in the data warehouse to construct a wide table; extracting data related to business activity metrics from the fact table, adding descriptive attribute information in the dimension table to the fact table by utilizing the foreign key relationship between the fact table and the dimension table to form a wide table containing attributes to be analyzed, and persistently storing the wide table to provide data support for subsequent data analysis and report generation;

[0142] Use data visualization tools to display real-time analysis results and enhance data comprehensibility and analytical capabilities.

[0143] In one implementation of the embodiment of the present application, the storage unit storing the non-real-time data includes: storing the non-real-time data separately as structured, semi-structured and unstructured data based on the Hadoop framework.

[0144] The embodiments of the present application also provide corresponding devices and computer storage media for implementing the solutions provided by the embodiments of the present application.

[0145] The device includes a memory and a processor, the memory is used to store instructions or codes, and the processor is used to execute the instructions or codes so that the device executes the method described in any embodiment of the present application.

[0146] The computer storage medium stores code, and when the code is executed, the device executing the code implements the method described in any embodiment of the present application.

[0147] Through the description of the above embodiments, it can be seen that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment methods can be implemented by means of software plus a general hardware platform. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product. The computer software product can be stored in a storage medium, such as a read-only memory (ROM) / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network communication device such as a router) to execute the methods described in various embodiments or certain parts of the embodiments of the present application.

[0148] It is understandable that in the specific implementation of this application, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved, when the above embodiments of this application are applied to specific products or technologies, need to obtain user permission or consent, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions.

[0149] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0150] It should also be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and apparatus embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments. The device and apparatus embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components indicated as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0151] The above is merely one specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A multi-source data processing method, characterized in that: The method comprises: Acquire multi-source data to be processed, wherein the multi-source data includes real-time data and non-real-time data; Performing data preprocessing on the multi-source data according to data types; For the data that has been pre-processed, the corresponding preset optimization strategy is used to process it based on the data type; The optimized data is stored.

2. The method according to claim 1, characterized in that The acquiring of multi-source data to be processed includes: Non-real-time data collection based on database or interface; Based on the real-time stream data processing engine, it receives real-time data from business systems and sensor channels.

3. The method according to claim 1, characterized in that The performing data preprocessing on the multi-source data according to data types includes: Use pre-trained data processing models to clean and standardize invalid, duplicate records, missing values, and outliers in real-time data according to preset real-time data processing strategies; Use pre-trained data processing models to clean and standardize invalid, duplicate records, missing values ​​and outliers in non-real-time data according to preset non-real-time data processing strategies.

4. The method according to claim 1, wherein The process of processing the real-time data based on the optimization strategy corresponding to the real-time data includes: First data is obtained by performing hybrid computing based on static data in a data warehouse and preprocessed real-time data; the static data includes fact tables and dimension tables stored in the data warehouse, the fact tables storing quantitative data related to business processes, and the dimension tables storing descriptive information. The hybrid computing supports the definition of external tables to achieve hybrid computing of offline data and real-time streaming programs, and employs refined processing of incremental tables, which contain new or changed data over a period of time, and is processed using technologies such as timestamps and log serialization. Optimizing management and resource allocation of the first data using a preset reinforcement learning method to obtain second data; the reinforcement learning method dynamically adjusts the real-time data pipeline based on the amount, structure, and importance of the input data to optimize data flow and resource allocation; Performing online calculations on the real-time data in the second data using stream processing technology, aggregating the real-time data in the incremental table to generate statistical indicators; and building a data analysis model based on the statistical indicators and historical data. The graph neural network model and the data analysis model are used to perform dynamic analysis and anomaly detection of graph structures.

5. The method according to claim 1, wherein The non-real-time data includes structured data, semi-structured data and unstructured data. The process of processing the non-real-time data based on the optimization strategy corresponding to the non-real-time data includes: Use SQL queries or data extraction tools to unify the structure of structured data; Process semi-structured data into data dictionaries or tables and standardize label naming; Perform feature extraction on unstructured data and convert the extracted features into TF-IDF or visual feature vectors.

6. The method according to claim 4, characterized in that Real-time data storage includes: The data obtained after processing the real-time data based on the optimization strategy corresponding to the real-time data is integrated with the fact table and dimension table information in the data warehouse to construct a wide table; by extracting data related to business activity metrics from the fact table, using the foreign key relationship between the fact table and the dimension table, the descriptive attribute information in the dimension table is added to the fact table to form a wide table containing the attributes to be analyzed, and the wide table is persistently stored to provide data support for subsequent data analysis and report generation; Use data visualization tools to display real-time analysis results and enhance data comprehensibility and analytical capabilities.

7. The method according to claim 1, characterized in that Storage of non-real-time data includes: The non-real-time data is stored separately according to structured, semi-structured and unstructured data based on the Hadoop framework.

8. A multi-source data processing device, characterized in that: The device comprises: An acquisition unit, configured to acquire multi-source data to be processed, wherein the multi-source data includes real-time data and non-real-time data; A first processing unit, configured to perform data preprocessing on the multi-source data according to data types; The second processing unit is used to process the data after data preprocessing by adopting corresponding optimization strategies based on the data type; The storage unit is used to store the data after optimization processing.

9. A computing device, characterized in that The computing device includes: a memory and a processor; The memory is used to store computer programs; The processor is configured to implement the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images