Feature conversion method and device of real-time data stream and electronic equipment

Abstract

The invention discloses a feature conversion method and device for a real-time data stream and electronic equipment. The method comprises the following steps: acquiring a heterogeneous feature matrix of the real-time data stream; performing comprehensive anomaly detection on the real-time data stream based on the heterogeneous feature matrix to obtain abnormal data and normal data; performing adaptive cleaning on the abnormal data according to the heterogeneous distance of the abnormal data to obtain cleaned multi-source data, and determining the cleaned multi-source data and the normal data as a target data set; constructing a semantic association graph of the target data set; aggregating cross-modal features in the semantic association graph through a graph convolutional network, and screening the cross-modal features based on information entropy to obtain a target cross-modal feature set; performing comprehensive anomaly detection on the real-time data stream based on the heterogeneous feature matrix, screening out and repairing abnormal data, and improving the purity and feature expression ability of the target data set; and the cross-modal features are aggregated and screened based on the graph convolutional network, so that the representation capability and task suitability of the target cross-modal feature set are improved.

Description

Technical Field

[0001] This invention relates to the field of big data processing technology, specifically to a method, apparatus, and electronic device for feature transformation of real-time data streams. Background Technology

[0002] Multi-source heterogeneous data has been widely used in fields such as financial risk control and smart cities, covering structured data (SQL databases), semi-structured data (JSON / XML logs), and unstructured data (text / images). Due to the diverse sources of this type of data, it suffers from problems such as large differences in format, inconsistent semantics, and a lot of noise interference, which places stringent requirements on real-time processing.

[0003] However, existing technologies generally rely on static rules or batch processing modes for feature extraction and transformation when processing multi-source heterogeneous data streams. This results in insufficient real-time data processing and low feature effectiveness, making it difficult to adapt to dynamically changing data distributions and the real-time analysis requirements of high throughput and low latency. Consequently, these technologies cannot meet the high-precision requirements of scenarios such as real-time financial risk control and real-time internet monitoring.

[0004] Therefore, existing technologies have the problem of being unable to efficiently and accurately acquire effective features of multi-source heterogeneous data in real-time data streams. Summary of the Invention

[0005] This invention provides a feature transformation method, apparatus, and electronic device for real-time data streams, aiming to achieve efficient and accurate acquisition of effective features of multi-source heterogeneous data in real-time data streams.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: A feature transformation method for real-time data streams includes: Obtain the heterogeneous feature matrix of the real-time data stream; Based on the heterogeneous feature matrix, a comprehensive anomaly detection is performed on the real-time data stream to obtain abnormal data and normal data. The abnormal data is adaptively cleaned based on the heterogeneous distance of the abnormal data to obtain cleaned multi-source data, and the cleaned multi-source data and the normal data are determined as the target dataset. Construct a semantic association graph of the target dataset; Cross-modal features in the semantic association graph are aggregated by a graph convolutional network, and the cross-modal features are filtered based on information entropy to obtain the target cross-modal feature set.

[0007] Optionally, obtaining the heterogeneous feature matrix of the real-time data stream includes: The heterogeneous attributes of the real-time data stream are obtained based on the metadata parsing engine. The real-time data stream includes at least one of structured data, semi-structured data, and unstructured data. The heterogeneous attributes include at least one of format, encoding, unit, and generation frequency. The heterogeneous feature matrix is ​​generated by using each real-time data stream as a row vector and the heterogeneous attributes as column vectors.

[0008] Optionally, based on the heterogeneous feature matrix, comprehensive anomaly detection is performed on the real-time data stream to obtain abnormal data, including: The heterogeneous feature matrix is ​​subjected to time anomaly detection using a dynamic time warping algorithm to obtain a dynamic time warping anomaly score. Numerical anomaly detection is performed on the heterogeneous feature matrix using the Euclidean distance algorithm to obtain an Euclidean distance anomaly score. The dynamic time warping anomaly score and the Euclidean distance anomaly score are dynamically weighted to obtain a comprehensive anomaly score for the real-time data stream, and the real-time data stream with a comprehensive anomaly score greater than a preset score threshold is identified as the anomalous data.

[0009] Optionally, the dynamic weighting of the dynamic time warping anomaly score and the Euclidean distance anomaly score includes: The first weighting coefficient of the dynamic time warping anomaly score is determined according to the data source type of the real-time data stream, and the second weighting coefficient of the Euclidean distance anomaly score is adjusted accordingly based on the first weighting coefficient. When the real-time data stream is text data, the second weighting coefficient is increased.

[0010] Optionally, the step of adaptively cleaning the abnormal data based on the heterogeneous distance of the abnormal data to obtain cleaned multi-source data includes: Calculate the heterogeneous distance between different data sources in the abnormal data based on the heterogeneous feature matrix; When the heterogeneous distance is greater than a preset distance threshold, the abnormal data is repaired by homogeneous data source interpolation to obtain the repaired abnormal data. When the heterogeneous distance is not greater than the preset distance threshold, the abnormal data is marked with missing values ​​to obtain marked abnormal data; The repaired abnormal data and the marked abnormal data are identified as the cleaned multi-source data.

[0011] Optionally, constructing the semantic association graph of the target dataset includes: Obtain the feature vector for each target dataset; The nodes of the initial semantic association graph are determined based on the feature vectors, the edges of the initial semantic association graph are determined based on the cosine similarity between the feature vectors, and the semantic association graph is constructed based on the nodes and the edges.

[0012] Optionally, the step of aggregating cross-modal features in the semantic association graph through a graph convolutional network and filtering the cross-modal features based on information entropy to obtain a target cross-modal feature set includes: By traversing each node of the semantic association graph through a graph convolutional network and aggregating the feature information of neighboring nodes, the cross-modal features of each node are obtained. Calculate the mutual information gain between the cross-modal features and the target variable, and determine the cross-modal features with mutual information gains higher than a preset threshold as target cross-modal features to form a target cross-modal feature set.

[0013] Optionally, after aggregating cross-modal features from the semantic association graph using a graph convolutional network and filtering the cross-modal features based on information entropy to obtain a target cross-modal feature set, the method further includes: The quality of the target cross-modal feature set is evaluated to obtain a comprehensive quality score for the target cross-modal feature set; The preset distance threshold, anomaly detection step size threshold, parameter weights of the graph convolutional network, and parameter weights of the semantic association graph are dynamically adjusted based on the quality comprehensive score. The updated target cross-modal feature set of the real-time data stream is then re-acquired until the updated target cross-modal feature set that meets the preset standard is obtained. The dimensions of quality assessment include completeness, consistency, and timeliness.

[0014] A feature conversion device for real-time data streams, comprising: The heterogeneous feature processing module is used to obtain the heterogeneous feature matrix of the real-time data stream; An anomaly detection module is used to perform comprehensive anomaly detection on the real-time data stream based on the heterogeneous feature matrix to obtain abnormal data and normal data; An adaptive cleaning module is used to adaptively clean the abnormal data based on the heterogeneity distance of the abnormal data, obtain cleaned multi-source data, and determine the cleaned multi-source data and the normal data as the target dataset. The semantic association module is used to construct a semantic association graph of the target dataset; The feature filtering module is used to aggregate cross-modal features in the semantic association graph through a graph convolutional network, and filter the cross-modal features based on information entropy to obtain a target cross-modal feature set.

[0015] An electronic device includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the following steps: Obtain the heterogeneous feature matrix of the real-time data stream; Based on the heterogeneous feature matrix, a comprehensive anomaly detection is performed on the real-time data stream to obtain abnormal data and normal data. The abnormal data is adaptively cleaned based on the heterogeneous distance of the abnormal data to obtain cleaned multi-source data, and the cleaned multi-source data and the normal data are determined as the target dataset. Construct a semantic association graph of the target dataset; Cross-modal features in the semantic association graph are aggregated by a graph convolutional network, and the cross-modal features are filtered based on information entropy to obtain the target cross-modal feature set.

[0016] In this embodiment, comprehensive anomaly detection is performed on the real-time data stream using the heterogeneous feature matrix of the real-time data stream to filter out abnormal data. Then, the abnormal data is adaptively cleaned based on the heterogeneous distance of the abnormal data to obtain repaired abnormal data, thereby obtaining a reliable target dataset, effectively improving the purity and feature representation capability of the target dataset. Next, a semantic association graph of the target dataset is constructed to capture the semantic relationships between cross-modal features. Cross-modal features are aggregated based on a graph convolutional network, and high information value features are selected as the target cross-modal feature set through mutual information gain. This improves the representation capability and task adaptability of the target cross-modal feature set, ensuring that the real-time data stream achieves efficient and accurate feature transformation in a dynamic data environment. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this embodiment, the accompanying drawings used in the description of the embodiment will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A schematic diagram of a scenario for an embodiment of the feature conversion system for real-time data streams provided by the present invention; Figure 2 A schematic diagram of another embodiment of the feature transformation system for real-time data streams provided by the present invention; Figure 3 A flowchart illustrating an embodiment of the feature transformation method for real-time data streams provided by the present invention; Figure 4 A schematic diagram of a closed-loop process for a quality assessment feedback adjustment embodiment provided by the present invention; Figure 5This invention provides a schematic diagram of the data processing flow for real-time data streams. Figure 6 A flowchart illustrating an embodiment of the present invention for generating an integrated high-order feature set; Figure 7 A schematic diagram of a structure of an embodiment of the feature conversion device for real-time data streams provided by the present invention; Figure 8 This is a schematic diagram of an embodiment of the electronic device provided by the present invention. Detailed Implementation

[0019] The technical solutions in this embodiment will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] In the following description, specific embodiments of the invention will be illustrated with reference to steps and symbols performed by one or more computers, unless otherwise stated. Therefore, these steps and operations will be referred to several times as being performed by a computer, and computer execution as referred to herein includes operations by a computer processing unit representing electronic signals of data in a structured format. This operation transforms the data or maintains it at a location in the computer's memory system, which can be reconfigured or otherwise alter the operation of the computer in a manner well known to those skilled in the art. The data structure maintained by the data is the physical location of the memory, which has specific characteristics defined by the data format. However, the principles of the invention described above are not intended to be limiting, and those skilled in the art will understand that many of the steps and operations described below can also be implemented in hardware.

[0021] The terms "module" or "unit" as used herein can be considered as software objects executing on the computing system. The different components, modules, engines, and services described herein can be considered as implementation objects on the computing system. The apparatus and methods described herein are preferably implemented in software, but can also be implemented in hardware, both of which are within the scope of this invention.

[0022] This invention provides a method, apparatus, and electronic device for feature transformation of real-time data streams.

[0023] Please see Figure 1 , Figure 1This is a schematic diagram illustrating an embodiment of the feature conversion system for real-time data streams provided by the present invention. The system may include a client 100 and a server 200, which are connected via a network. The server 200 integrates a feature conversion device for real-time data streams. The server 200 may be a platform server (i.e., a server loaded with a platform), such as... Figure 1 In this embodiment, the server 200 can be accessed by the client 100. The server 200 is primarily used to acquire the heterogeneous feature matrix of the real-time data stream; perform comprehensive anomaly detection on the real-time data stream based on the heterogeneous feature matrix to obtain abnormal and normal data; adaptively clean the abnormal data according to the heterogeneous distance to obtain cleaned multi-source data, and determine the cleaned multi-source data and normal data as the target dataset; construct a semantic association graph of the target dataset; aggregate cross-modal features in the semantic association graph through a graph convolutional network, and filter the cross-modal features based on information entropy to obtain the target cross-modal feature set.

[0024] In this embodiment, the server 200 can be a standalone server, a server network, or a server cluster. For example, the server 200 described in this embodiment includes, but is not limited to, computers, network hosts, single network servers, multiple network server sets, or cloud servers composed of multiple servers. The cloud server is composed of a large number of computers or network servers based on cloud computing. In this embodiment, communication between the server and the client can be achieved through any communication method, including but not limited to mobile communication based on the 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), and Worldwide Interoperability for Microwave Access (WiMAX), or computer network communication based on the TCP / IP Protocol Suite (TCP / IP) and User Datagram Protocol (UDP).

[0025] It is understood that the client 100 used in this embodiment can be understood as a client device. A client device includes both receiving and transmitting hardware, that is, a device with receiving and transmitting hardware capable of performing bidirectional communication on a bidirectional communication link. Such a client device may include: cellular or other communication devices, having a single-line display, a multi-line display, or a cellular or other communication device without a multi-line display. Specifically, the client 100 may be a desktop terminal or a mobile terminal, specifically a mobile phone, tablet computer, laptop computer, etc.

[0026] Those skilled in the art will understand that Figure 1 The application environment shown is merely one application scenario for the solution in this application and does not constitute a limitation on the application scenario of the solution in this application. Other application environments may include those that are more specific to this application. Figure 1 The number of more or fewer servers shown, or the server network connectivity relationships, for example... Figure 1 Only one server and two clients are shown in the diagram. It is understood that the feature conversion system for the real-time data stream may also include one or more other servers, and / or one or more clients connected to the server network, which is not limited here.

[0027] In some embodiments of the present invention, the working platform may be an enterprise office platform, such as WeChat for Business. Taking server 200 as an example, it may further include an enterprise office platform contact server, an enterprise office platform configuration management server, and a web management server. Enterprise users or developers can access the web management server using a web browser terminal to configure the field configuration information on the enterprise office platform configuration management server, and set and store the enterprise user information of enterprise employees of the enterprise office platform on the enterprise office platform contact server.

[0028] In addition, such as Figure 2 As shown, Figure 2 This is a schematic diagram of another embodiment of the feature conversion system for real-time data streams provided by the present invention. The feature conversion system for real-time data streams may further include a storage terminal 300 for storing data, such as a storage object database. The object database stores object data, which may include application templates (such as approval templates, attendance templates, and other application templates), file data (such as Word files, Excel files, or PPT files in various formats), image data (such as images in various formats such as JPG, PNG, and BMP), and other types of data. Correspondingly, the object database may also be divided into multiple types of data, such as application databases, file databases, or image databases.

[0029] It should be noted that, Figure 1-2The schematic diagram of the feature conversion system for real-time data streams shown is merely an example. The feature conversion system and scenario for real-time data streams described in this embodiment are intended to more clearly illustrate the technical solution of this embodiment and do not constitute a limitation on the technical solution provided by this invention. As those skilled in the art will know, with the evolution of feature conversion systems for real-time data streams and the emergence of new business scenarios, the technical solution provided by this invention is also applicable to similar technical problems.

[0030] The following detailed description is based on specific embodiments.

[0031] In this embodiment, the description will be from the perspective of a feature conversion device for real-time data streams, which can be specifically integrated into server 200.

[0032] This invention provides a feature transformation method for real-time data streams. Please refer to [link / reference]. Figure 3 , Figure 3 A flowchart illustrating an embodiment of the feature transformation method for real-time data streams provided by the present invention includes: S301: Obtain the heterogeneous feature matrix of the real-time data stream; It should be noted that real-time data streams refer to data sequences acquired in scenarios such as financial risk control, environmental monitoring, and e-commerce intelligent operation. These data can be continuous, high-speed generated data streams or discrete, non-uniformly distributed data events, without any restrictions.

[0033] Heterogeneous features refer to features used to describe the attributes of the real-time data stream itself (such as data format, encoding method, update frequency, etc.), rather than the feature set of the data itself.

[0034] A heterogeneous feature matrix is ​​a two-dimensional matrix structure that organizes multiple heterogeneous features according to preset rules, reflecting the distribution differences of various features across different data sources. Specifically, the heterogeneous feature matrix standardizes all data through a unified data access layer, ensuring that data from different sources can be effectively integrated and parsed. For example, in e-commerce intelligent operation scenarios, a heterogeneous feature matrix can include features from user behavior logs such as click sequences (textual), transaction amounts (numerical), product categories (categorical), and order timestamps (time series); in financial risk control transaction monitoring scenarios, a heterogeneous feature matrix can cover multi-dimensional information such as account balance (numerical), transaction channels (categorical), textual descriptions of operational behaviors (textual), and transaction occurrence time (time series); in other scenarios, the heterogeneous feature matrix can also include other data types and feature combinations, without limitation.

[0035] S302: Perform comprehensive anomaly detection on real-time data streams based on heterogeneous feature matrices to obtain abnormal and normal data; It should be noted that comprehensive anomaly detection refers to a multi-dimensional detection mechanism that combines statistical analysis, machine learning models, and rule engines. It constructs detection strategies for various features in the heterogeneous feature matrix to determine whether there are abnormal data points in the real-time data stream that deviate from the normal pattern.

[0036] Specifically, for numerical features, a method combining dynamic threshold detection and time series prediction models is adopted; for categorical features, abnormal jumps are identified through frequency statistics and Markov transition matrices; for textual and time series features, semantic vectors are extracted using pre-trained models and cosine similarity is calculated to discover abnormal patterns. Detection methods can also be combined to determine whether there are anomalies in the real-time data stream, without any restrictions.

[0037] S303: Adaptively clean the abnormal data based on the heterogeneity distance of the abnormal data to obtain cleaned multi-source data, and determine the cleaned multi-source data and normal data as the target dataset; It should be noted that heterogeneous distance is an indicator used to quantify the degree of difference between multi-source heterogeneous data in real-time data streams. That is, it is used to measure the degree of distribution deviation of different data sources in heterogeneous feature space, and can be calculated by weighted Mahalanobis distance or semantic distance based on embedding representation.

[0038] Adaptive cleaning refers to dynamically adjusting the cleaning strategy and threshold parameters based on the distribution characteristics and deviation of abnormal data in a heterogeneous feature matrix, so as to selectively filter or correct abnormal data.

[0039] S304: Construct a semantic association graph of the target dataset; It should be noted that the semantic association graph is a graph structure used to mine cross-modal semantic associations of multi-source heterogeneous data in real-time data streams. Its core is to quantify the semantic association strength between different modalities of data in the form of nodes and edges, providing a foundation for cross-modal feature aggregation.

[0040] S305: Aggregate cross-modal features in the semantic association graph through graph convolutional networks, and filter the cross-modal features based on information entropy to obtain the target cross-modal feature set.

[0041] It should be noted that Graph Convolutional Network (GCN) is a deep learning model mainly used to aggregate cross-modal features of multi-source heterogeneous data in order to discover semantic associations between different modalities.

[0042] Mutual information gain measures the information redundancy and complementarity between different modal features. It involves calculating the change in mutual information between each modal feature and the target variable to select cross-modal feature combinations that significantly enhance the prediction task, while eliminating redundant or weakly correlated features, thereby improving the model's expressive power and generalization performance. This process adaptively retains the most semantically discriminative feature subset, ensuring the efficiency and accuracy of subsequent analysis.

[0043] The target variable refers to the variable that needs to be predicted or classified in a specific task. Its specific form is determined by the application scenario. For example, in equipment failure prediction, it is the status label of the target equipment, and in user behavior analysis, it is the user's operation intention category. There is no limit to this, and it depends on the specific prediction task.

[0044] In summary, this embodiment performs comprehensive anomaly detection on the real-time data stream using the heterogeneous feature matrix of the real-time data stream, filters out abnormal data, and then adaptively cleans the abnormal data based on the heterogeneous distance of the abnormal data to obtain repaired abnormal data, thereby obtaining a reliable target dataset, effectively improving the purity and feature representation capability of the target dataset. Then, a semantic association graph of the target dataset is constructed to capture the semantic relationships between cross-modal features. Cross-modal features are aggregated based on a graph convolutional network, and high information value features are filtered out as the target cross-modal feature set through mutual information gain, which improves the representation capability and task adaptability of the target cross-modal feature set, ensuring that the real-time data stream achieves efficient and accurate feature transformation in a dynamic data environment.

[0045] In one specific embodiment, in S301, obtaining the heterogeneous feature matrix of the real-time data stream includes: obtaining the heterogeneous attributes of the real-time data stream based on the metadata parsing engine, wherein the real-time data stream includes at least one of structured data, semi-structured data, and unstructured data, and the heterogeneous attributes include at least one of format, encoding, unit, and generation frequency; and generating a heterogeneous feature matrix with each real-time data stream as a row vector and the heterogeneous attributes as column vectors.

[0046] It should be noted that the system uses a dynamic adapter (compatible with multiple protocols such as Kafka and MQTT, supporting concurrent access of tens of thousands of data sources) to stream multi-source heterogeneous data in real time. Based on the metadata parsing engine (with 200+ built-in data source type identification rules, supporting automatic identification of data semantics such as time series and spatial geography), the system identifies the data source type and constructs a heterogeneous feature matrix.

[0047] Heterogeneous attributes include 12 key attributes such as the format, encoding, unit, and generation frequency of real-time data streams, covering metadata features of modalities such as text, images, and time-series signals.

[0048] In other words, the real-time data stream is multi-source heterogeneous data that is streamed in real time from protocols such as Kafka and MQTT through dynamic adapters, covering structured, semi-structured, and unstructured forms. Furthermore, in order to automatically construct a heterogeneous feature matrix, a metadata parsing engine performs deep analysis on various data sources, extracting 12 key attributes such as format, encoding, unit, and generation frequency, and automatically mapping them into standardized feature columns to ensure that heterogeneous data is aligned in a unified semantic space, and also realizes "on-demand" multi-source data.

[0049] Clearly, since heterogeneous features are used to describe the attributes of the real-time data stream itself, the metadata parsing engine can automatically and quickly extract and standardize heterogeneous features, significantly improving data preprocessing efficiency and consistency, and avoiding errors caused by manual intervention.

[0050] Specifically, multi-source heterogeneous data includes structured data (SQL databases, such as Oracle and MySQL), semi-structured data (JSON / XML, such as logs and API interface data), and unstructured data (text / images, such as news and surveillance videos); the dynamic adapter achieves high-throughput (single node supports 100,000 data entries / second) real-time streaming access through the Kafka message queue.

[0051] In this embodiment, a metadata parsing engine extracts heterogeneous attributes such as format, encoding, unit, and generation frequency of real-time data streams, including structured, semi-structured, and unstructured data streams. A heterogeneous feature matrix is ​​generated using the data stream as row vectors and the heterogeneous attributes as column vectors. This achieves a unified multi-dimensional feature characterization of heterogeneous real-time data, transforming heterogeneous data into a structured feature representation, thereby improving the accuracy and efficiency of real-time data processing and facilitating subsequent data analysis.

[0052] In one specific embodiment, in S302, comprehensive anomaly detection is performed on the real-time data stream based on the heterogeneous feature matrix to obtain abnormal data. This includes: performing time-series anomaly detection on the heterogeneous feature matrix using a dynamic time warping algorithm to obtain a dynamic time warping anomaly score; performing numerical anomaly detection on the heterogeneous feature matrix using an Euclidean distance algorithm to obtain an Euclidean distance anomaly score; dynamically weighting the dynamic time warping anomaly score and the Euclidean distance anomaly score to obtain a comprehensive anomaly score for the real-time data stream, and determining real-time data streams with a comprehensive anomaly score greater than a preset score threshold as abnormal data.

[0053] It should be noted that Dynamic Time Warping (DTW) is a time series data anomaly detection algorithm used to detect data anomalies that are adapted to millisecond-level time series fluctuations. It captures the morphological differences between different time series through non-linear alignment, thereby effectively identifying hidden anomalies caused by delays, jitter, or periodic misalignments.

[0054] The Dynamic Time Warping Anomaly Score is a difference metric calculated by the Dynamic Time Warping algorithm. It reflects the degree of deviation of the real-time data stream in terms of temporal pattern. The higher the score, the greater the temporal alignment deviation from the normal pattern and the stronger the possibility of anomaly.

[0055] Euclidean distance is a classic distance calculation method used to measure the similarity or difference between numerical data points. As an anomaly detection metric for numerical data, it identifies outliers at the millisecond level by calculating the straight-line distance between data points.

[0056] The Euclidean distance anomaly score is a numerical deviation measure obtained by calculating the Euclidean distance between the feature vector of the real-time data stream and the center point of the normal pattern. The larger the distance, the more the data distribution in the multidimensional space deviates from the normal range, and the higher the degree of anomaly.

[0057] The preset scoring threshold is a judgment boundary set based on historical data distribution and actual business needs, usually through statistical methods (such as 3). The algorithm (using principles or box plots) is dynamically adjusted based on expert experience to balance the false positive rate and the false negative rate. When the overall anomaly score exceeds a certain threshold, the real-time data stream is determined to be abnormal data.

[0058] Furthermore, in order to achieve an adaptive anomaly detection mechanism, the adaptability of the comprehensive anomaly score is dynamically adjusted by dynamically weighting the dynamic time warping anomaly score and the Euclidean distance anomaly score. This includes: determining the first weight coefficient of the dynamic time warping anomaly score based on the data source type of the real-time data stream, and adjusting the second weight coefficient of the Euclidean distance anomaly score accordingly based on the first weight coefficient; wherein, when the real-time data stream is text data, the second weight coefficient is increased.

[0059] Specifically, heterogeneous feature matrices ,in, For the first The heterogeneous feature vectors of each data source contain attributes such as data format, encoding method, unit of measurement, generation frequency, and semantic type; heterogeneous distance is obtained through cosine similarity.

[0060] The formula for calculating the comprehensive anomaly score is as follows:

[0061] in, It is a single data point in a real-time data stream (such as a sensor data point or a log record). For dynamic time regularization anomalies, Score the Euclidean distance anomalies. The first weighting coefficient has a value range of (0, 1), ) is the second weighting coefficient.

[0062] Data source types include text, images, and time-series sensor data, and different types of inputs have varying sensitivities to anomalous features. For example, when the data source is sensor data (such as temperature and pressure sensors, with time-series features accounting for 80%), the first weighting coefficient... Set to 0.8; when the data source is text data (such as news, comments, with time-series features accounting for 30%), the second weight coefficient ( Set it to 0.3.

[0063] In this embodiment, a dynamic time warping algorithm is used to detect time-series morphological anomalies, resulting in a dynamic time warping anomaly score. A Euclidean distance algorithm is used to detect numerical deviation anomalies, resulting in an Euclidean distance anomaly score. This combines the advantages of both algorithms in detecting anomalies in both time-series morphology and numerical dimensions, covering as many anomaly types as possible. Furthermore, dynamic weighting is applied based on the data source type of the real-time data stream, making the weight allocation more consistent with the anomaly characteristics of different data types. This improves the accuracy of comprehensive anomaly detection in real-time data streams and its adaptability to different data source types. Clearly, comprehensive anomaly detection not only quickly and accurately identifies anomalous data but also differentiates data from different data source types according to the characteristics of the actual scenario, enhancing the reliability and response flexibility of anomalous data to better adapt to the needs of complex scenarios in multiple fields such as financial transactions, industrial monitoring, and public opinion analysis.

[0064] In one specific embodiment, in S303, after obtaining the abnormal data, adaptive cleaning is performed on the abnormal data based on the heterogeneous distance of the abnormal data to obtain cleaned multi-source data, including: calculating the heterogeneous distance between different data sources in the abnormal data according to the heterogeneous feature matrix; when the heterogeneous distance is greater than a preset distance threshold, isomorphic data source interpolation is performed on the abnormal data to obtain repaired abnormal data; when the heterogeneous distance is not greater than the preset distance threshold, missing value marking is performed on the abnormal data to obtain marked abnormal data; and the repaired abnormal data and the marked abnormal data are determined to be cleaned multi-source data.

[0065] It should be noted that heterogeneous distance is an indicator used to quantify the degree of difference between multi-source heterogeneous data. It is calculated by cosine similarity and is based on a heterogeneous feature matrix containing 12 types of attributes such as data format, encoding method, measurement unit, and generation frequency. This enables the quantification of differences between different data sources, thereby determining a more accurate cleaning strategy and avoiding resource waste and instability caused by indiscriminately processing abnormal data.

[0066] Homogeneous data source interpolation repair utilizes data from other data sources that are highly homogeneous (low in difference) with the abnormal data source. It repairs the abnormal data using methods such as linear interpolation and K-nearest neighbor interpolation, thereby ensuring the accuracy of data repair while avoiding erroneous repairs caused by differences in heterogeneous data.

[0067] Missing value marking refers to directly marking the abnormal data as missing values, preserving its original location and timestamp, so that contextual inference or business rules can be performed in the subsequent data fusion stage to complete the data and ensure that the temporal integrity of the data is not compromised.

[0068] In one specific embodiment, the anomaly detection accuracy is improved by 25% due to the dynamic adjustment of the weight of the first weight coefficient; the data integrity is improved by 30% due to the adaptive interpolation repair based on heterogeneous distance; and the cross-modal graph convolution improves the semantic correlation of features by 40%, solving the problem of "information islands" for single-modal features.

[0069] In this embodiment, adaptive judgment is made based on the heterogeneous distance between different data sources in the abnormal data. For abnormal data with a heterogeneous distance greater than a preset threshold, homogeneous data source interpolation is used for repair, thereby improving the repair accuracy based on the similarity of homogeneous data. For abnormal data with a heterogeneous distance not greater than the preset threshold, missing values ​​are marked to avoid errors introduced by forced repair. This achieves differentiated cleaning of abnormal data, improves the quality and reliability of multi-source data after cleaning, and provides a better foundation for subsequent data processing or analysis.

[0070] In one specific embodiment, in S304, after determining the target dataset based on the cleaned multi-source data and normal data, it is also necessary to obtain reliable features in order to determine the feature distribution pattern of the real-time data stream. Specifically, constructing a semantic association graph of the target dataset includes: obtaining the feature vector of each target dataset; determining the nodes of the initial semantic association graph based on the feature vectors; determining the edges of the initial semantic association graph based on the cosine similarity between the feature vectors; and constructing the semantic association graph based on the nodes and edges.

[0071] It should be noted that a feature vector is a vector that can represent the features of each data point in the target dataset. Specifically, the feature vector for text data is the Word2Vec / BERT vector, the feature vector for image data is the CNN feature vector, and the feature vector for numerical data is the statistical feature vector.

[0072] Specifically, semantic association graph ) In the middle, node V For the feature vectors of each modality data, the edges E The weight is cross-modal similarity. Specifically, cross-modal similarity can be the cosine similarity between text word vectors and image visual features, which can quantify the semantic association strength between different modalities.

[0073] In this embodiment, by using the feature vectors of each target dataset as nodes in the initial semantic association graph to preserve their semantic information, and using the cosine similarity between feature vectors to determine the edges to quantify the semantic similarity between datasets, the constructed semantic association graph presents the semantic relationship between target datasets in a structured way. This provides a semantic-level basis for feature transformation of real-time data streams, which helps to quickly capture the semantic dependencies between datasets in real-time processing and improve the semantic accuracy and processing efficiency of feature transformation.

[0074] In one specific embodiment, in S305, after obtaining the semantic association graph, cross-modal fusion is required to obtain a usable feature representation of the real-time data stream. Specifically, cross-modal features in the semantic association graph are aggregated through a graph convolutional network, and the cross-modal features are filtered based on information entropy to obtain a target cross-modal feature set. This includes: traversing each node of the semantic association graph through a graph convolutional network and aggregating the feature information of neighboring nodes to obtain the cross-modal features of each node; calculating the mutual information gain between the cross-modal features and the target variable, and determining cross-modal features with mutual information gains higher than a preset threshold as target cross-modal features to constitute the target cross-modal feature set.

[0075] It should be noted that the formula for aggregating cross-modal features using graph convolutional networks is:

[0076] in, For nodes The neighborhood; For the first l Layer weight matrix; For the first l Layer bias term; For activation functions; For nodes After the first After the layer graph convolution calculation, the resulting th +1 layer feature vectors; : It is a node A specific node within the neighborhood. It is a node The neighborhood of is with The set of associated nodes; ∑ represents summing all nodes in the neighborhood; The normalization coefficient is... Represents a node The number of neighboring nodes, Represents a node The number of neighboring nodes is used to avoid the impact of differences in the number of node connections on features.

[0077] It should be noted that feature selection based on information entropy is achieved by calculating the mutual information gain of cross-modal features, retaining features with mutual information gain higher than a preset threshold (default Top 30%), thus forming a compact feature set. The feature dimensions are reduced by 40%, and the key information retention rate is 95%.

[0078] In this embodiment, the neighborhood feature information of each node in the semantic association graph is aggregated by a graph convolutional network, which effectively captures the semantic association between cross-modal features, so that the cross-modal features of each node are more comprehensively integrated with the contextual association information. Then, by calculating the mutual information gain between the cross-modal features and the target variable and filtering out features that are higher than a preset threshold, redundant or irrelevant cross-modal features are removed, while features that are more valuable to the target variable are retained. This ensures both the semantic relevance and comprehensiveness of the cross-modal features and improves the relevance of features to the target task. It also improves the effectiveness and relevance of real-time data stream feature transformation, which helps to improve the performance of subsequent tasks (such as classification, prediction, etc.) based on the target cross-modal feature set.

[0079] Furthermore, the quality of the target cross-modal feature set cannot be guaranteed. Therefore, after aggregating cross-modal features from the semantic association graph through a graph convolutional network and filtering the cross-modal features based on information entropy to obtain the target cross-modal feature set, the process further includes: evaluating the quality of the target cross-modal feature set to obtain a comprehensive quality score; dynamically adjusting the preset distance threshold, anomaly detection step size threshold, parameter weights of the graph convolutional network, and parameter weights of the semantic association graph based on the comprehensive quality score, and re-acquiring the updated target cross-modal feature set from the real-time data stream until an updated target cross-modal feature set with a comprehensive quality score that meets the preset standard is obtained; wherein, the dimensions of quality evaluation include completeness, consistency, and timeliness.

[0080] In one specific embodiment, a multi-dimensional quality assessment system is established that includes completeness (missing value ratio ≤5% is acceptable), consistency (cross-source data conflict rate ≤3% is acceptable), and timeliness (processing delay ≤200ms is acceptable). Based on the assessment results, the cleaning strategy parameters (such as anomaly detection threshold and feature fusion weight) are adjusted in reverse to form a real-time closed-loop optimization, so as to realize the real-time evaluation of the obtained target cross-modal feature set.

[0081] It should be noted that the calculation formula for the multidimensional quality assessment system is as follows:

[0082]

[0083]

[0084]

[0085]

[0086] in, The overall quality score of the target cross-modal feature set. C The integrity score for the target cross-modal feature set. For the consistency score of the target cross-modal feature set, The timeliness score for the target cross-modal feature set, Custom constant weights can be adaptively adjusted as needed, and no restrictions are imposed here.

[0087] Furthermore, to visually illustrate the quality assessment process, please refer to [link / reference]. Figure 4 , Figure 4 This is a schematic diagram of a closed-loop process for a quality assessment feedback adjustment embodiment provided by the present invention. After data cleaning, features are extracted from the cleaned target dataset to obtain a target cross-modal feature set. Then, the target cross-modal feature set is quality-assessed. If the quality meets the standards, i.e., the integrity score, consistency score, and timeliness score all reach the acceptable thresholds, the target cross-modal feature set is deemed to meet the requirements and is output. If any of the integrity score, consistency score, or timeliness score fails to reach the acceptable threshold, parameters need to be dynamically adjusted. Specifically, adjustable parameters include the anomaly detection threshold, temporal feature weights, graph convolutional network weight matrix, and indicator weights. Then, based on the adjusted parameters, data cleaning, feature extraction, and quality assessment are repeated until the assessment meets the standards and the features are output.

[0088] In one specific embodiment, integrity score The acceptable threshold is ≥0.9; consistency score The passing threshold is ≥0.85; Timeliness score The acceptable threshold is ≥0.6; in financial scenarios, In other embodiments, Other values ​​are also possible, and there are no restrictions here.

[0089] In addition, in the closed-loop optimization mechanism that dynamically adjusts the preset distance threshold, anomaly detection step size threshold, parameter weights of the graph convolutional network, and parameter weights of the semantic association graph based on the overall quality score, and re-acquires the updated target cross-modal feature set from the real-time data stream until an updated target cross-modal feature set with an overall quality score that meets the preset standard is obtained, the adjusted cleaning strategy parameters include the anomaly detection threshold (dynamically adjusted step size of 0.05) and temporal feature weights. (Adjust granularity to 0.1), Graph convolutional network weight matrix (Updated via gradient descent) and indicator weights (Respond to changes in business priorities in real time) and dynamically optimize data processing results through real-time feedback adjustments.

[0090] In one specific embodiment, information entropy screening reduces feature redundancy by 60% and increases model training speed by 50%; "multi-dimensional evaluation + closed-loop optimization" enables continuous iteration of data quality, meeting the real-time and reliability requirements of Internet, finance and other scenarios.

[0091] In this embodiment, the target cross-modal feature set is evaluated from three dimensions: completeness, consistency, and timeliness, and a comprehensive score is obtained. Based on the score, the preset distance threshold, anomaly detection step size threshold, graph convolutional network, and semantic association graph parameter weights are dynamically adjusted to regenerate and update the target cross-modal feature set until it meets the preset standard. This can continuously optimize the quality of the target cross-modal feature set and improve the accuracy, reliability, and real-time performance of the feature set.

[0092] In a specific implementation scenario, please refer to Figure 5 , Figure 5 The data processing flowchart for real-time data streams provided by this invention specifically includes: First, data access and heterogeneous parsing are performed: Multi-source heterogeneous data is collected extensively from channels such as financial transaction systems and smart city sensing devices, including structured transaction data from SQL databases, semi-structured log data in JSON format, and unstructured public opinion data in text / image formats. Data is streamed in real-time through a dynamic adapter (based on a Kafka message queue). The metadata parsing engine automatically identifies the data source type, constructs a heterogeneous feature matrix, and calculates the heterogeneous distance between data sources, for example, distinguishing the heterogeneous attribute differences between transaction data and public opinion text data.

[0093] Then, the anomaly detection and cleaning phase is performed: the preprocessed data is divided into real-time streams, and a combination of dynamic time warping (DTW) for temporal anomaly detection and Euclidean distance for numerical anomaly detection is used to calculate a comprehensive anomaly score. If the data source is sensor data, a first weighting coefficient is set. The value is 0.8; if the data is text, the first weighting coefficient is set to 0.8. It is 0.3; based on the heterogeneous distance Choose a cleaning strategy: If ( (For the preset distance threshold), interpolation based on a homogeneous data source is used to repair abnormal data; if Mark abnormal data as missing values.

[0094] Among them, for heterogeneous feature matrices heterogeneous distance The calculation formula is:

[0095] in, Indicates the first i Abnormal data from each data source With the j Abnormal data from each data source Heterogeneous distance between them express The norm of .

[0096] Next, the feature fusion and extraction stage will be carried out: constructing a semantic association graph of the multimodal data. Node V represents the feature vectors of each modality of data (e.g., statistical feature vectors of transaction data, word vectors of text, and visual feature vectors of images), and the weights of edges E are cross-modal similarities (e.g., the cosine similarity between text word vectors and image visual features). Cross-modal features are aggregated using a graph convolutional network formula, and then processed by mutual information gain based on information entropy. ( For the first i A cross-modal feature, To select features for the target variable and form a compact feature set. .

[0097] Finally, in the quality assessment and feedback optimization phase: when performing real-time data processing, a comprehensive score for completeness, consistency, and timeliness is calculated using a multi-dimensional quality assessment formula. Based on the assessment results, the cleaning strategy parameters, including the anomaly detection threshold and the first weighting coefficient, are adjusted in reverse. Graph convolutional network weight matrix and indicator weights And re-detect anomalies to form a real-time closed-loop optimization.

[0098] Through the above methods, users can filter and adjust the data cleaning effect and feature quality according to actual business needs such as financial risk control and smart cities, so as to ensure that the output data meets the needs of subsequent modeling and decision-making.

[0099] Furthermore, to visually illustrate the difference between the target cross-modal features and the original target dataset's single-modal features, please refer to [link to relevant documentation]. Figure 6 , Figure 6 This is a flowchart illustrating an embodiment of generating an integrated high-order feature set provided by the present invention. The high-order feature set is the target cross-modal feature set that needs to be obtained in the end. Text features, image features, and numerical features are the single-modal features of the real-time data stream itself (such as text word vectors, numerical statistical features, and image features). By performing cross-modal fusion on all single-modal features, the integrated high-order feature set—the target cross-modal feature set—is finally obtained.

[0100] In this embodiment, the original single-modal features such as text features, image features, and numerical features of real-time data streams are fused across modally to generate an integrated high-order feature set (target cross-modal feature set). Compared with single text, image, or numerical features, it can integrate multimodal information, more comprehensively and accurately reflect the essential characteristics of the data, and effectively improve the expressive power of the features so that users can use them directly.

[0101] To facilitate better implementation of the feature transformation method for real-time data streams provided by this invention, this invention also provides an apparatus based on the aforementioned feature transformation method for real-time data streams. The meanings of the terms used are the same as in the aforementioned feature transformation method for real-time data streams, and specific implementation details can be found in the descriptions of the method embodiments.

[0102] Please see Figure 7 , Figure 7 A schematic diagram of an embodiment of the feature conversion device for real-time data streams provided by the present invention is shown, wherein the feature conversion device 700 for real-time data streams may include: Heterogeneous feature processing module 701 is used to obtain the heterogeneous feature matrix of real-time data stream; Anomaly detection module 702 is used to perform comprehensive anomaly detection on real-time data streams based on heterogeneous feature matrices to obtain abnormal data and normal data. The adaptive cleaning module 703 is used to adaptively clean abnormal data based on the heterogeneous distance of the abnormal data, obtain cleaned multi-source data, and determine the cleaned multi-source data and normal data as the target dataset. Semantic association module 704 is used to construct a semantic association graph of the target dataset; The feature filtering module 705 is used to aggregate cross-modal features in the semantic association graph through a graph convolutional network and filter the cross-modal features based on information entropy to obtain the target cross-modal feature set.

[0103] The present invention also provides an electronic device, such as... Figure 8 As shown, Figure 8 This is a schematic diagram of an embodiment of the electronic device provided by the present invention, specifically: The electronic device may include components such as a processor 801 with one or more processing cores, a memory 802 with one or more computer-readable storage media, a power supply 803, and an input unit 804. Those skilled in the art will understand that... Figure 8 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein: The processor 801 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 802, and by calling data stored in the memory 802, it performs various functions and processes data, thereby providing overall monitoring of the electronic device. Optionally, the processor 801 may include one or more processing cores; preferably, the processor 801 may integrate an application processor and a modem processor. The application processor mainly handles the operation of the storage medium, user interface, and application programs, while the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 801.

[0104] The memory 802 can be used to store software programs and modules. The processor 801 executes various functional applications and data processing by running the software programs and modules stored in the memory 802. The memory 802 may mainly include a program storage area and a data storage area. The program storage area may store application programs required for operating the storage medium and at least one function (such as sound playback function, image playback function, etc.); the data storage area may store data created according to the use of the electronic device. In addition, the memory 802 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 802 may also include a memory controller to provide the processor 801 with access to the memory 802.

[0105] The electronic device also includes a power supply 803 that supplies power to various components. Preferably, the power supply 803 can be logically connected to the processor 801 via a power management storage medium, thereby enabling functions such as charging, discharging, and power consumption management through the power management storage medium. The power supply 803 may also include one or more DC or AC power supplies, recharge storage media, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0106] The electronic device may also include an input unit 804, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0107] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 801 in the electronic device loads the executable files corresponding to the processes of one or more application programs into the memory 802 according to the following instructions, and the processor 801 runs the application programs stored in the memory 802 to realize various functions, as follows: Obtain the heterogeneous feature matrix of the real-time data stream; perform comprehensive anomaly detection on the real-time data stream based on the heterogeneous feature matrix to obtain anomalous data and normal data; perform adaptive cleaning of the anomalous data according to the heterogeneous distance of the anomalous data to obtain cleaned multi-source data, and determine the cleaned multi-source data and normal data as the target dataset; construct a semantic association graph of the target dataset; aggregate cross-modal features in the semantic association graph through a graph convolutional network, and filter the cross-modal features based on information entropy to obtain the target cross-modal feature set.

[0108] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0109] Therefore, the present invention provides a computer-readable storage medium having a computer program stored thereon, the computer program being loaded by a processor to execute the steps in any of the feature transformation methods for real-time data streams provided by the present invention. For example, the computer program, when loaded by a processor, can execute the following steps: Obtain the heterogeneous feature matrix of the real-time data stream; perform comprehensive anomaly detection on the real-time data stream based on the heterogeneous feature matrix to obtain anomalous data and normal data; perform adaptive cleaning of the anomalous data according to the heterogeneous distance of the anomalous data to obtain cleaned multi-source data, and determine the cleaned multi-source data and normal data as the target dataset; construct a semantic association graph of the target dataset; aggregate cross-modal features in the semantic association graph through a graph convolutional network, and filter the cross-modal features based on information entropy to obtain the target cross-modal feature set.

[0110] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0111] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0112] Since the computer program stored in the computer-readable storage medium can execute the steps in any of the real-time data stream feature conversion methods provided by the present invention, the beneficial effects that any of the real-time data stream feature conversion methods provided by the present invention can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0113] The foregoing has provided a detailed description of a feature conversion method, apparatus, and electronic device for real-time data streams provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A feature transformation method for real-time data streams, characterized in that, include: Obtain the heterogeneous feature matrix of the real-time data stream; Based on the heterogeneous feature matrix, a comprehensive anomaly detection is performed on the real-time data stream to obtain abnormal data and normal data. The abnormal data is adaptively cleaned based on the heterogeneous distance of the abnormal data to obtain cleaned multi-source data, and the cleaned multi-source data and the normal data are determined as the target dataset. Construct a semantic association graph of the target dataset; Cross-modal features in the semantic association graph are aggregated by a graph convolutional network, and the cross-modal features are filtered based on information entropy to obtain the target cross-modal feature set.

2. The feature transformation method for real-time data streams according to claim 1, characterized in that, The acquisition of the heterogeneous feature matrix of the real-time data stream includes: The heterogeneous attributes of the real-time data stream are obtained based on the metadata parsing engine. The real-time data stream includes at least one of structured data, semi-structured data, and unstructured data. The heterogeneous attributes include at least one of format, encoding, unit, and generation frequency. The heterogeneous feature matrix is ​​generated by using each real-time data stream as a row vector and the heterogeneous attributes as column vectors.

3. The feature transformation method for real-time data streams according to claim 1, characterized in that, Based on the heterogeneous feature matrix, a comprehensive anomaly detection is performed on the real-time data stream to obtain abnormal data, including: The heterogeneous feature matrix is ​​subjected to time anomaly detection using a dynamic time warping algorithm to obtain a dynamic time warping anomaly score. Numerical anomaly detection is performed on the heterogeneous feature matrix using the Euclidean distance algorithm to obtain an Euclidean distance anomaly score. The dynamic time warping anomaly score and the Euclidean distance anomaly score are dynamically weighted to obtain a comprehensive anomaly score for the real-time data stream, and the real-time data stream with a comprehensive anomaly score greater than a preset score threshold is identified as the anomalous data.

4. The feature transformation method for real-time data streams according to claim 3, characterized in that, The dynamic weighting of the dynamic time warping anomaly score and the Euclidean distance anomaly score includes: The first weighting coefficient of the dynamic time warping anomaly score is determined according to the data source type of the real-time data stream, and the second weighting coefficient of the Euclidean distance anomaly score is adjusted accordingly based on the first weighting coefficient. When the real-time data stream is text data, the second weighting coefficient is increased.

5. The feature transformation method for real-time data streams according to claim 1, characterized in that, The step of adaptively cleaning the abnormal data based on the heterogeneous distance of the abnormal data to obtain cleaned multi-source data includes: Calculate the heterogeneous distance between different data sources in the abnormal data based on the heterogeneous feature matrix; When the heterogeneous distance is greater than a preset distance threshold, the abnormal data is repaired by homogeneous data source interpolation to obtain the repaired abnormal data. When the heterogeneous distance is not greater than the preset distance threshold, the abnormal data is marked with missing values ​​to obtain marked abnormal data; The repaired abnormal data and the marked abnormal data are identified as the cleaned multi-source data.

6. The feature transformation method for real-time data streams according to claim 1, characterized in that, The construction of the semantic association graph of the target dataset includes: Obtain the feature vector for each target dataset; The nodes of the initial semantic association graph are determined based on the feature vectors, the edges of the initial semantic association graph are determined based on the cosine similarity between the feature vectors, and the semantic association graph is constructed based on the nodes and the edges.

7. The feature transformation method for real-time data streams according to claim 1, characterized in that, The cross-modal features in the semantic association graph are aggregated through a graph convolutional network, and the cross-modal features are filtered based on information entropy to obtain a target cross-modal feature set, including: By traversing each node of the semantic association graph through a graph convolutional network and aggregating the feature information of neighboring nodes, the cross-modal features of each node are obtained. Calculate the mutual information gain between the cross-modal features and the target variable, and determine the cross-modal features with mutual information gains higher than a preset threshold as target cross-modal features to form a target cross-modal feature set.

8. The feature transformation method for real-time data streams according to claim 1, characterized in that, After aggregating cross-modal features from the semantic association graph using a graph convolutional network and filtering the cross-modal features based on information entropy to obtain the target cross-modal feature set, the method further includes: The quality of the target cross-modal feature set is evaluated to obtain a comprehensive quality score for the target cross-modal feature set; The preset distance threshold, anomaly detection step size threshold, parameter weights of the graph convolutional network, and parameter weights of the semantic association graph are dynamically adjusted based on the quality comprehensive score. The updated target cross-modal feature set of the real-time data stream is then re-acquired until the updated target cross-modal feature set that meets the preset standard is obtained. The dimensions of quality assessment include completeness, consistency, and timeliness.

9. A feature conversion device for real-time data streams, characterized in that, include: The heterogeneous feature processing module is used to obtain the heterogeneous feature matrix of the real-time data stream; An anomaly detection module is used to perform comprehensive anomaly detection on the real-time data stream based on the heterogeneous feature matrix to obtain abnormal data and normal data; An adaptive cleaning module is used to adaptively clean the abnormal data based on the heterogeneity distance of the abnormal data, obtain cleaned multi-source data, and determine the cleaned multi-source data and the normal data as the target dataset. The semantic association module is used to construct a semantic association graph of the target dataset; The feature filtering module is used to aggregate cross-modal features in the semantic association graph through a graph convolutional network, and filter the cross-modal features based on information entropy to obtain a target cross-modal feature set.

10. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor causes the processor to perform the following steps: Obtain the heterogeneous feature matrix of the real-time data stream; Based on the heterogeneous feature matrix, a comprehensive anomaly detection is performed on the real-time data stream to obtain abnormal data and normal data. The abnormal data is adaptively cleaned based on the heterogeneous distance of the abnormal data to obtain cleaned multi-source data, and the cleaned multi-source data and the normal data are determined as the target dataset. Construct a semantic association graph of the target dataset; Cross-modal features in the semantic association graph are aggregated by a graph convolutional network, and the cross-modal features are filtered based on information entropy to obtain the target cross-modal feature set.

Images