Data association method and device for multiple data sources, electronic equipment and medium

Abstract

The application provides a data correlation method and device for multiple data sources, electronic equipment and medium. The method obtains original data from different data sources, the original data including at least two of structured data, unstructured data and time series data, performs feature extraction on the original data to obtain a data entity feature vector of a data entity, calculates a similarity probability matrix between different data entities according to the data entity feature vector, obtains a similarity threshold value according to the similarity probability matrix, establishes a directed edge between the data entities according to the similarity probability matrix and the similarity threshold value, and further constructs a data correlation graph to complete data correlation. The method provided by the application is more practical and accurate for data correlation of multiple data sources.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus, electronic device and medium for data association of multiple data sources. Background Technology

[0002] With the increasing prevalence of data-driven business models, enterprises and organizations often need to process heterogeneous data from multiple different data sources simultaneously. In existing multi-source data aggregation technologies, a common practice is to rely on a rule configuration center, where developers or data analysts manually define the metadata relationships between different data sources in advance, such as field mappings, data type conversions, filtering conditions, and join keys. This statically configured approach is feasible in scenarios where the data model is stable and the query patterns are fixed.

[0003] However, in interactive analytics scenarios, users need to explore different data combinations in real time. Traditional manual configuration methods hinder the agility of the analysis process and cannot meet the needs of rapid iterative queries. Furthermore, the lack of automated association capabilities for unstructured data limits the richness of analytical dimensions. For example, when field names in different data sources are different but semantically equivalent, traditional methods require manual mapping configuration, which is time-consuming and error-prone. In addition, related technologies are inefficient at cross-source data integration when handling multimodal data (such as a mixture of text, images, and structured data), making it difficult to meet real-time analysis requirements. Clearly, there is an urgent need for a new data association method for multiple data sources to address at least one of the above problems.

[0004] It should be noted that the above content only provides background information related to this application and does not necessarily constitute prior art. Summary of the Invention

[0005] This application provides a method, apparatus, electronic device, and medium for data association among multiple data sources, in order to solve the technical problem of being unable to establish relationships between multiple data sources.

[0006] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0007] This application provides a data association method for multiple data sources, comprising: obtaining raw data from different data sources, wherein the raw data includes at least two of structured data, unstructured data, and time-series data; performing feature extraction on the raw data to obtain data entity feature vectors of data entities; calculating a similarity probability matrix between different data entities based on the data entity feature vectors; obtaining a similarity threshold based on the similarity probability matrix; establishing directed edges between data entities based on the similarity probability matrix and the similarity threshold, thereby constructing a data association graph to complete the data association.

[0008] In one embodiment of this application, based on the aforementioned scheme, after obtaining raw data from different data sources, the raw data is preprocessed, and the preprocessed raw data replaces the original data. The preprocessing of the raw data includes: if the raw data includes structured data, then a first preprocessing is performed on the structured data, the first preprocessing including at least one of mean completion, outlier filtering, and normalization; if the raw data includes unstructured data, then a second preprocessing is performed on the unstructured data, the second preprocessing including at least one of word segmentation, lexical reconstruction, and text vectorization; if the raw data includes time-series data, then a third preprocessing is performed on the time-series data, the third preprocessing including at least one of missing value completion and outlier filtering. The preprocessing includes the first preprocessing, the second preprocessing, and the third preprocessing.

[0009] In one embodiment of this application, based on the aforementioned scheme, feature extraction is performed on the original data to obtain the data entity feature vector, including: extracting local semantic features from the original data through a constructed feature extraction network to obtain feature vectors corresponding to various types of data, wherein the feature extraction network is a two-dimensional convolutional neural network; and concatenating and fusing the feature vectors corresponding to various types of data included in the data entity to obtain the data entity feature vector.

[0010] In one embodiment of this application, based on the aforementioned scheme, a feature extraction network is constructed to extract local semantic features from the original data to obtain feature vectors corresponding to various types of data. This includes: converting the original data into input matrices corresponding to various types of data according to their types, wherein the structured data is converted into a two-dimensional matrix, the time-series data is converted into a time-series waveform matrix, and the unstructured data is converted into a word embedding matrix; inputting the input matrices corresponding to various types of data into the two-dimensional convolutional neural network, extracting local semantic features through convolutional layers, and generating feature vectors corresponding to various types of data through pooling layers, wherein the two-dimensional convolutional neural network includes the convolutional layers and the pooling layers.

[0011] In one embodiment of this application, based on the aforementioned scheme, calculating the similarity probability matrix between different data entities according to the feature vectors of the data entities includes: decoupling the feature vectors of the data entities to separate the feature vectors corresponding to various types of data; calculating the initial similarity of the feature vectors corresponding to various types of data between different data entities using similarity measurement methods adapted to various types of data; determining the similarity weight of each initial similarity through the analytic hierarchy process (AHP); performing a weighted average of each initial similarity according to the similarity weight to obtain a fused similarity; and transforming the fused similarity through a normalized exponential function to obtain the similarity probability matrix between different data entities.

[0012] In one embodiment of this application, based on the aforementioned scheme, similarity measurement methods adapted to various types of data are used to calculate the initial similarity of feature vectors corresponding to various types of data between different data entities, including: if the feature vectors corresponding to various types of data include structured feature vectors, then cosine similarity is used to calculate the initial similarity of structured feature vectors between different data entities; if the feature vectors corresponding to various types of data include unstructured feature vectors, then a weighted combination of semantic similarity and lexical similarity is used to calculate the initial similarity of unstructured feature vectors between different data entities; if the feature vectors corresponding to various types of data include temporal feature vectors, then dynamic time warping algorithm is used to calculate sequence distance, and normalization processing is used to obtain the initial similarity of temporal feature vectors between different data entities.

[0013] In one embodiment of this application, based on the aforementioned scheme, a similarity threshold is obtained according to the similarity probability matrix, and directed edges between data entities are established according to the similarity probability matrix and the similarity threshold to construct a data association graph. This includes: calculating the mean and standard deviation of all elements in the similarity probability matrix; calculating the similarity threshold based on the mean and standard deviation; using all data entities as graph nodes, traversing each element in the similarity probability matrix, where each element corresponds to the similarity probability between a first data entity and a second data entity; if the similarity probability is greater than or equal to the similarity threshold, establishing a directed edge from the first data entity to the second data entity, and using the similarity probability as the weight of the directed edge; and generating the data association graph based on all established directed edges.

[0014] This application also provides a data association device for multiple data sources, comprising: a data acquisition module for acquiring raw data from different data sources, wherein the raw data includes at least two of structured data, unstructured data, and time-series data; a feature extraction module for extracting features from the raw data to obtain data entity feature vectors of data entities; a similarity calculation module for calculating a similarity probability matrix between different data entities based on the data entity feature vectors; and a data association module for obtaining a similarity threshold based on the similarity probability matrix, establishing directed edges between data entities based on the similarity probability matrix and the similarity threshold, thereby constructing a data association graph to complete the data association.

[0015] This application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the data association method for multiple data sources as described in any of the above embodiments.

[0016] This application also provides a computer-readable storage medium storing a computer program that is executed by a processor to implement the data association method for multiple data sources as described in any of the above embodiments.

[0017] The beneficial effects of this application are as follows: Based on feature extraction and similarity calculation, it can identify implicit associations with different field names but semantic equivalence, avoiding the tediousness and errors of manual mapping configuration; through a dynamic similarity threshold solution mechanism and a self-learning weight matrix, it automatically adapts to data source change scenarios, significantly reducing manual configuration costs and improving the efficiency of multi-source data association; the constructed feature association graph supports real-time analysis and complex association queries, providing an efficient underlying association model for subsequent processing such as data aggregation, cleaning, and analysis, which can meet the needs of enterprises for real-time data-driven decision-making. This application breaks through the limitations of traditional rule configuration from a technical perspective, providing a universal solution for multi-source data association and integration, and has significant engineering application value and technological innovation.

[0018] In addition, this application effectively supports unified feature representation of multimodal data such as structured data, unstructured data, and time-series data through differentiated preprocessing and feature extraction mechanisms, solving the problem of insufficient processing capability of traditional methods for unstructured data.

[0019] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0021] In the attached diagram: Figure 1 This is a schematic diagram illustrating an exemplary system architecture as shown in an exemplary embodiment of this application; Figure 2 This is a schematic flowchart illustrating a data association method for multiple data sources, as shown in an exemplary embodiment of this application. Figure 3 This is a schematic diagram illustrating the structure of a data association method for multiple data sources, as shown in an exemplary embodiment of this application; Figure 4 This is a schematic flowchart illustrating a data association apparatus for multiple data sources, as shown in an exemplary embodiment of this application. Figure 5 This is a block diagram illustrating a data association apparatus for multiple data sources, as shown in another exemplary embodiment of this application; Figure 6 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0022] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0023] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. The drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the shape, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0024] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the present application. However, it will be apparent to those skilled in the art that embodiments of the present application may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the present application.

[0025] First, it's important to clarify that multi-source data refers to data collections from multiple different sources, possessing diverse formats and structures. Common types include structured data (such as database tables), semi-structured data (such as XML and JSON), and unstructured data (such as text, images, and videos). Its core characteristics are diverse sources, heterogeneous structures, dynamic changes, and complex processing. By collecting, cleaning, integrating, and analyzing this data, a more comprehensive and accurate information view can be formed, thereby improving decision-making quality and system performance. The key technology is data fusion, involving multiple processing layers such as the data layer, feature layer, and decision layer. This technology has been widely applied in fields such as weather forecasting, smart cities, precision agriculture, environmental monitoring, traffic management, and business intelligence, serving as a crucial foundation for driving digital transformation and intelligent upgrading.

[0026] In mathematics, especially in probability theory and related fields, the normalized exponential function, or softmax function, is a generalization of the logistic function. It can compress a K-dimensional vector z containing arbitrary real numbers into another K-dimensional real vector σ(z), such that each element is in the range (0,1) and the sum of all elements is 1.

[0027] Dynamic Time Warping (DTW) is an algorithm for measuring the similarity of time series, based on the concept of dynamic programming. This algorithm constructs a distance matrix between two curves and uses path constraints with slopes ranging from 0.5 to 2 to find the optimal path with the minimum cumulative distance. The basic metric is typically Euclidean distance. In isolated word speech recognition, it effectively solves the template matching problem with varying pronunciation lengths and has the advantage of low training computation.

[0028] The Jaccard coefficient, also known as the Jaccard similarity coefficient, is a statistical indicator used to measure the similarity between two sets. It is defined as the ratio of the size of the intersection to the size of the union of the two sets, calculated using the formula J(A,B)=|A∩B| / |A∪B|, with values ​​ranging from 0 to 1. The corresponding Jaccard distance is obtained by subtracting the similarity coefficient from 1, and is used to characterize the dissimilarity between the sets.

[0029] With the rise of agile analytics needs such as interactive data analysis and real-time exploratory queries, traditional methods have revealed significant shortcomings. On the one hand, in interactive analytics scenarios, users need to try different data combinations and association methods in real time. However, manually configuring mapping rules is cumbersome and has a long response time, severely hindering the agility of the analysis process and making it difficult to meet the needs of rapid iterative queries. On the other hand, when field names in different data sources are different but semantically equivalent, traditional methods still require manual identification and configuration of mapping relationships, which is not only time-consuming and error-prone, but also extremely costly to maintain, especially when the number of fields is large or the data source changes dynamically.

[0030] Furthermore, the related technologies have relatively weak capabilities in automating the association of unstructured data. Most systems can only handle explicit field matching between structured tabular data, lacking effective semantic understanding and association mechanisms for unstructured or semi-structured data such as text, logs, and images. This results in a large amount of valuable information being unable to be integrated into the cross-source analysis process, limiting the richness of analytical dimensions. Moreover, in multimodal data scenarios (such as a mixture of text, images, and structured data), the related technologies lack unified feature representations and cross-modal association models. Data from different modalities are often processed in isolation, leading to low integration efficiency and making it difficult to support the real-time or near-real-time multi-source data fusion analysis requirements.

[0031] In summary, how to achieve automatic, efficient, and semantically interpretable association between multi-source heterogeneous data (especially including unstructured / semi-structured and multimodal data) without relying on cumbersome manual configuration has become an urgent technical problem to be solved.

[0032] To address the aforementioned problems, embodiments of this application propose a data association method for multiple data sources, a data association device for multiple data sources, an electronic device, and a computer-readable storage medium, which will be described in detail below.

[0033] Figure 1 This is a schematic diagram illustrating an exemplary system architecture as shown in an exemplary embodiment of this application.

[0034] Reference Figure 1As shown, the system architecture may include a data acquisition device 101 and a computer device 102. The computer device 102 may be at least one of a desktop graphics processing unit (GPU) computer, a GPU computing cluster, or a neural network computer. The data acquisition device 101 is used to acquire raw data from different data sources. The raw data includes at least two of structured data, unstructured data, and time-series data. In this embodiment, after acquiring the above data, the data acquisition device 101 provides it to the computer device 102 for processing. Those skilled in the art can use the computer device 102 to extract features from the raw data, obtain data entity feature vectors, calculate a similarity probability matrix between different data entities based on the data entity feature vectors, obtain a similarity threshold based on the similarity probability matrix, establish directed edges between data entities based on the similarity probability matrix and the similarity threshold, and then construct a data association graph to complete the data association. It should be noted that the data acquisition device 101 and computer device 102 provided in this embodiment are merely examples and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0035] It should be noted that the data association method for multiple data sources provided in this application embodiment is generally executed by computer device 102, and correspondingly, the data association device for multiple data sources is generally disposed in computer device 102.

[0036] Figure 2 This is a schematic flowchart illustrating a data association method for multiple data sources, as shown in an exemplary embodiment of this application. This data association method for multiple data sources can be executed by a computing processing device, which may be... Figure 1 The computer device 102 shown is illustrated. (Refer to...) Figure 2 As shown, the data association method for multiple data sources includes at least steps S210 to S240, which are described in detail below: In step S210, raw data is obtained from different data sources.

[0037] The original data includes at least two of the following: structured data, unstructured data, and time-series data.

[0038] In this embodiment, configuration item information and metadata information of the underlying databases corresponding to multiple data sources are obtained to obtain the raw data. Through a unified loading engine, heterogeneous raw data is loaded from multiple data sources such as text data sources, IT (Information Technology) systems, and OT (Operational Technology) equipment.

[0039] Figure 3This is a schematic diagram illustrating the structure of a data association method for multiple data sources, as shown in an exemplary embodiment of this application. Specifically, as... Figure 3 As shown, in this embodiment, the acquisition of raw data from different data sources is accomplished through execution engines adapted to various types of data sources. These execution engines are adapted to data sources including relational databases (e.g., MySQL), time-series databases (e.g., IOT_DB), and text data sources. The acquired raw data includes, but is not limited to, IT data (structured data), OT data (time-series data), and text (descriptive) data (unstructured data). Specifically, IT data can come from structured data sources such as business systems and relational databases; OT data can come from time-series data sources such as sensors and control systems in industrial settings; and text data can come from unstructured data sources such as technical documents and log files. It can be understood that IT data refers to business and information flow data generated, processed, and managed by information technology systems, such as ERP (Enterprise Resource Planning) system records, financial transactions, customer relationships, emails, and office documents; OT data refers to physical equipment and production process data collected in real time by operational technology systems, such as sensor temperature, equipment vibration, valve opening, and production cycle time.

[0040] It should be noted that the time-series data in this embodiment belongs to structured data in terms of storage format. However, since its preprocessing method is significantly different from that of static structured data, this application describes the two as independent data types to clarify their respective preprocessing operations. In other words, in this application, structured data specifically refers to non-time-series static structured data, while time-series data refers to structured data with time-series characteristics. The two have significant differences in preprocessing methods, and therefore are described separately.

[0041] Typically, each data source has a single data type (e.g., relational databases output structured data, time-series databases output time-series data, and text files output unstructured data). Therefore, each data source usually provides only one type.

[0042] In one embodiment of this application, after obtaining raw data from different data sources, the raw data is preprocessed, and the preprocessed raw data replaces the original data. The preprocessing of the raw data includes: if the raw data includes structured data, a first preprocessing is performed on the structured data, the first preprocessing including at least one of mean completion, outlier filtering, and normalization; if the raw data includes unstructured data, a second preprocessing is performed on the unstructured data, the second preprocessing including at least one of word segmentation, lexical reconstruction, and text vectorization; if the raw data includes time-series data, a third preprocessing is performed on the time-series data, the third preprocessing including at least one of missing value completion and outlier filtering. The preprocessing includes the first, second, and third preprocessing steps.

[0043] In this embodiment, the raw data undergoes preprocessing, including but not limited to data cleaning, format conversion, and normalization. Specifically, structured data processing employs one or more of the following: mean completion, outlier filtering, and normalization. Unstructured data processing employs one or more of the following: text decomposition (i.e., word segmentation), word root reconstruction (i.e., lexical reconstruction), and numerical vector conversion (i.e., text vectorization). Time-series data preprocessing employs one or more of the following: nearest-neighbor interpolation for missing value completion and exponential smoothing for outlier filtering (also known as anomaly detection based on exponential smoothing residuals). It is understood that exponential smoothing is a weighted moving average prediction and smoothing technique designed to highlight trends and reduce noise.

[0044] Specifically, IT data is preprocessed using at least one of the following methods: mean completion, outlier filtering, and mapping data to [0,1] through normalization. Text data is preprocessed using at least one of the following methods: text decomposition, word root restoration, and numerical vector transformation. Time-series data is preprocessed using at least one of the following methods: missing value completion using nearest neighbor interpolation and outlier filtering using exponential smoothing. Preprocessing the original data makes the data used for subsequent feature extraction more complete, reduces the impact of outlier noise, and improves the reliability of feature extraction.

[0045] In step S220, feature extraction is performed on the original data to obtain the data entity feature vector of the data entity.

[0046] In one embodiment of this application, a feature extraction network is constructed to extract local semantic features from the original data to obtain feature vectors corresponding to various types of data. The feature extraction network is a two-dimensional convolutional neural network. The feature vectors corresponding to various types of data included in the data entity are concatenated and fused to obtain the data entity feature vector.

[0047] In one embodiment of this application, a feature extraction network is constructed to extract local semantic features from the original data to obtain feature vectors corresponding to various types of data. This includes: converting the original data into input matrices corresponding to various types of data according to their types, wherein structured data is converted into two-dimensional matrices, time-series data is converted into time-series waveform matrices, and unstructured data is converted into word embedding matrices; inputting the input matrices corresponding to various types of data into a two-dimensional convolutional neural network, extracting local semantic features through convolutional layers, and generating feature vectors corresponding to various types of data through pooling layers. The two-dimensional convolutional neural network includes convolutional layers and pooling layers.

[0048] In this embodiment, the feature extraction network adopts a two-dimensional convolutional neural network consisting of N convolutional layers, where N≥1; the original data is input into the two-dimensional convolutional neural network, local semantic features are extracted through the convolutional layers, and feature vectors of a preset dimension are generated through the pooling layers; feature vectors of multiple types of data are merged by feature concatenation to generate a data entity feature vector of a unified dimension to represent the data entity features.

[0049] Specifically, a two-dimensional convolutional neural network consisting of three convolutional layers is employed. Each convolutional layer is configured with 64 3×3 convolutional kernels and uses the ReLU activation function (Linear rectification function, also known as the ReLU function) to capture local semantic association features, i.e., local semantic features. First, structured data is converted into a two-dimensional matrix, time-series data is converted into a time-series waveform matrix, and unstructured text data is converted into a word embedding matrix, which are then input into the two-dimensional convolutional neural network. The first convolutional layer extracts basic features, and a 2×2 max pooling layer compresses the feature dimension to half its original size. The second convolutional layer further extracts mid-level features, and another pooling layer reduces the dimensionality. The third convolutional layer extracts deep abstract features, and finally, a global average pooling layer generates a 128-dimensional fixed-dimensional feature vector, thus obtaining feature vectors corresponding to various data types. If the data entity contains multiple data types, in other words, for the feature vectors of multiple data types, a concatenation and fusion strategy is adopted to connect the feature vectors corresponding to different data types end-to-end in a preset order, forming a unified representation vector containing multi-dimensional information, i.e., the data entity feature vector. It is understood that the preset order can be set according to requirements, and this application does not impose any restrictions on it. If a data entity includes only one type of data, there is no need to concatenate and fuse the feature vectors corresponding to that type of data; the feature vectors corresponding to that type of data can be directly used to represent the features of the data entity. Alternatively, if a data entity includes only one type of data, the feature vectors corresponding to that type of data are concatenated and fused to generate a data entity feature vector to represent the features of the data entity. The data type can be determined based on the metadata information of the data source, field types, data source type identifiers, or classification results from the preprocessing stage. The method of determining the data type is merely illustrative, and this application does not impose any restrictions on it.

[0050] In step S230, the similarity probability matrix between different data entities is calculated based on the feature vectors of the data entities.

[0051] In one embodiment of this application, calculating the similarity probability matrix between different data entities based on the feature vectors of data entities includes: decoupling the feature vectors of data entities to separate the feature vectors corresponding to various types of data; calculating the initial similarity of the feature vectors corresponding to various types of data between different data entities using similarity measurement methods adapted to various types of data; determining the similarity weight of each initial similarity through the analytic hierarchy process (AHP); performing a weighted average of each initial similarity based on each similarity weight to obtain the fused similarity; and transforming the fused similarity through a normalized exponential function to obtain the similarity probability matrix between different data entities.

[0052] In one embodiment of this application, similarity measurement methods adapted to various types of data are used to calculate the initial similarity of feature vectors corresponding to various types of data between different data entities. This includes: if the feature vectors corresponding to various types of data include structured feature vectors, then cosine similarity is used to calculate the initial similarity of structured feature vectors between different data entities; if the feature vectors corresponding to various types of data include unstructured feature vectors, then a weighted combination of semantic similarity and lexical similarity is used to calculate the initial similarity of unstructured feature vectors between different data entities; if the feature vectors corresponding to various types of data include temporal feature vectors, then dynamic time warping algorithm is used to calculate sequence distance, and normalization processing is used to obtain the initial similarity of temporal feature vectors between different data entities.

[0053] In this embodiment, data entities are matched by data entity features, and the extracted data entity feature vectors are used to calculate the similarity probability matrix between data entities.

[0054] In some embodiments, single-feature similarity, i.e., initial similarity, is calculated for the feature vector of each dimension, i.e., the feature vector corresponding to each type of data; a feature weight matrix is ​​constructed by multi-feature fusion and weighting. The weights of each feature dimension, i.e., the similarity weights of each initial similarity, are determined using the analytic hierarchy process (AHP). A weighted average is then used to fuse the single-feature similarities to obtain the fused similarity. Equation (1) in, For fusion similarity, n is the total number of features (i.e., the number of single feature similarities participating in fusion), and i is the feature index (from 1 to n). The weight of the i-th feature dimension is also the weight of the i-th feature dimension. The similarity weights of the initial similarity values ​​(determined by the analytic hierarchy process). For the first The similarity of the th feature is also known as the th feature similarity. Initial similarity.

[0055] The similarity probability is transformed by using the Softmax function, converting the fused similarity into a probability value in the 0-1 range, as shown in equation (2): Equation (2) in, The transformed similarity probability (range 0-1), where e is the natural constant. To fuse similarity, i.e., in equation (1) .

[0056] Output the similarity probability matrix between data entities Where P is the similarity probability matrix, This indicates that the matrix is ​​an m-row, m-column real matrix, where m is the total number of data entities.

[0057] In other embodiments, data entity features are matched using data entity feature vectors, and the extracted data entity feature vectors are used to calculate the similarity probability between data entities through fusion similarity calculation. Specifically, the following operations are performed: First, modal decoupling is performed on the unified representation vector, i.e., the feature vector of the data entity, to separate the structured feature vector. (Corresponding IT data), time-series feature vector (Corresponding to OT data) and unstructured feature vectors (text feature vectors are used in this embodiment). The text data is described below, and each represents a deep feature representation of a different data modality. Decoupling can be performed based on the pre-defined modal order and dimensional information during concatenation. It should be understood that this decoupling method is merely illustrative and is not intended to limit the scope of the application.

[0058] Furthermore, for IT data, the corresponding feature vectors are structured feature vectors, and cosine similarity is used to measure the distribution consistency of data-type features: Equation (3) in, For data entities The initial similarity of the structured feature vectors between them. and Data entities The structured feature vectors.

[0059] For text data, the initial similarity of text feature vectors between different data entities is obtained by combining semantic and lexical layer similarity: Equation (4) in, For data entities The initial similarity between the text feature vectors is calculated, where α is the semantic similarity weight (which can take a value of 0.7) and 1-α is the lexical similarity weight. This is the cosine similarity of the word embedding vectors, i.e., the cosine similarity of the two text feature vectors (based on semantic similarity of word embeddings). and Data entities The text feature vector, The Jaccard coefficient after text segmentation is the Jaccard coefficient of the text segmentation set, which is also the similarity based on the degree of word overlap (lexical similarity). Data entities A collection of text segmentations.

[0060] To address the dynamic characteristics of time-series (OT) data, the Dynamic Time Warping (DTW) algorithm is used to calculate sequence similarity, which is then normalized to obtain: Equation (5) in, For data entities The initial similarity between the temporal feature vectors of the two. and Data entities The temporal feature vector, The DTW distance, or sequence distance, represents the distance between two time-series feature vectors.

[0061] Furthermore, construct the feature weight matrix. ,in, The similarity weights corresponding to the OT data (i.e., the similarity weights corresponding to the time sequence feature vectors). The similarity weights corresponding to IT data (i.e., the similarity weights corresponding to structured feature vectors). The similarity weights corresponding to the text data (i.e., the similarity weights corresponding to the text feature vectors) are determined using the analytic hierarchy process (AHP). The similarity weights for each modal feature, i.e., each initial similarity, are determined through AHP.

[0062] Establish a judgment matrix This reflects the relative importance of feature vectors corresponding to different types of data, such as:

[0063] The rows (and columns) of matrix A correspond in the following order: structured feature vector, temporal feature vector, and textual feature vector. The element a in matrix A... pq This represents the importance ratio of the feature vector corresponding to the p-th type of data to the feature vector corresponding to the q-th type of data. This matrix shows that structured feature vectors are more important than time-series feature vectors (3:1), structured feature vectors are significantly more important than text feature vectors (5:1), and so on, which will not be elaborated further here.

[0064] The weight vector is obtained by calculating the eigenvector corresponding to the largest eigenvalue of matrix A, and then performing normalization and consistency checks. Weighted average fusion of single-feature similarity is used: Equation (6) in, This represents the fusion similarity (weighted average result) between data entities i and j, where k is the modality index, and k=1,2,3 correspond to the structured feature vector, temporal feature vector, and text feature vector, respectively. The weight of the k-th mode is also the similarity weight of each initial similarity (determined by the analytic hierarchy process). The single feature similarity of the k-th modality (e.g.) , , That is, the initial similarity of the feature vectors corresponding to various types of data between data entities i and j.

[0065] The fused similarity is mapped to a probability distribution using the Softmax function, as shown in equation (7): Equation (7) in, The element in the i-th row and j-th column of the similarity probability matrix P represents the association probability, i.e., the similarity probability, between data entity i and data entity j, providing a quantitative basis for subsequent dynamic threshold filtering. The similarity probability matrix represents the fusion similarity between data entities i and j. , where m is the total number of data entities.

[0066] In step S240, a similarity threshold is obtained based on the similarity probability matrix. Directed edges between data entities are established based on the similarity probability matrix and the similarity threshold, thereby constructing a data association graph to complete the data association.

[0067] In one embodiment of this application, a similarity threshold is obtained based on a similarity probability matrix, and directed edges between data entities are established based on the similarity probability matrix and the similarity threshold to construct a data association graph. This includes: calculating the mean and standard deviation of all elements in the similarity probability matrix; calculating the similarity threshold based on the mean and standard deviation; using all data entities as graph nodes, traversing each element in the similarity probability matrix, where each element corresponds to the similarity probability between a first data entity and a second data entity; if the similarity probability is greater than or equal to the similarity threshold, establishing a directed edge from the first data entity to the second data entity, and using the similarity probability as the weight of the directed edge; and generating a data association graph based on all established directed edges.

[0068] In this embodiment, the mean of all elements in the statistical similarity probability matrix is ​​calculated. with standard deviation According to the mean with standard deviation Set a dynamic similarity threshold, refer to formula (8): Equation (8) Where τ is the similarity threshold, The mean of all elements in the similarity probability matrix is ​​denoted as . The standard deviation of all elements in the similarity probability matrix. This is the adjustment coefficient.

[0069] Use all data entities as graph nodes If the nodes corresponding to data entities i and j and The similarity probability between them is greater than or equal to the similarity threshold, i.e. Then at node and Establish directed edges between The weight of this directed edge is The final result is a weighted directed data relationship graph. ,in The set of edges that satisfy the threshold condition.

[0070] In another embodiment, filtering data entities based on a dynamic similarity threshold and constructing a corresponding data association graph includes the following steps: Perform statistical analysis on the similarity probability matrix P and calculate the mean of all elements in the matrix. with standard deviation A dynamic similarity threshold is set based on the data distribution characteristics. This threshold mechanism can dynamically adapt to the association strength requirements of different data sources according to the overall distribution of data similarity, avoiding the problem of overly dense or sparse associations caused by a fixed threshold.

[0071] Furthermore, with data entity sets Each data entity in the graph is treated as a node, and each element in the similarity probability matrix P is traversed. ,element This corresponds to the similarity probability between data entity i and data entity j. When the similarity exceeds the threshold, at the node and Establish directed edges between them The weight of the edge is assigned to represent the entity. and The association confidence score is equivalent to the similarity probability. This ultimately generates a weighted directed data association graph. ,in, The set of edges that satisfy the threshold condition, where τ is the similarity threshold.

[0072] Figure 4 This is a schematic flowchart illustrating a data association apparatus for multiple data sources, as shown in an exemplary embodiment of this application. (Refer to...) Figure 4As shown, in one embodiment of this application, the data association device for multiple data sources includes a data acquisition module for acquiring raw data from different data sources, including OT data (also known as time series data), IT data, and text data; a data preprocessing module for performing data preprocessing according to the type of the raw data, including data completion, outlier handling, and numerical vector transformation; a feature extraction module for automatically learning multi-level features of the data based on the preprocessed data set, generating data entity feature vectors, and storing them in a feature pool to form a feature set; and a feature matching module for calculating the similarity of data entity feature vectors by fusing multiple similarity algorithms to construct a data association graph. It is understood that the data association device for multiple data sources provided in this embodiment and the data association method for multiple data sources provided in the above embodiments belong to the same concept. The specific methods by which each module and unit performs operations have been described in detail in the method embodiments and will not be repeated here. In practical applications, the data association device for multiple data sources provided in the above embodiments can allocate the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not a limitation here.

[0073] This application effectively supports unified feature representation of multimodal data, including structured, unstructured, and time-series data, through differentiated preprocessing and feature extraction mechanisms, solving the problem of insufficient processing capabilities for unstructured data by traditional methods. Based on deep feature extraction and fusion similarity calculation, it can identify implicit associations such as those with different field names but semantic equivalence, avoiding the tediousness and errors of manual mapping configuration. Through a dynamic threshold mechanism and self-learning weight matrix, it automatically adapts to data source change scenarios, significantly reducing manual configuration costs and improving the efficiency of multi-source data association. The constructed feature relationship topology graph supports real-time analysis and complex association queries, providing an efficient underlying association model for subsequent processing such as data aggregation, cleaning, and analysis, meeting the needs of enterprises for real-time data-driven decision-making. Technically, it breaks through the limitations of traditional rule configuration, providing a universal solution for multi-source data association and integration, with significant engineering application value and technological innovation.

[0074] Figure 5 This is a block diagram illustrating a data association apparatus for multiple data sources, as shown in another exemplary embodiment of this application. This apparatus can be applied to… Figure 1 The implementation environment shown is specifically configured in computer device 102. This device can also be applied to other exemplary implementation environments and specifically configured in other devices. This embodiment does not limit the implementation environment to which this device is applicable.

[0075] like Figure 5As shown, the exemplary data association device for multiple data sources includes: a data acquisition module 510, a feature extraction module 520, a similarity calculation module 530, and a data association module 540.

[0076] The data acquisition module 510 is used to acquire raw data from different data sources, including at least two types of structured data, unstructured data, and time-series data; the feature extraction module 520 is used to extract features from the raw data to obtain data entity feature vectors; the similarity calculation module 530 is used to calculate the similarity probability matrix between different data entities based on the data entity feature vectors; and the data association module 540 is used to obtain a similarity threshold based on the similarity probability matrix, establish directed edges between data entities based on the similarity probability matrix and the similarity threshold, and then construct a data association graph to complete the data association.

[0077] It should be noted that the data association device for multiple data sources provided in the above embodiments and the data association method for multiple data sources provided in the above embodiments belong to the same concept. The specific ways in which each module and unit performs operations have been described in detail in the method embodiments, and will not be repeated here. In practical applications, the data association device for multiple data sources provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above, and this is not a limitation here.

[0078] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement a data association method for multiple data sources as described in any of the above embodiments.

[0079] Figure 6 A schematic diagram of a computer system suitable for implementing the embodiments of this application is shown. It should be noted that... Figure 6 The computer system 600 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of the embodiments of this application.

[0080] like Figure 6As shown, the computer system 600 includes a Central Processing Unit (CPU) 601, which can perform various appropriate actions and processes, such as executing the methods provided in the various embodiments described above, based on a program stored in Read-Only Memory (ROM) 602 or a program loaded from Storage Section 608 into Random Access Memory (RAM) 603. The RAM 603 also stores various programs and data required for system operation. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An Input / Output (I / O) interface 605 is also connected to the bus 604.

[0081] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0082] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit (CPU) 601, it performs the various functions defined in this application.

[0083] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0084] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0085] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0086] Another aspect of this application provides a computer-readable storage medium storing a computer program that is executed by a processor to implement a data association method for multiple data sources as described in any of the above embodiments. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not incorporated into the electronic device.

[0087] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0088] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data association method for multiple data sources provided in the various embodiments described above.

[0089] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0090] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0091] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A data association method for multiple data sources, characterized in that, include: Raw data is obtained from different data sources, including at least two of the following: structured data, unstructured data, and time-series data. Feature extraction is performed on the original data to obtain the data entity feature vector of the data entity; Calculate the similarity probability matrix between different data entities based on the feature vectors of the data entities; A similarity threshold is obtained based on the similarity probability matrix. Directed edges between data entities are established based on the similarity probability matrix and the similarity threshold, thereby constructing a data association graph to complete the data association.

2. The data association method for multiple data sources according to claim 1, characterized in that, After obtaining raw data from different data sources, the raw data is preprocessed, and the preprocessed raw data is used to replace the original data. The preprocessing of the raw data includes: If the original data includes the structured data, then the structured data undergoes a first preprocessing, which includes at least one of mean completion, outlier filtering, and normalization. If the original data includes the unstructured data, then the unstructured data undergoes a second preprocessing, which includes at least one of word segmentation, lexical reconstruction, and text vectorization. If the original data includes the time series data, then the time series data is subjected to a third preprocessing, the third preprocessing including at least one of missing value completion and outlier filtering, the preprocessing including the first preprocessing, the second preprocessing and the third preprocessing.

3. The data association method for multiple data sources according to claim 1, characterized in that, Feature extraction is performed on the original data to obtain the data entity feature vector, including: The original data is subjected to local semantic feature extraction by a constructed feature extraction network to obtain feature vectors corresponding to various types of data. The feature extraction network is a two-dimensional convolutional neural network. The feature vectors corresponding to the various types of data included in the data entity are concatenated and fused to obtain the feature vector of the data entity.

4. The data association method for multiple data sources according to claim 3, characterized in that, The original data is subjected to local semantic feature extraction using a constructed feature extraction network to obtain feature vectors corresponding to various types of data, including: The original data is converted into input matrices corresponding to various data types according to their types, wherein the structured data is converted into a two-dimensional matrix, the time series data is converted into a time series waveform matrix, and the unstructured data is converted into a word embedding matrix. The input matrices corresponding to various types of data are respectively input into the two-dimensional convolutional neural network. Local semantic features are extracted through the convolutional layer, and feature vectors corresponding to various types of data are generated through the pooling layer. The two-dimensional convolutional neural network includes the convolutional layer and the pooling layer.

5. The data association method for multiple data sources according to any one of claims 1 to 4, characterized in that, Calculate the similarity probability matrix between different data entities based on the feature vectors of the data entities, including: The feature vectors of the data entities are decoupled to separate the feature vectors corresponding to various types of data; We employ similarity measurement methods adapted to different data types to calculate the initial similarity of feature vectors corresponding to various data types between different data entities. The similarity weights of each initial similarity are determined by the analytic hierarchy process (AHP), and the initial similarities are weighted and averaged according to their respective weights to obtain the fused similarity. The fused similarity is transformed using a normalized exponential function to obtain the similarity probability matrix between different data entities.

6. The data association method for multiple data sources according to claim 5, characterized in that, We employ similarity measurement methods adapted to different data types to calculate the initial similarity of feature vectors corresponding to various data types between different data entities, including: If the feature vectors corresponding to various types of data include structured feature vectors, then cosine similarity is used to calculate the initial similarity of structured feature vectors between different data entities. If the feature vectors corresponding to various types of data include unstructured feature vectors, then a weighted combination of semantic similarity and lexical similarity is used to calculate the initial similarity of unstructured feature vectors between different data entities. If the feature vectors corresponding to various types of data include time-series feature vectors, then the dynamic time warping algorithm is used to calculate the sequence distance, and the initial similarity of time-series feature vectors between different data entities is obtained through normalization.

7. The data association method for multiple data sources according to any one of claims 1 to 4, characterized in that, A similarity threshold is obtained based on the similarity probability matrix. Directed edges are established between data entities based on the similarity probability matrix and the similarity threshold, thereby constructing a data association graph, including: Calculate the mean and standard deviation of all elements in the similarity probability matrix, and calculate the similarity threshold based on the mean and standard deviation; Using all data entities as graph nodes, traverse each element in the similarity probability matrix, where each element corresponds to the similarity probability between the first data entity and the second data entity; If the similarity probability is greater than or equal to the similarity threshold, then a directed edge is established from the first data entity to the second data entity, and the similarity probability is used as the weight of the directed edge. Generate the data association graph based on all the established directed edges.

8. A data association device for multiple data sources, characterized in that, include: A data acquisition module is used to acquire raw data from different data sources, wherein the raw data includes at least two of the following: structured data, unstructured data, and time-series data. The feature extraction module is used to extract features from the original data to obtain the data entity feature vector of the data entity; The similarity calculation module is used to calculate the similarity probability matrix between different data entities based on the feature vectors of the data entities; The data association module is used to obtain a similarity threshold based on the similarity probability matrix, establish directed edges between data entities based on the similarity probability matrix and the similarity threshold, and then construct a data association graph to complete the data association.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the data association method for multiple data sources as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the data association method for multiple data sources as described in any one of claims 1 to 7.

Images