Multi-source data integration method and system for computer software data interface

Abstract

The invention relates to the technical field of data processing, and provides a multi-source data integration method and system for a computer software data interface, and the method comprises the steps: collecting multi-source data, and obtaining interface information; constructing a structural feature vector of the interface; obtaining a content feature vector of the interface; constructing a call graph and obtaining a context feature vector of the interface; clustering to obtain a plurality of structure clusters; respectively obtaining a structure distance weight and a content distance weight between the interfaces; obtaining a context distance weight based on an edge weight in the call graph, and adjusting to obtain a structure clustering weight, a content clustering weight and a context clustering weight; obtaining a multi-modal feature distance between the interfaces, and clustering the internal interfaces of the structure cluster to obtain a plurality of interface clusters; and judging and merging to obtain a plurality of integration clusters, and carrying out multi-source data integration. The method aims at solving the problem that semantic mismatch is easily caused due to the fact that only structural features are considered in multi-source data clustering analysis.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and specifically to a method and system for integrating multi-source data through a computer software data interface. Background Technology

[0002] With the increasing complexity of information systems and the diversification of business needs, data from a single source is no longer sufficient to support intelligent analysis and decision-making in complex scenarios. Many industry software programs are often composed of multiple independent modules or heterogeneous systems, with significant differences in data interface formats, transmission protocols, encoding standards, data granularity, and update frequencies between these modules. This makes the integration and unified modeling of multi-source data a core challenge for system intelligence. With the development of machine learning technologies, there is an increasing use of feature extraction and clustering algorithms to aggregate or classify the feature spaces of data from different sources. This is typically used for similarity grouping of integrated data, thereby discovering potential semantic relationships between data and assisting in the integration of data patterns.

[0003] However, existing clustering methods typically rely on Euclidean distance or cosine similarity in a single numerical space. When data interfaces contain multimodal features such as structural information (e.g., field hierarchy, naming semantics, data type), content information (e.g., field value distribution, update frequency), and contextual metadata (e.g., interface call relationships, dependency paths), there may be multi-source data with some interfaces that are structurally similar but semantically different, or with large structural differences but similar semantics. Traditional distance metrics are difficult to characterize the relationships in these dimensions simultaneously, causing clustering results to be biased towards high-dimensional or high-variance modal features, thus leading to semantic mismatch and unstable cluster partitioning. Summary of the Invention

[0004] This invention provides a method and system for integrating multi-source data from computer software data interfaces to address the problem that existing multi-source data clustering analysis, which only considers structural features, easily leads to semantic mismatch. The specific technical solution adopted is as follows: This invention proposes a method for integrating multi-source data from a computer software data interface, the method comprising the following steps: Collect data from multiple sources and obtain interface information to get the field content of each interface; Based on the hierarchical distribution of fields in the interface's field content, construct the interface's structural feature vector; analyze the value distribution and semantics of the interface's field content to obtain the interface's content feature vector; based on the number of calls between interfaces, the amount of data transmitted, and the call latency, obtain the edge weights of the calls between interfaces, and then construct the call graph to obtain the interface's context feature vector. Based on the similarity of structural feature vectors of each interface and the differences in the hierarchical distribution of fields in the interface's field content, several structural clusters are obtained. Based on the differences in interface field hierarchy and type, as well as interface semantic similarity, structural distance weights and content distance weights between interfaces are obtained respectively. Context distance weights are obtained based on the edge weights in the call graph. According to the deviation between the content feature vectors and context feature vectors of interfaces within a structural cluster, the structural cluster weights, content cluster weights, and context cluster weights are adjusted. In this way, the multimodal feature distance between interfaces is obtained, and the interfaces within the structural cluster are clustered to obtain several interface clusters. Analyze the multimodal feature distance between interfaces in different interface clusters, determine the merged clusters, and perform multi-source data integration based on the field content of each interface in the integrated clusters.

[0005] Optionally, the specific methods for constructing the structural feature vector of the interface include: For all fields in the interface, a tree structure encoding model based on hierarchical semantics is used to construct the model, and the nodes corresponding to each field are obtained. For any node, the similarity between the field name of the node and the field name of its parent node is obtained. The product of the obtained similarity and the path depth of the node and the result is normalized and used as the parent node association degree of the node. Record the number of child nodes and data type encoding of the nodes corresponding to each field, and use the parent node correlation degree, the number of child nodes and data type encoding as the field vector of the nodes corresponding to each field; The field vectors of the nodes corresponding to each field in any interface are recursively aggregated by the Tree-LSTM model, and the vector of the interface is obtained by outputting through the root node, which serves as the structural feature vector of the interface.

[0006] Optionally, the specific method for obtaining the content feature vector of the interface includes: For any interface's field content, the semantic vectors of the naming and annotations of each field content are extracted using the BERT model; statistical features are obtained from the value distribution in the field content, and value vectors are represented by combining the word vectors of the values ​​to obtain the value vector; the semantic vector and the value vector are fused together to serve as the content feature vector of the interface.

[0007] Optionally, the specific method for obtaining the edge weights of calls between interfaces, and then constructing a call graph and obtaining the context feature vectors of the interfaces, includes: For the interface and interfaces Get the interface Calling the interface Number of times, get interface Each time the interface is called The amount of data transferred is calculated, and the average amount of data transferred across all calls is used as the interface. Calling the interface The average transmission volume; the number of times any interface calls other interfaces is linearly normalized, and the result is used as the call frequency of any interface calling other interfaces; the average transmission volume of any interface calling other interfaces is linearly normalized, and the result is used as the transmission factor of any interface calling other interfaces; the interface Calling the interface The product of the call frequency and the transmission factor is used as the interface. Calling the interface The intensity score of the call; Get Interface Each time the interface is called The latency is calculated, and then the mean and standard deviation of the latency for all calls are obtained. Based on the mean and standard deviation, the coefficient of variation of the latency for all calls is obtained. The difference between 1 and the coefficient of variation is used as the interface latency. Calling the interface Temporal dependency score; Interface Calling the interface The average of the call intensity score and the timing dependency score is used as the interface... Calling the interface The edge weights; obtain the edge weights of any interface calling other interfaces, and construct the call graph accordingly; By propagating node features on the call graph through a graph convolutional network, global context representations are extracted to obtain the context content of each interface, which in turn constitutes the context feature vector of each interface.

[0008] Optionally, the specific methods for obtaining several structural clusters include: For the interface and interfaces For the interface and interfaces Obtain cosine similarity from structural feature vectors; obtain interface The depth of the node corresponding to each field in the field content is used as the average depth of all nodes as the interface. The field depth is linearly normalized for all interfaces, and the result is used as the field depth coefficient for each interface. Get Interface Field types and interfaces in the field content The intersection-union ratio of the field types in the field content is obtained from the interface. Field depth coefficient and interface The absolute value of the difference between the field depth coefficients, the difference obtained by subtracting the intersection-union ratio from 1, and the mean of the absolute values ​​of the differences are used as the interface. With Interface Simplify the edit distance; use the ratio of the simplified edit distance to the cosine similarity as the interface. With Interface The distance between structural layers; Obtain the structural layer distance between any two interfaces, and use the K-means clustering algorithm to cluster all interfaces. The parameter K is determined by the elbow method, and the distance metric is the structural layer distance of the interfaces. Several clusters are obtained and denoted as several structural clusters.

[0009] Optionally, the structural distance weight and content distance weight between the interfaces are obtained using the following method: For the interface and interfaces Get the interface The set of node paths corresponding to each field in the field content, used as the interface. field path collection Get the interface field path collection ;Will and The number of paths in the intersection, and and The ratio of the maximum number of paths in the two sets is used as the interface. With Interface The similarity of the hierarchical structure; for the interface The field names in the field content are consistent with the interface. The field names in the content are matched, and the Levenshtein edit distance is calculated and averaged for each pair of matched field names, which is then used as the interface. With Interface Field naming differences; Interface Field types and interfaces in the field content The intersection-union ratio of the field types in the field content, compared with the interface With Interface The product is obtained by multiplying the hierarchical structure similarity, and then multiplying the product with the interface. With Interface The ratio of field naming differences and the normalized result are used as the interface. With Interface Structural distance weights; Interface Field naming and interface of each field in the field content Calculate the cosine similarity between the semantic vectors of the field names in the field content, and then combine the mean of the cosine similarities between all semantic vectors with the interface. semantic vectors and interfaces of field annotations The mean of the cosine similarity of the semantic vectors of the field annotations is used as the interface. With Interface Content semantic similarity; for interfaces and interfaces If the field content is numeric, calculate the KL divergence; if it is discrete, calculate the Jaccard coefficient. The result is used as the interface. With Interface Content distribution distance; The result of multiplying the difference obtained by subtracting the semantic similarity of the content from 1 with the content distribution distance and normalizing the result is used as the interface. With Interface Content distance weight.

[0010] Optionally, the specific method for obtaining the context distance weight based on the edge weight in the call graph, and adjusting the obtained structure clustering weight, content clustering weight, and context clustering weight according to the deviation between the content feature vector and the context feature vector of the interface within the structure cluster, includes: Interface With Interface The average weight of the call edges between corresponding nodes in the call graph is used as the interface. With Interface Context distance weight; For any structural cluster, calculate the Euclidean distance between the content feature vectors of any two interfaces within the cluster. The interface with the smallest mean Euclidean distance between its content feature vectors and those of other interfaces within the cluster is designated as the content center interface of that cluster. Obtain the squared Euclidean distance between the content feature vectors of each interface within the cluster and the content center interface. Then, sum the squared distances of all interfaces within the cluster to obtain the sum of squared content deviations for that cluster. Linearly normalize the sum of squared content deviations for all structural clusters, and use the result as the content influence factor for each structural cluster. For interfaces... and interfaces The sum of the structural cluster in which it is located, plus the content influence factor of that structural cluster, is the same as the interface. With Interface The product of the content distance weights is used as the interface. With Interface Content clustering coefficient; Based on the context feature vector of the interface, the context center interface of any structural cluster is obtained, and then the sum of squared context biases of the structural cluster is obtained. Normalization is then performed to obtain the context influence factor of the structural cluster. For the interface... and interfaces The sum of 1 and the context influence factor of the structural cluster in which it is located is the same as the interface. With Interface The product of the context distance weights is used as the interface. With Interface Context clustering coefficients; Interface With Interface The structural distance weight, content clustering coefficient, and context clustering coefficient are weighted and normalized, and the result is used as the interface. With Interface The structural clustering weights, content clustering weights, and context clustering weights.

[0011] Optionally, the method for obtaining the multimodal feature distance between interfaces and clustering the interfaces within the structural cluster to obtain several interface clusters includes: For the interface and interfaces For the interface and interfaces The Euclidean distance is calculated from the structural feature vector, content feature vector, and context feature vector, respectively. Then, the Euclidean distance is weighted and summed based on the structural clustering weight, content clustering weight, and context clustering weight, and the result is used as the interface. and interfaces Multimodal feature distance; For the interface and interfaces The structure cluster in which it is located is subjected to DBSCAN clustering of all interfaces in the structure cluster. The distance metric is the multimodal feature distance between interfaces, resulting in several clusters, which serve as several interface clusters of the structure cluster. Interface clusters are obtained from all structure clusters, resulting in several interface clusters.

[0012] Optionally, the specific methods for obtaining several integrated clusters include: For any interface cluster, obtain the multimodal feature distance between each interface in the interface cluster, and take the interface with the smallest mean multimodal feature distance with other interfaces as the central interface of the interface cluster. Obtain the central interface of each interface cluster, and take the multimodal feature distance between the central interfaces of any two interface clusters as the cross-cluster distance between the two interface clusters; The multimodal feature distances between all interfaces are linearly normalized, and the results are used as the distance factors corresponding to each multimodal feature distance. A cross-cluster threshold is preset. If the distance factor corresponding to the cross-cluster distance between two interface clusters is less than the cross-cluster threshold, the two interface clusters are merged into one integrated cluster. For any two interface clusters, obtain the cross-cluster distance and perform merging judgment in order of cross-cluster distance from smallest to largest. After all judgments are completed, the resulting merged clusters and all unmerged interface clusters are taken as the integrated clusters.

[0013] The present invention also proposes a multi-source data integration system for a computer software data interface, the system comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the above method.

[0014] The beneficial effects of this invention are as follows: This invention obtains the field content of the interface by acquiring multi-source data and its interface information; it constructs the structural feature vector and semantic feature vector of the interface by performing hierarchical analysis, semantic analysis, and value distribution quantization on the fields in the field content; it constructs directed edges and obtains edge weights by analyzing the call relationships between interfaces, thereby obtaining a call graph; it extracts the context feature vector of the interface through graph structure analysis, thereby realizing the acquisition of multimodal feature vectors of the interface; based on the structural feature vectors, it initially performs structural layer clustering on the interfaces to obtain several structural clusters, which satisfy the preliminary analysis of similar structures between interfaces; further, in the further clustering analysis process in the structural clusters, it comprehensively considers the differences in the dependency relationships between interfaces on different feature vectors, and obtains the structural distance weight, content distance weight, and context distance weight respectively, while based on... The deviation between the content feature vector and the context feature vector within the structural cluster is further adjusted to obtain the structural clustering weight, content clustering weight, and context clustering weight. A comprehensive weighted average is then used to obtain the multimodal feature distance, which is further used for clustering analysis of interfaces within the structural cluster, resulting in several interface clusters. Subsequently, considering the situation where interface clusters have similar semantics but different structures, multimodal feature distance is further judged between different interface clusters, and these are merged to obtain an integrated cluster. Multi-source data is then integrated based on the field content of each interface in the integrated cluster. Addressing the core pain point of similar but semantically different multi-source interfaces, distance measurement and dynamic weight adjustment using structural, content, and context multimodal features significantly improve the accuracy of clustering and data integration, reducing false and missed clustering issues, and providing high-quality, unified data source support for subsequent data analysis and business decisions. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic flowchart of a multi-source data integration method for a computer software data interface provided in one embodiment of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Please see Figure 1 The diagram illustrates a flowchart of a multi-source data integration method for a computer software data interface according to an embodiment of the present invention. The method includes the following steps: Step S001: Collect multi-source data and obtain interface information to get the field content of each interface.

[0019] The purpose of this embodiment is to integrate multi-source data for computer software data. First, it is necessary to collect the data and its structure and interface information completely. The similarity of multi-source interfaces is determined by multiple dimensions. Relying solely on structural features may misjudge semantic differences, and relying solely on content features may ignore the inherent relationship of interface design. It is necessary to collect three types of features—structure, content, and context—to fully characterize the interface attributes.

[0020] Specifically, by accessing all interfaces of the computer software to be integrated through the API gateway, and recording basic information such as interface name and system to which the interface belongs, the field content of the interface is obtained. For the structural characteristics of the interface, the hierarchical relationship of each field in the field content of the interface (such as the key-value pair hierarchy in JSON), field naming and data type are obtained through interface documentation or protocol parsing tools. For the content characteristics of the interface, the value distribution and value format of each field in the field content are obtained through interface call sampling. The context of the interface is recorded, and the call relationship between interfaces, the amount of data transmitted in each call and the latency are obtained through the service registry. The collected field content and related data of various characteristics of the interface are preprocessed, including data cleaning such as missing data completion and anomaly filtering. These are existing technologies and will not be described in detail in this embodiment, in order to ensure data quality and consistency.

[0021] Step S002: Based on the hierarchical distribution of fields in the interface's field content, construct the interface's structural feature vector; analyze the value distribution and semantics of the interface's field content to obtain the interface's content feature vector; based on the number of calls between interfaces, the amount of data transmitted, and the call latency, obtain the edge weights of the calls between interfaces, and then construct the call graph and obtain the interface's context feature vector.

[0022] It should be noted that clustering algorithms rely on the distance between interfaces for analysis, but structural features are tree-like hierarchies (such as XML node trees), content features are text or numerical distributions, and context features are relational networks. Therefore, these features need to be transformed into feature vectors of a unified dimension to facilitate subsequent analysis and distance calculation.

[0023] Preferably, in one embodiment of the present invention, the method for constructing the structural feature vector of the interface based on the hierarchical distribution of fields in the interface's field content includes: It should be noted that, firstly, the hierarchy and dependency relationships between interface fields are captured through a tree structure, thereby constructing the structural feature vector of the interface.

[0024] Specifically, for all fields in the interface, a tree-LSTM encoding model based on hierarchical semantics is used to construct the model, obtaining the nodes corresponding to each field. For any node, the similarity between the field name of the node and the field name of its parent node is obtained (similarity is calculated based on the encoding, using cosine similarity in vector form). The product of the obtained similarity and the path depth of the node is normalized, and the result is used as the parent node association degree of the node. The normalization is performed linearly on the product obtained for all fields corresponding to the nodes. The number of child nodes and data type encoding (e.g., int=1, float=2, string=3) of the nodes corresponding to each field are recorded. The parent node association degree, the number of child nodes, and the data type encoding are used as the field vector of the nodes corresponding to each field. The field vectors of the nodes corresponding to each field in any interface are recursively aggregated through the Tree-LSTM model, and the vector of the interface is output through the root node, which is used as the structural feature vector of the interface. The tree-LSTM encoding model is an existing technology and will not be described in detail in this embodiment.

[0025] Preferably, in one embodiment of the present invention, the method for analyzing the value distribution and semantics of the field content of the interface to obtain the content feature vector of the interface includes: For any interface's field content, this embodiment employs a dual-path fusion strategy. The BERT model is used to extract semantic vectors of the naming and annotations of each field within the field content. Statistical features are obtained from the value distribution within the field content. In this embodiment, the statistical features include the mean, variance, and entropy of the value distribution, i.e., statistical features are obtained based on the quantity of each field within the field content. Simultaneously, the word vectors of the fields (obtained through Word2Vec encoding) are combined to represent the values, resulting in a value vector. The semantic vector and the value vector are then fused to form the content feature vector of the interface. The specific fusion process employs existing vector fusion techniques, which will not be elaborated upon in this embodiment.

[0026] It should be noted that for content features, semantic vectors and value vectors are extracted through semantic analysis and value distribution quantification of field content to construct the content feature vector of the interface. For context features, a directed graph needs to be constructed based on the call relationship between interfaces. The directed edges between nodes in the graph corresponding to the interfaces need to take into account the call intensity and time-series dependencies. The more calls there are, the larger the amount of data transmitted, and the higher the collaboration density between interfaces during runtime. At the same time, the absolute number of calls will be unbalanced with different system scales, so normalization processing is also required. The smaller and more stable the interface call latency, the stronger the time-series dependency.

[0027] Preferably, in one embodiment of the present invention, the edge weights of the calls between interfaces are obtained based on the number of calls between interfaces, the amount of data transmitted, and the call latency, thereby constructing a call graph and obtaining the context feature vectors of the interfaces. The specific method includes: For the interface and interfaces Get the interface Calling the interface Number of times, get interface Each time the interface is called The amount of data transferred is calculated, and the average amount of data transferred across all calls is used as the interface. Calling the interface The average transmission volume; the number of times any interface calls other interfaces is linearly normalized, and the result is used as the call frequency of any interface calling other interfaces; the average transmission volume of any interface calling other interfaces is linearly normalized, and the result is used as the transmission factor of any interface calling other interfaces; the interface Calling the interface The product of the call frequency and the transmission factor is used as the interface. Calling the interface The intensity score of the call.

[0028] Furthermore, obtain the interface. Each time the interface is called The latency (in seconds) is calculated, and then the mean and standard deviation of the latency for all calls are obtained. Based on the mean and standard deviation, the coefficient of variation of the latency for all calls is obtained (the ratio of the standard deviation to the mean; a hyperparameter is added to both the numerator and denominator during the ratio calculation to avoid the denominator being 0, which would render the ratio meaningless; in this embodiment, the hyperparameter is described as 0.01). The difference obtained by subtracting the coefficient of variation from 1 is used as the interface... Calling the interface The time-dependent score.

[0029] It should be noted that the smaller the standard deviation and mean of latency, the smaller the overall latency and the smaller the fluctuation, the more stable the calling process, and the higher the timing dependency of the corresponding interface calls.

[0030] Furthermore, the interface Calling the interface The average of the call intensity score and the timing dependency score is used as the interface... Calling the interface The edge weights are denoted as... Obtain the edge weights of any interface calling other interfaces using the method described above, and construct the call graph accordingly. ,in This represents the set of graph nodes corresponding to the interface. This represents the set of call edges between graph nodes corresponding to interfaces. If there are call relationships between interfaces, a call edge is constructed using directed edges. For example, for each interface... Calling the interface Then the directed edge is the interface. Pointer to interface , represented as And its edge weight is .

[0031] Furthermore, a graph convolutional network (GCN) is used to propagate node features on the call graph to extract the global context representation, thereby obtaining the context content of each interface and constructing the context feature vector of each interface. The extraction of context content using a graph convolutional network is an existing technology and will not be described in detail in this embodiment. It should be noted that the structural feature vector, content feature vector, and context feature vector of the same interface need to be layer normalized according to the dimension, that is, the elements of each dimension are linearly normalized to eliminate the scale deviation of different modal features.

[0032] Thus, we obtain the interface's structural feature vector, content feature vector, and context feature vector.

[0033] Step S003: Based on the similarity of the structural feature vectors of each interface and the differences in the hierarchical distribution of fields in the interface's field content, several structural clusters are obtained; based on the differences in interface field hierarchy and type, as well as the semantic similarity of interfaces, the structural distance weight and content distance weight between interfaces are obtained respectively; based on the edge weights in the call graph, the context distance weight is obtained; and based on the deviation between the content feature vectors and context feature vectors of interfaces within a structural cluster, the structural clustering weights, content clustering weights, and context clustering weights are adjusted; in this way, the multimodal feature distance between interfaces is obtained, and the interfaces within the structural cluster are clustered to obtain several interface clusters.

[0034] It should be noted that during the integration process, it is necessary to analyze various features of multi-source data. However, for data interfaces with significant differences in structural features, their semantic correlation is usually weak. Therefore, structural feature vectors are used to initially classify multi-source data. Structural feature vectors reflect the hierarchical depth, field organization, and data type characteristics of the interface, and can reveal the intrinsic structural differences of the interface without semantic or contextual interference.

[0035] Preferably, in one embodiment of the present invention, based on the similarity of the structural feature vectors of each interface and the differences in the hierarchical distribution of fields in the field content of the interfaces, several structural clusters are obtained by clustering, including the following specific methods: For the interface and interfaces For the interface and interfaces Obtain cosine similarity from structural feature vectors; obtain interface The depth of the node corresponding to each field in the field content is used as the average depth of all nodes as the interface. The field depth is calculated, and the field depth of all interfaces is linearly normalized. The result is used as the field depth coefficient for each interface. (The last part, "Get Interface," appears to be an error and doesn't translate directly.) Field types and interfaces in the field content The intersection-union ratio of the field types in the field content is obtained from the interface. Field depth coefficient and interface The absolute value of the difference between the field depth coefficients, the difference obtained by subtracting the intersection-union ratio from 1, and the mean of the absolute values ​​of the differences are used as the interface. With Interface Simplify the edit distance; use the ratio of the simplified edit distance to the cosine similarity as the interface. With Interface Regarding the structural layer distance, it should be noted that a hyperparameter is added to both the numerator and denominator during the ratio calculation process to avoid the fraction being meaningless due to the denominator being 0. In this embodiment, the hyperparameter is described as 0.01.

[0036] Furthermore, following the above method, the structural layer distance between any two interfaces is obtained, and the K-means clustering algorithm is used to cluster all interfaces. The parameter K is determined by the elbow method, and the distance metric is the structural layer distance between the interfaces. Several clusters are obtained and denoted as several structural clusters.

[0037] It should be further noted that after obtaining the structural clusters, although different interfaces within the same structural cluster have certain similarities at the structural level, their semantic expressions, value distributions, and contextual dependencies often have significant differences. Using structural differences as the basis for clustering alone will lead to interfaces with similar structures but completely different semantics being incorrectly clustered into the same cluster. Therefore, by constructing multimodal feature distance, the structural, semantic, and contextual features of the interface are comprehensively reflected in a unified metric space, achieving a two-way balanced clustering of structural differences and semantic similarities. Furthermore, different interfaces have different distance dependencies on the three types of feature vectors: structural feature vector, content feature vector, and contextual feature vector. It is necessary to consider the dynamic adjustment of the weights of the interface features.

[0038] Preferably, in one embodiment of the present invention, the structural distance weight and content distance weight between interfaces are obtained based on differences in interface field hierarchy and type, as well as semantic similarity of interfaces. The specific method includes: It should be noted that the structure is the core of the interface's data organization, directly defining the field hierarchy topology and data association method. By integrating the core idea of ​​weighted tree edit distance, it quantifies structural differences from three dimensions: hierarchy matching degree, field name similarity, and type consistency. This not only accurately captures the core features of the structural topology but also preserves the interpretability of the hierarchical relationship, providing an intuitive basis for adjusting the weight of the structural vector distance.

[0039] For the interface and interfaces Get the interface The set of node paths corresponding to each field in the field content, used as the interface. field path collection Similarly, obtain the interface. field path collection ;Will and The number of paths in the intersection, and and The ratio of the maximum number of paths in the two sets is used as the interface. With Interface The similarity of the hierarchical structure; for the interface The field names in the field content are consistent with the interface. The field names in the content are matched, and the Levenshtein edit distance is calculated and averaged for each pair of matched field names, which is then used as the interface. With Interface The field naming difference is calculated using existing technologies, and will not be described further in this embodiment.

[0040] Furthermore, the interface Field types and interfaces in the field content The intersection-union ratio of the field types in the field content, compared with the interface With Interface The product is obtained by multiplying the hierarchical structure similarity, and then multiplying the product with the interface. With Interface The ratio of field naming differences and the normalized result are used as the interface. With Interface The structural distance weights are normalized using the sigmoid function in this embodiment.

[0041] It should be further clarified that structural similarity is not the same as functional or semantic consistency. The differences in interface content need to be further characterized from both semantic and value-based dimensions. The semantic layer focuses on the core meaning conveyed by field names and comments, while the value-based layer focuses on the statistical regularity and semantic tendency of field values. For interfaces with more significant differences in semantic and value-based layers, the weight of their content feature vectors in multi-feature distance calculation should be higher, thereby strengthening the role of the content dimension in judging semantic consistency.

[0042] Furthermore, regarding the interface Field naming and interface of each field in the field content Calculate the cosine similarity between the semantic vectors of the field names in the field content, and then combine the mean of the cosine similarities between all semantic vectors with the interface. semantic vectors and interfaces of field annotations The mean of the cosine similarity of the semantic vectors of the field annotations is used as the interface. With Interface Content semantic similarity; for interfaces and interfaces If the field content is numeric, calculate the KL divergence; if it is discrete, calculate the Jaccard coefficient. The result is used as the interface. With Interface The content distribution distance.

[0043] Furthermore, the product of the difference obtained by subtracting the semantic similarity of the content from 1 and the content distribution distance, and the result after normalization, is used as the interface. With Interface The content distance weights are normalized using the sigmoid function in this embodiment.

[0044] It should be further explained that interfaces do not exist in isolation, but are embedded in complex call and dependency networks. Context information is the key clue to reveal the functional correlation of interfaces. By constructing a call graph based on dependency relationships, the correlation weights of interface nodes in the graph are mapped to the calculation weights of context distance, allowing the context dimension to more accurately reflect the inherent correlation of interface functions. At the same time, for interfaces within the same structural cluster, if the difference between their content feature vector and context feature vector is significant, it indicates that there is a high probability that there are structurally similar but semantically or functionally different interfaces in the cluster. In this case, when calculating the multimodal feature distance, the weight of the structural feature vector should be reduced accordingly to avoid structural features from overly dominating the clustering results and to ensure the unity of clustering accuracy and semantic consistency.

[0045] Preferably, in one embodiment of the present invention, the context distance weight is obtained based on the edge weight in the call graph, and the structure clustering weight, content clustering weight, and context clustering weight are adjusted according to the deviation between the content feature vector and the context feature vector of the interface within the structure cluster. The specific method includes: Interface With Interface The average weight of the call edges between corresponding nodes in the call graph is used as the interface. With Interface The context distance weight is 0 if there is no call edge, and is directly used as the context distance weight if there is only one call edge.

[0046] Furthermore, for any structural cluster, the Euclidean distance is calculated between the content feature vectors of any two interfaces within the cluster. The interface with the smallest mean Euclidean distance between its content feature vectors and those of other interfaces within the cluster is designated as the content center interface of that cluster. The squared Euclidean distances between the content feature vectors of each interface within the cluster and the content center interface are then calculated. These squared distances are then summed across all interfaces within the cluster (excluding the content center interface) to obtain the sum of squared content deviations for that cluster. The sums of squared content deviations across all clusters are then linearly normalized, and the resulting sum is used as the content influence factor for each cluster. It should be noted that since further clustering analysis is performed within the structural clusters, the content distance weights can be adjusted based on the content influence factor of the same cluster. For interfaces… and interfaces The sum of the structural cluster in which it is located, plus the content influence factor of that structural cluster, is the same as the interface. With Interface The product of the content distance weights is used as the interface. With Interface The content clustering coefficient is calculated as follows: It should be noted that, since cross-cluster analysis is required subsequently, the interfaces of different structural clusters need to have their multimodal feature distances obtained. Therefore, the average of the content influence factors of the structural clusters to which the two interfaces are located is calculated, and 1 is added to obtain the sum. The weights of the content distance between the two interfaces are then adjusted to obtain the content clustering coefficient. The subsequent processing method for the context clustering coefficient is the same.

[0047] Furthermore, similarly, based on the context feature vector of the interface, the context center interface of any structural cluster is obtained, and then the sum of squared context biases of the structural cluster is obtained, which is normalized to obtain the context influence factor of the structural cluster; for the interface and interfaces The sum of 1 and the context influence factor of the structural cluster in which it is located is the same as the interface. With Interface The product of the context distance weights is used as the interface. With Interface Context clustering coefficients.

[0048] Furthermore, regarding the interface With Interface The structural distance weight, content clustering coefficient, and context clustering coefficient are weighted and normalized, and the result is used as the interface. With Interface The structural clustering weights, content clustering weights, and context clustering weights.

[0049] It should be further noted that, for obtaining the structural clustering weight, content clustering weight, and context clustering weight between interfaces respectively, the multimodal feature distance needs to be analyzed by weighting the Euclidean distance between the three types of feature vectors.

[0050] Preferably, in one embodiment of the present invention, the multimodal feature distance between interfaces is obtained in this way, and the interfaces within the structure cluster are clustered to obtain several interface clusters. The specific method includes: For the interface and interfaces For the interface and interfaces The Euclidean distance is calculated from the structural feature vector, content feature vector, and context feature vector, respectively. Then, the Euclidean distance is weighted and summed based on the structural clustering weight, content clustering weight, and context clustering weight, and the result is used as the interface. and interfaces Multimodal feature distance.

[0051] Furthermore, regarding the interface and interfaces For each structural cluster, DBSCAN clustering is performed on all interfaces within that cluster. The distance metric is the multimodal feature distance between interfaces, resulting in several clusters that serve as interface clusters for that structural cluster. By performing interface cluster acquisition on all structural clusters using the same method, several interface clusters are finally obtained.

[0052] At this point, several interface clusters have been obtained.

[0053] Step S004: Analyze the multimodal feature distance between interfaces in different interface clusters, determine and merge to obtain several integrated clusters, and perform multi-source data integration based on the field content of each interface in the integrated cluster.

[0054] It should be noted that due to structural differences, two-layer clustering may divide semantically similar interfaces into different structural clusters at the structural layer, resulting in semantically similar interfaces being in different interface clusters. In this case, it is necessary to supplement by the cross-cluster distance between interface clusters. By analyzing the magnitude of the cross-cluster distance, interface clusters can be merged to ensure that semantically consistent or similar interfaces are in the same cluster, so as to facilitate the integration of multi-source data in the future.

[0055] Specifically, for any interface cluster, the multimodal feature distances between each interface in the cluster are obtained, and the interface with the smallest mean multimodal feature distances with other interfaces is taken as the center interface of the cluster. The center interfaces of each interface cluster are obtained using the above method, and the multimodal feature distance between the center interfaces of any two interface clusters is taken as the cross-cluster distance between the two clusters. The multimodal feature distances between all interfaces are linearly normalized, and the result is taken as the distance factor corresponding to each multimodal feature distance. A cross-cluster threshold is preset; in this embodiment, the cross-cluster threshold is described as 0.25. If the distance factor corresponding to the cross-cluster distance between two interface clusters is less than the cross-cluster threshold, the two interface clusters are merged. Then, as a merged cluster, obtain the cross-cluster distance for any two interface clusters according to the above method, and perform merge judgment in order of cross-cluster distance from smallest to largest; it should be noted that if interface cluster A and interface cluster B have already been merged into a merged cluster during the merging process, and interface cluster C is judged during the judgment process, the distance factor corresponding to the cross-cluster distance with interface cluster A and interface cluster B must be less than the cross-cluster threshold in order to merge interface cluster C with interface cluster A and interface cluster B, otherwise it cannot be merged, and so on, that is, in the subsequent merging process, it is necessary to ensure that all interface clusters in a merged cluster that has been successfully merged meet the merging conditions; the merged clusters obtained after all judgments are completed, as well as all unmerged interface clusters, are all regarded as integrated clusters.

[0056] Furthermore, for any integration cluster, the field content corresponding to each interface in the integration cluster is extracted, and the common fields in the field content of all interfaces in the integration cluster are obtained. By obtaining the common fields of each integration cluster, the integration of multi-source data is realized.

[0057] This concludes the embodiment.

[0058] Another embodiment of the present invention provides a multi-source data integration system for a computer software data interface. The system includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the above-described method steps S001 to S004.

[0059] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for multi-source data integration of computer software data interfaces, characterized in that, The method comprises the following steps: Collecting multi-source data and obtaining interface information to obtain field content of each interface; Based on the field level distribution in the field content of the interface, the structure feature vector of the interface is constructed; the value distribution and its semantics in the field content of the interface are analyzed to obtain the content feature vector of the interface; the edge weight of the call between interfaces is obtained according to the call times and the transmission data volume between interfaces, as well as the call delay, and then the call graph is constructed to obtain the context feature vector of the interface; Based on the structural feature vector similarity of each interface and the field level distribution difference in the field content of the interface, a plurality of structural clusters are obtained; the structural distance weight and the content distance weight between interfaces are obtained based on the field level difference and the type difference of the interface, as well as the semantic similarity of the interface; the context distance weight is obtained based on the edge weight in the call graph, and the structural clustering weight, the content clustering weight and the context clustering weight are adjusted according to the deviation of the content feature vector and the context feature vector of the interface in the structural cluster; the multi-modal feature distance between interfaces is obtained, the interfaces in the structural cluster are clustered, and a plurality of interface clusters are obtained; The multi-modal feature distance between interfaces in different interface clusters is analyzed to determine the merging to obtain a plurality of integrated clusters, and the multi-source data integration is performed based on the field content of each interface in the integrated cluster.

2. The method of claim 1, wherein, The specific method for constructing the structural feature vector of the interface comprises: For the fields in all interfaces, a hierarchical semantic-based tree structure coding model is used for model construction to obtain nodes corresponding to each field; for any node, the similarity between the field name of the node and the field name of its parent node is obtained, and the product of the obtained similarity and the path depth of the node is normalized to obtain the parent node association degree of the node; The number of child nodes and the data type code of the nodes corresponding to each field are recorded, and the parent node association degree, the number of child nodes and the data type code are taken as the field vector of the nodes corresponding to each field; The field vectors of the nodes corresponding to each field in any interface are recursively aggregated by a Tree-LSTM model, and the vector of the interface is obtained through the root node output, which is taken as the structural feature vector of the interface.

3. The method of claim 2, wherein, The specific method for obtaining the content feature vector of the interface comprises: For the field content of any interface, the semantic vector of the name and annotation of each field of the field content is extracted by using a BERT model; statistical features are obtained for the value distribution in the field content, and the value vector is represented by combining the value word vector to obtain the value vector; the semantic vector and the value vector are fused to obtain the content feature vector of the interface.

4. The method of claim 1, wherein, The specific method for obtaining the edge weight of the call between interfaces, and then constructing the call graph and obtaining the context feature vector of the interface comprises: For the interface and interfaces Get the interface Calling the interface Number of times, get interface Each time the interface is called The amount of data transferred is calculated, and the average amount of data transferred across all calls is used as the interface. Calling the interface The average transmission volume; the number of times any interface calls other interfaces is linearly normalized, and the result is used as the call frequency of any interface calling other interfaces; the average transmission volume of any interface calling other interfaces is linearly normalized, and the result is used as the transmission factor of any interface calling other interfaces; the interface Calling the interface The product of the call frequency and the transmission factor is used as the interface. Calling the interface The intensity score of the call; Get Interface Each time the interface is called The latency is calculated, and then the mean and standard deviation of the latency for all calls are obtained. Based on the mean and standard deviation, the coefficient of variation of the latency for all calls is obtained. The difference between 1 and the coefficient of variation is used as the interface latency. Calling the interface Temporal dependency score; Call interfaces Call interfaces The average of the call intensity score and the timing dependency score of the interface Call interfaces The edge weight of the interface; get the edge weight of any interface calling other interfaces, and build a call graph The node features are propagated on the call graph by a graph convolution network to extract global context representation, so as to obtain the context content of each interface, and then the context feature vector of each interface is formed.

5. The method of claim 2, wherein, The specific method for obtaining a plurality of structural clusters comprises: For the interface and interfaces For the interface and interfaces Obtain cosine similarity from structural feature vectors; obtain interface The depth of the node corresponding to each field in the field content is used as the average depth of all nodes as the interface. The field depth is linearly normalized for all interfaces, and the result is used as the field depth coefficient for each interface. Get Interface Field types and interfaces in the field content The intersection-union ratio of the field types in the field content is obtained from the interface. Field depth coefficient and interface The absolute value of the difference between the field depth coefficients, the difference obtained by subtracting the intersection-union ratio from 1, and the mean of the absolute values ​​of the differences are used as the interface. With Interface Simplify the edit distance; use the ratio of the simplified edit distance to the cosine similarity as the interface. With Interface The distance between structural layers; Obtaining the structural layer distance between any two interfaces, using K-means clustering algorithm to cluster all interfaces, the parameter K is determined by elbow method, the distance measurement uses the structural layer distance of the interface, and a plurality of clusters are obtained, denoted as a plurality of structural clusters.

6. The method of claim 3, wherein, The structural distance weight and the content distance weight between the interfaces, the specific acquisition method is: For interface and interface , get the path set of the node corresponding to each field in the field content of interface as the field path set of interface , get the field path set of interface ; get the ratio of the number of paths in the intersection of and to the maximum of the number of paths in the two sets of and as the hierarchical structure similarity of interface and interface ; match the field naming of each field in the field content of interface with the field naming of each field in the field content of interface , and respectively calculate the Levenshtein edit distance based on each pair of matched field naming and take the average as the field naming difference degree of interface and interface ;​​ The fields of the interface are multiplied by the intersection over union of the field types in the field content of the interface and the interface The product is multiplied by the hierarchical structure similarity of the interface and the interface The result of dividing the product by the field naming difference of the interface and the interface is normalized and used as the structural distance weight of the interface and the interface Interface Field naming and interface of each field in the field content Calculate the cosine similarity between the semantic vectors of the field names in the field content, and then combine the mean of the cosine similarities between all semantic vectors with the interface. semantic vectors and interfaces of field annotations The mean of the cosine similarity of the semantic vectors of the field annotations is used as the interface. With Interface Content semantic similarity; for interfaces and interfaces If the field content is numeric, calculate the KL divergence; if it is discrete, calculate the Jaccard coefficient. The result is then used as the interface. With Interface Content distribution distance; The product of the difference of 1 minus the content semantic similarity and the content distribution distance, normalized, is the content distance weight of the interface with the interface .

7. The method of claim 1, wherein, The context distance weight is obtained based on the edge weight in the call graph, and the structural clustering weight, the content clustering weight and the context clustering weight are adjusted according to the deviation of the content feature vector and the context feature vector of the interface in the structural cluster, including the specific method: call edges between corresponding graph nodes in the call graph with the interface the mean of the call edge weights between corresponding graph nodes in the call graph as the interface with the interface context distance weight For any structure cluster, the Euclidean distance between the content feature vectors of any two interfaces in the structure cluster is calculated, the interface with the minimum average Euclidean distance between the content feature vectors of the interface and other interfaces in the structure cluster is taken as the content center interface of the structure cluster, the square value of the Euclidean distance between the content feature vectors of each interface in the structure cluster and the content center interface is obtained, and then the sum of the square values corresponding to all interfaces in the structure cluster is taken as the content deviation sum of squares of the structure cluster, the content deviation sum of squares of all structure clusters is linearly normalized, and the obtained result is taken as the content influence factor of each structure cluster; for the interface and the structure cluster where the interface is located, the sum value obtained by adding 1 to the content influence factor of the structure cluster is multiplied by the content distance weight of the interface and the interface , and the product is taken as the content clustering coefficient of the interface and the interface . Based on the interface-based context feature vector, the context center interface of any structure cluster is obtained, and then the context deviation sum of squares of the structure cluster is obtained, and the context influence factor of the structure cluster is normalized; for the interface and the structure cluster where the interface is located, the sum value obtained by adding 1 to the context influence factor of the structure cluster, and the product of the context distance weight of the interface and the interface , are taken as the context clustering coefficient of the interface and the interface . interface with the interface structure distance weight, content clustering coefficient and context clustering coefficient are weight normalized, and the obtained result is taken as the structure clustering weight, content clustering weight and context clustering weight of the interface with the interface structure clustering weight, content clustering weight and context clustering weight.

8. The method of claim 1, wherein, The multi-modal feature distance between the interfaces is obtained, the interfaces in the structural cluster are clustered, and a plurality of interface clusters are obtained, including the specific method: For the interface and interfaces For the interface and interfaces The Euclidean distance is calculated from the structural feature vector, content feature vector, and context feature vector, respectively. Then, the Euclidean distance is weighted and summed based on the structural clustering weight, content clustering weight, and context clustering weight, and the result is used as the interface. and interfaces Multimodal feature distance; For interfaces and interfaces The structure cluster where the interface is located is clustered by DBSCAN, the distance measure is the multimodal feature distance between interfaces, and several interface clusters are obtained as the several interface clusters of the structure cluster. The interface cluster acquisition is performed on all structural clusters, and a plurality of interface clusters are obtained.

9. The method of claim 1, wherein, The plurality of integrated clusters are obtained, including the specific method: For any interface cluster, the multi-modal feature distance between each interface in the interface cluster is obtained, and the interface with the minimum average multi-modal feature distance with other interfaces is taken as the center interface of the interface cluster; The center interfaces of each interface cluster are obtained, and the multi-modal feature distance between the center interfaces of any two interface clusters is taken as the cross-cluster distance of the two interface clusters; The corresponding multi-modal feature distances between all interfaces are linearly normalized, and the obtained results are taken as the distance factors corresponding to each multi-modal feature distance; The preset The cross-cluster threshold value, if the distance factor corresponding to the cross-cluster distance of the two interface clusters is less than the cross-cluster threshold value, the two interface clusters are merged as an integrated cluster; The cross-cluster distance is obtained for any two interface clusters, and the merging judgment is performed in the order from small to large according to the cross-cluster distance, and the plurality of merged clusters obtained after the complete judgment and all unmerged interface clusters are taken as integrated clusters.

10. A multi-source data integration system for computer software data interfaces, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein, The processor executes the computer program to realize the steps of the multi-source data integration method of the computer software data interface according to any one of claims 1-9.

Images