Potential information identification method and system combined with heterogeneous network environment

Abstract

The present invention provides a method and system for identifying potential information in a heterogeneous network environment. First, a heterogeneous network environment data set containing original information resources in various data formats of a target network environment is obtained, and features are extracted to obtain a related feature set. Then, a hierarchical pattern matching process is performed on the related feature set based on a pre-trained information pattern analysis model to generate a potential information identification result set containing attribute identifiers and distribution paths of significant value information resources. Then, a dynamic resource integration strategy is generated based on the potential information identification result set to adjust resource scheduling rules and storage path mapping relationships. Finally, the dynamic resource integration strategy is fed back to a data management system to trigger a resource directory update operation, thereby improving the utilization and management efficiency of heterogeneous network resources.

Description

Technical Field

[0001] The present invention relates to the technical field of information resource optimization, and in particular to a method and system for identifying potential information in a heterogeneous network environment. Background Art

[0002] In today's network environment, with the rapid development of information technology, network architecture is becoming increasingly complex, and heterogeneous network environments are gradually becoming the mainstream. Heterogeneous network environments include many different types of network devices, operating systems, and applications, resulting in the existence of a large number of raw information resources in different data formats.

[0003] Existing latent information identification methods often only work with a single data format or homogeneous network environments. They lack effective integration and analysis tools for raw information resources in multiple data formats within heterogeneous network environments. These methods struggle to mine the semantic associations and dynamic interactions between raw information resources, making it difficult to accurately identify valuable latent information.

[0004] Furthermore, existing methods cannot generate reasonable resource integration strategies based on the identification results. This makes it difficult to effectively adjust resource scheduling rules and storage path mapping relationships in heterogeneous network environments, resulting in inefficient network resource utilization and numerous difficulties in updating and maintaining data management systems. Therefore, a potential information identification method that can integrate heterogeneous network environments is needed to address these issues. Summary of the Invention

[0005] In view of the above-mentioned problems, in combination with the first aspect of the present invention, an embodiment of the present invention provides a method for identifying potential information in a heterogeneous network environment, the method comprising:

[0006] Acquire a heterogeneous network environment data set of a target network environment, wherein the heterogeneous network environment data set includes original information resources in multiple data formats;

[0007] Extracting features from the heterogeneous network environment data set to obtain a correlation feature set, wherein the correlation feature set includes semantic correlation features and dynamic interaction features between the original information resources;

[0008] Based on a pre-trained information pattern analysis model, hierarchical pattern matching processing is performed on the associated feature set to generate a potential information recognition result set, wherein the potential information recognition result set includes attribute identifiers and distribution paths of multiple significant value information resources;

[0009] Generating a dynamic resource integration strategy based on the potential information identification result set, wherein the dynamic resource integration strategy is used to adjust resource scheduling rules and storage path mapping relationships in a heterogeneous network environment;

[0010] Feedback the dynamic resource integration strategy to the data management system of the target network environment, triggering the data management system to execute a resource directory update operation.

[0011] On the other hand, an embodiment of the present invention also provides a potential information identification system in combination with a heterogeneous network environment, including a processor and a machine-readable storage medium, wherein the machine-readable storage medium is connected to the processor, the machine-readable storage medium is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the machine-readable storage medium to implement the above method.

[0012] Based on the above aspects, the embodiment of the present invention obtains a heterogeneous network environment data set containing original information resources in various data formats, extracts features from the heterogeneous network environment data set to obtain a correlation feature set of semantic correlation features and dynamic interaction features, explores the intrinsic connections between information resources at a deeper level, and performs hierarchical pattern matching processing based on a pre-trained information pattern analysis model to accurately identify the attribute identifiers and distribution paths of significant value information resources. Based on the identification results, a dynamic resource integration strategy is generated, resource scheduling rules and storage path mapping relationships are effectively adjusted, and resource utilization efficiency in a heterogeneous network environment is improved. The dynamic resource integration strategy is fed back to the data management system to trigger a resource directory update operation, making data management of the entire heterogeneous network environment more scientific and efficient, and comprehensively improving the information processing and management capabilities of the heterogeneous network environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 The figure is a schematic diagram of the execution flow of the potential information identification method in a heterogeneous network environment provided by an embodiment of the present invention.

[0014] Figure 2 Schematic diagram of exemplary hardware and software components of a potential information identification system in a heterogeneous network environment provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0015] The present invention will be described in detail below with reference to the accompanying drawings. Figure 1 FIG1 is a flow chart of a method for identifying potential information in a heterogeneous network environment provided by an embodiment of the present invention. The method for identifying potential information in a heterogeneous network environment is introduced in detail below.

[0016] Step S110: Acquire a heterogeneous network environment data set of a target network environment, wherein the heterogeneous network environment data set includes original information resources in multiple data formats.

[0017] In this embodiment, the target network environment has multiple different data sources. Data source A generates information in data format X, data source B generates information in data format Y, and data source C generates information in data format Z. Data formats X, Y, and Z here represent different data organizations and representations. For example, data format X may be a structured record format, data format Y may be a semi-structured markup language format, and data format Z may be an unstructured data stream format. Through network connections, a data acquisition module is used to obtain data from these data sources. The information in data format X from data source A, the information in data format Y from data source B, and the information in data format Z from data source C are integrated together to form a heterogeneous network environment data set. For example, the data acquisition module establishes a connection with data source A via a network protocol and collects the data output by data source A according to its data interface specification. Similarly, operations are performed on data sources B and C, ultimately resulting in a heterogeneous network environment data set that contains raw information resources in multiple data formats.

[0018] Step S120: extracting features from the heterogeneous network environment data set to obtain a correlation feature set, where the correlation feature set includes semantic correlation features and dynamic interaction features between the original information resources.

[0019] In this embodiment, feature extraction operations are performed on different data types in the heterogeneous network environment data set.

[0020] Step S121: performing semantic vectorization processing on the text data units in the heterogeneous network environment data set to generate a text semantic feature set, wherein each text semantic feature in the text semantic feature set corresponds to a semantic vector representation of a text data unit.

[0021] Assume that there are several text data units in a heterogeneous network environment data set, represented by T1, T2, ..., Tn. For each text data unit Ti, it is first segmented and divided into individual words, represented by w1, w2, ..., wm. Then, a semantic vectorization method is used, such as a language model trained based on related technologies, to assign a vector value Vj to each word wj, and the dimension of this vector value is assumed to be d. Then, the semantic vector representation of the text data unit Ti is calculated through some aggregation method, such as weighted summation. Assuming that the weight of each word wj is aj, the semantic vector representation Si of the text data unit Ti is: Si=a1V1+a2V2+...+amVm. By performing this processing on all text data units T1, T2, ..., Tn, a set of text semantic features is obtained, where each text semantic feature is the semantic vector representation Si of the corresponding text data unit.

[0022] Step S122: performing spatial feature encoding processing on the image data units in the heterogeneous network environment data set to generate an image spatial feature set, wherein each image spatial feature in the image spatial feature set includes object distribution information and contextual relationship information in the image data unit.

[0023] Assume that there are image data units I1, I2, ..., In in a heterogeneous network environment data set. For each image data unit Ij, a spatial feature encoding algorithm is used. First, the image is divided into multiple regions. For example, according to a fixed grid division method, the image Ij is divided into regions A1, A2, ..., Ak. Then, for each region A1, a feature extraction operator, such as an edge detection operator or a corner detection operator, is used to extract features in region A1, such as edge features and corner features, and these features are represented by Fl. At the same time, the relationship between region A1 and the surrounding regions is analyzed using conventional context analysis methods in related technologies to obtain contextual relationship information Cl. The feature Fl of region A1 and the contextual relationship information Cl are combined to form a feature description corresponding to region A1. This processing is performed on all regions A1, A2, ..., Ak in image Ij, and then the feature descriptions of these regions are integrated in some way, such as by splicing, to obtain the image spatial feature of image data unit Ij, which is recorded as Ij-Feature. By performing such processing on all image data units I1, I2, ..., In, an image space feature set is generated, where each image space feature Ij-Feature contains the object distribution information (reflected by regional features) and contextual relationship information in the image data unit Ij.

[0024] Step S123: performing dynamic pattern analysis processing on the time series data units in the heterogeneous network environment data set to generate a time series dynamic feature set, wherein each time series dynamic feature in the time series dynamic feature set includes data change trend and periodic fluctuation information.

[0025] Assume that a heterogeneous network environment data set contains time series data units S1, S2, ..., Sn. For each time series data unit Sk, it is a data sequence arranged in chronological order, represented by sk1, sk2, ..., skm. First, a data analysis method, such as the sliding window method, is used to slide a window of size w on the data sequence sk1, sk2, ..., skm. Within each window, the data change trend is analyzed by calculating the slope of the data. Assume that within the window [ski, ​​ski+1, ..., ski+w-1], the slope between adjacent data points is calculated. For example, the slope from ski to ski+1 is (ski+1-ski) / (t(ski+1)-t(ski)), where t(ski) represents the time corresponding to the data point ski. By analyzing multiple slopes within the window, the change trend feature Tk of the data within the window is obtained. At the same time, the periodic fluctuation information of the data sequence is analyzed using methods such as Fourier transform to obtain the periodic fluctuation feature Pk. The change trend feature Tk and the periodic fluctuation feature Pk are combined to form the time series dynamic feature of the time series data unit Sk, denoted as Sk-Feature. This process is performed on all time series data units S1, S2, ..., Sn to generate a time series dynamic feature set, where each time series dynamic feature Sk-Feature contains information about the data change trend and periodic fluctuation.

[0026] Step S124: performing cross-modal alignment processing on the text semantic feature set, the image spatial feature set, and the temporal dynamic feature set, determining semantic association features between different modal features, and generating dynamic interaction features based on feature interaction frequency.

[0027] In this embodiment, a cross-modal alignment operation is performed on a text semantic feature set, an image spatial feature set, and a temporal dynamic feature set.

[0028] Step S1241: For each text data unit corresponding to a text semantic feature, match the image data unit and the time series data unit with a synchronized timestamp within a preset time window according to its timestamp.

[0029] Assume that there is a text semantic feature T-Si in the text semantic feature set, and its corresponding text data unit is Ti, which has a timestamp t(Ti). The preset time window is [-Δt, +Δt]. Within this time window, search for image data unit Ij in the image spatial feature set and time series data unit Sk in the time series dynamic feature set, so that |t(Ti)-t(Ij)|≤Δt and |t(Ti)-t(Sk)|≤Δt, where t(Ij) and t(Sk) are the timestamps of image data unit Ij and time series data unit Sk, respectively. After finding the matching image data unit Ij and time series data unit Sk, their corresponding image spatial feature Ij-Feature and time series dynamic feature Sk-Feature are associated with the text semantic feature T-Si to form a feature triple (T-Si, Ij-Feature, Sk-Feature). This matching operation is performed on all text semantic features in the text semantic feature set to obtain a series of feature triplets.

[0030] Step S1242: After mapping the text semantic features, image spatial features, and temporal dynamic features to a unified dimensional space through a projection layer, a cross-modal alignment model is called to perform joint encoding processing on the mapped features to generate cross-modal joint encoding features.

[0031] Assume that the dimension of the text semantic feature T-Si is d1, the dimension of the image spatial feature Ij-Feature is d2, and the dimension of the temporal dynamic feature Sk-Feature is d3. Through the projection layer, the text semantic feature T-Si is projected into a space of uniform dimension d, resulting in the projected text semantic feature T-Si'. The image spatial feature Ij-Feature is projected into a space of the same dimension d, resulting in the projected image spatial feature Ij-Feature'. The temporal dynamic feature Sk-Feature is projected into a space of dimension d, resulting in the projected temporal dynamic feature Sk-Feature'. The projected feature triple (T-Si', Ij-Feature', Sk-Feature') is then input into the cross-modal alignment model. The cross-modal alignment model comprises multiple neural network layers. For example, the input layer receives the projected feature triple. After a series of nonlinear transformations in the hidden layer, the features of different modalities are fused and encoded. Finally, the output layer outputs the cross-modal joint encoded feature, denoted as C(T-Si, Ij, Sk). All feature triplets are processed in this way to obtain multiple cross-modal joint encoding features.

[0032] Step S1243: performing similarity calculation processing on the cross-modal joint encoding features to determine a first correlation score between the text semantic features and the image spatial features, and a second correlation score between the text semantic features and the temporal dynamic features.

[0033] For each cross-modal joint coding feature C(T-Si, Ij, Sk), the part related to the text semantic feature and the image spatial feature is extracted from the joint coding feature, and the similarity between the text semantic feature T-Si' and the image spatial feature Ij-Feature' is calculated by a similarity calculation method, such as a cosine similarity calculation method, to obtain a first correlation score R1(T-Si, Ij). Similarly, the part related to the text semantic feature and the temporal dynamic feature is extracted from the cross-modal joint coding feature C(T-Si, Ij, Sk), and the similarity between the text semantic feature T-Si' and the temporal dynamic feature Sk-Feature' is calculated to obtain a second correlation score R2(T-Si, Sk). Such calculations are performed on all cross-modal joint coding features to obtain a series of first correlation scores R1(T-Si, Ij) and second correlation scores R2(T-Si, Sk).

[0034] Step S1244: construct a semantic association matrix based on the first association score and the second association score, wherein the rows of the semantic association matrix represent text data unit identifiers, the columns represent image data units or time series data unit identifiers, and the matrix elements are corresponding association scores.

[0035] Assume that the set of text data unit identifiers is {T1, T2, …, Tn}, the set of image data unit identifiers is {I1, I2, …, In}, and the set of time series data unit identifiers is {S1, S2, …, Sn}. Construct a semantic association matrix M, whose number of rows equals the number of text data units n, and whose number of columns equals the sum of the number of image data units and time series data units 2n. For the i-th row and j-th column of matrix M (1≤j≤n), if j corresponds to image data unit Ij, then M(i, j) = R1(Ti, Ij); for the i-th row and k-th column (n+1≤k≤2n), if kn corresponds to time series data unit Sk-n, then M(i, k) = R2(Ti, Sk-n). In this way, all first and second association scores are populated into the semantic association matrix M.

[0036] Step S1245: extracting feature combinations corresponding to association scores exceeding a preset threshold in the semantic association matrix to generate the semantic association features.

[0037] Set a preset threshold θ and traverse all elements in the semantic association matrix M. For element M(i, j), if M(i, j)>θ, when 1≤j≤n, extract the text semantic feature T-Si of the corresponding text data unit Ti and the image spatial feature Ij-Feature of the image data unit Ij to form a feature combination (T-Si, Ij-Feature); when n+1≤j≤2n, extract the text semantic feature T-Si of the corresponding text data unit Ti and the temporal dynamic feature Sj-n-Feature of the temporal data unit Sj-n to form a feature combination (T-Si, Sj-n-Feature). Collect all feature combinations that meet the conditions to form a semantic association feature set, that is, obtain the semantic association feature.

[0038] Step S1246: Divide each data unit in the heterogeneous network environment data set into blocks according to data category, count the number of interaction events between data units in each block, and construct a block interaction frequency matrix, where the rows and columns of the interaction frequency matrix represent different data unit identifiers, and the matrix elements represent the number of interactions between corresponding data units.

[0039] The heterogeneous network environment data set is divided into different blocks according to data categories, for example, into text data blocks, image data blocks and time series data blocks. For the text data block, it is assumed that there are text data units T1, T2, ..., Tn. Define an interaction event, for example, the text data unit Ti sends a message to the text data unit Tj as an interaction event. Count the number of interaction events between each text data unit Ti and other text data units Tj in the text data block within a period of time, recorded as N(Ti, Tj). Similarly, similar operations are performed on the image data block and the time series data block to count the number of interaction events between the image data units and the time series data units. Then construct a block interaction frequency matrix F, the number of its rows and columns of which is equal to the total number of all data units. For the i-th row and j-th column of matrix F, if the i-th data unit is a text data unit Ti and the j-th data unit is a text data unit Tj, then F(i, j) = N(Ti, Tj); if the i-th data unit is an image data unit Ii and the j-th data unit is an image data unit Ij, then F(i, j) = N(Ii, Ij); if the i-th data unit is a time series data unit Si and the j-th data unit is a time series data unit Sj, then F(i, j) = N(Si, Sj). In this way, the block interaction frequency matrix F is filled.

[0040] Step S1247: performing time dimension decomposition processing on the interaction frequency matrix to obtain short-term interaction frequency characteristics and long-term interaction trend characteristics.

[0041] A time dimension decomposition method is used, for example, dividing time into short-term and long-term time periods. Assume the short-term time period is [t1, t2] and the long-term time period is [t3, t4], with [t1, t2] contained in [t3, t4]. For the block interaction frequency matrix F, within the short-term time period [t1, t2], the interaction frequency between each data unit pair is counted to obtain a short-term interaction frequency matrix Fs, where the element Fs(i, j) represents the interaction frequency between the i-th data unit and the j-th data unit within the short-term time period. Features are extracted from the short-term interaction frequency matrix Fs, such as the average interaction frequency and standard deviation of each data unit, to form short-term interaction frequency features. Within the long-term time period [t3, t4], the interaction frequency trend between each data unit pair is counted. For example, a curve of the interaction frequency over time is fitted using linear regression to obtain long-term interaction trend features. Long-term interaction trend features can be represented by parameters such as the slope and intercept of the curve.

[0042] Step S1248: performing a difference calculation between the short-term interaction frequency feature and the long-term interaction trend feature to generate a dynamic fluctuation feature.

[0043] Calculate the difference between the short-term interaction frequency feature and the long-term interaction trend feature. Assume that the short-term interaction frequency feature is represented by a set of parameters A1, A2, …, Am, and the long-term interaction trend feature is represented by a set of parameters B1, B2, …, Bm, where m is the number of feature parameters. For each parameter pair Ai and Bi, calculate their difference Di = Ai - Bi. Combine all the differences D1, D2, …, Dm to form a dynamic fluctuation feature. The dynamic fluctuation feature reflects the short-term and long-term changes in the interaction frequency between data units.

[0044] Step S1249: Determine the dynamic interaction intensity between the data units according to the change direction and amplitude of the dynamic fluctuation characteristics.

[0045] Analyze each difference Di in the dynamic fluctuation characteristics. If Di>0, it means that the short-term interaction frequency is higher than the long-term interaction trend, and the larger the absolute value of Di, the higher the dynamic interaction intensity; if Di<0, it means that the short-term interaction frequency is lower than the long-term interaction trend, and the larger |Di| is, the lower the dynamic interaction intensity. According to this rule, a dynamic interaction intensity value is determined for each data unit pair. For example, for the data unit pair (i, j), the dynamic interaction intensity value I(i, j) is calculated based on the corresponding difference Di. In this way, the dynamic interaction intensity between all data units is determined.

[0046] Step S12410: performing weighted correction processing on the semantic association feature based on the dynamic interaction strength to generate the dynamic interaction feature.

[0047] For each feature combination in the semantic association feature, such as the feature combination (T-Si, Ij-Feature), a weighted correction is performed on the feature combination based on the dynamic interaction strength I(Ti, Ij) between the corresponding text data unit Ti and image data unit Ij. Assuming that the feature combination (T-Si, Ij-Feature) can be expressed as a vector form V(T-Si, Ij-Feature), the weighted correction feature combination is V'(T-Si, Ij-Feature) = I(Ti, Ij) * V(T-Si, Ij-Feature). This weighted correction process is performed on all feature combinations in the semantic association feature, and then the weighted correction feature combinations are spliced ​​(not added) to form the dynamic interaction feature.

[0048] Step S125: Aggregate the semantic association features and the dynamic interaction features to construct the association feature set.

[0049] Aggregate all feature combinations and dynamic interaction features in the semantic association feature set. For example, assuming the semantic association feature set is {Semantic1, Semantic2, …, Semantick} and the dynamic interaction feature is Dynamic, concatenate them together in a set order to form an associated feature set, denoted as AssociatedFeatures = [Semantic1, Semantic2, …, Semantick, Dynamic]. In this way, an associated feature set is constructed.

[0050] Step S130: Based on the pre-trained information pattern analysis model, hierarchical pattern matching processing is performed on the associated feature set to generate a potential information recognition result set, which includes attribute identifiers and distribution paths of multiple significant value information resources.

[0051] In this embodiment, the associated feature set is processed using a pre-trained information pattern analysis model.

[0052] Step S131: input the associated feature set into the feature coding layer of the information pattern analysis model, perform layer-by-layer nonlinear transformation processing, and generate a multi-layer abstract feature representation sequence, wherein the dimension of the n-th layer abstract feature representation and the dimension of the n-1-th layer abstract feature representation are in a preset dimensionality reduction ratio.

[0053] The feature encoding layer of the information pattern analysis model consists of multiple neural network layers. The associated feature set, AssociatedFeatures, is input to the first layer of the feature encoding layer. The first layer performs a nonlinear transformation on the input associated features, for example, using an activation function (such as the Reluctant Unified Unit (ReLU) function) to map the associated features from their original dimension to a new dimension. Assume that the original dimension of the associated feature set is d0, and the dimension of the features output by the first layer is d1, with d1 and d0 satisfying a predetermined dimensionality reduction ratio r, i.e., d1 = r * d0. The features output by the first layer are then input to the second layer, which similarly performs a nonlinear transformation on the input features, outputting features of dimension d2 = r * d1. This process continues in this order. After several layers of nonlinear transformations, a multi-layer sequence of abstract feature representations is generated, denoted as [L1, L2, …, Ln], where Li represents the abstract feature representation of the i-th layer. Each layer's nonlinear transformation transforms features in different feature spaces. Through this layer-by-layer processing, the original associated feature set is gradually abstracted, allowing higher-level abstract features to capture more complex and representative pattern information. For example, the first layer may initially integrate and transform some local features in the associated feature set, and the second layer further mines higher-level feature combination information based on the output of the first layer. As the number of layers increases, the feature representation becomes more and more abstract, and key features related to the underlying information pattern are gradually extracted.

[0054] Step S132: In the pattern matching layer of the information pattern analysis model, cross-layer correlation calculation is performed on the multi-layer abstract feature representation sequence to determine the pattern similarity index between the abstract feature representations of adjacent layers.

[0055] From the multi-layer abstract feature representation sequence [L1, L2, …, Ln], select two adjacent layers, namely, the k-th layer abstract feature representation Lk and the k+1-th layer abstract feature representation Lk+1, for processing. First, calculate the similarity between the two at the same feature unit position. Here, using a common similarity calculation method as an example, calculate the cosine similarity between them. Assume that Lk consists of a series of feature vectors, denoted as Lk=[v1_k, v2_k, …, vm_k], and Lk+1 consists of the corresponding feature vectors, denoted as Lk+1=[v1_k+1, v2_k+1, …, vm_k+1] (this assumes that the number of feature vectors in the two layers is the same; in actual situations, some alignment method may be required to ensure dimensionality matching). For each pair of feature vectors vi_k and vi_k+1 at the same position, calculate their cosine similarity. The cosine similarity calculation logic is as follows: First, calculate the dot product of the two vectors (i.e., multiply and add the elements of each corresponding dimension of vi_k and vi_k+1). Then, calculate the modulus of each vector (i.e., the square root of the sum of the squares of the elements of each dimension). Finally, divide the dot product by the product of the moduli of the two vectors to obtain the cosine similarity value of this pair of feature vectors. The cosine similarity values ​​of the feature vectors at all positions are combined to form a local similarity matrix, denoted as LocalSimMatrix_k.

[0056] Next, a sliding window mean pooling process is performed on the local similarity matrix LocalSimMatrix_k. A sliding window is set whose size is related to the dimensional scaling factor of the abstract feature representation of the kth layer. For example, if the dimensional scaling factor from the kth layer to the k+1th layer is s, then the sliding window size may be set to a value related to s, such as an integer multiple of s for the window side length. The window is slided on the local similarity matrix, and for each element in the window, their average value is calculated. For example, in the area covered by the window, all elements are added and divided by the number of elements in the window to obtain the mean value of the window position. The mean values ​​of each window position are combined into a new distribution map, namely the windowed similarity distribution map WindowedSimDistribution_k.

[0057] Next, we extract regions exceeding a preset activation threshold from the windowed similarity distribution map WindowedSimDistribution_k. This threshold is set based on experience or previous experiments and is used to identify key regions with high similarity. We iterate through each value in the windowed similarity distribution map and mark regions corresponding to values ​​greater than the threshold. These marked regions form the target association region set TargetAssociationRegions_k.

[0058] Next, we calculate the coverage ratio of the target association region set TargetAssociationRegions_k in the windowed similarity distribution map WindowedSimDistribution_k. We calculate the ratio of the number of elements in the target association region set to the total number of elements in the windowed similarity distribution map, yielding a value, denoted as CoverageRatio_k. This value reflects the proportion of regions with high similarity between two adjacent layers in the overall distribution map.

[0059] The pattern similarity index is then generated by multiplying the coverage ratio value CoverageRatio_k by the preset level attenuation coefficient. The preset level attenuation coefficient is a pre-set parameter used to adjust the similarity index. Assuming the preset level attenuation coefficient is α, the pattern similarity index SimilarityIndex_k = CoverageRatio_k × α.

[0060] Finally, the comparison result between the pattern similarity index SimilarityIndex_k and the preset hierarchical transition threshold determines whether the abstract feature representation of the corresponding layer will be added to the hierarchical feature subset. The preset hierarchical transition threshold is another value set based on experience or previous experiments, which is used to determine whether the pattern similarity between two adjacent layers is high enough to perform a hierarchical transition. If the pattern similarity index SimilarityIndex_k is greater than the preset hierarchical transition threshold, the abstract feature representations of the kth layer and the k+1th layer will be added to the hierarchical feature subset at the same time; if the pattern similarity index SimilarityIndex_k is less than or equal to the preset hierarchical transition threshold and the k+1th layer is the bottom layer, only the abstract feature representation of the k+1th layer will be added to the hierarchical feature subset. In this way, by calculating the cross-layer correlation of the abstract feature representations of adjacent layers, it is possible to screen out layers with higher pattern similarity, preparing for the subsequent generation of hierarchical feature subsets.

[0061] Step S133: Based on the comparison result between the pattern similarity index and the preset dynamic threshold, abstract feature representations that meet the hierarchical transition conditions are screened from the multi-layer abstract feature representation sequence to generate a hierarchical feature subset.

[0062] After calculating the pattern similarity index SimilarityIndex_k between adjacent layers as described in the above steps, these indicators are compared with the preset dynamic threshold. The preset dynamic threshold may be a value that is dynamically adjusted with model training or data characteristics. Here, it is assumed to be DynamicThreshold. For each pattern similarity index SimilarityIndex_k, if SimilarityIndex_k is greater than DynamicThreshold, it means that the pattern similarity between the kth layer and the k+1th layer meets the hierarchical transition condition. At this time, the kth layer abstract feature representation Lk and the k+1th layer abstract feature representation Lk+1 are selected. In this way, all adjacent layers in the multi-layer abstract feature representation sequence [L1, L2,…, Ln] are judged, and the abstract feature representations of adjacent layers that meet the hierarchical transition conditions are selected to form a hierarchical feature subset HierarchicalFeatureSubset. This hierarchical feature subset includes abstract feature representations that are screened out from a multi-layer abstract feature representation sequence, have high pattern similarity and meet the hierarchical transition conditions. These feature representations are more likely to contain key information related to the underlying information pattern, providing more targeted data for subsequent pattern clustering processing.

[0063] Step S134: performing pattern clustering processing on the hierarchical feature subset to generate an initial candidate pattern set, wherein each initial candidate pattern includes a pattern center vector and a membership range.

[0064] The hierarchical feature subset HierarchicalFeatureSubset is used as input and a clustering algorithm, such as the K-Means clustering algorithm, is used. (This is just an example of a clustering algorithm type; you might actually use a more appropriate clustering algorithm for this scenario.) At the beginning of the clustering process, the number of clusters is determined, assuming it is set to K. Then, K feature vectors are randomly selected from the hierarchical feature subset as the initial pattern center vectors, denoted as CenterVectors = [c1, c2, …, cK].

[0065] For each feature vector f in the hierarchical feature subset, calculate the distance between it and each pattern center vector ci. Distance can be calculated using common distance metrics such as Euclidean distance. Taking Euclidean distance as an example, assuming that the feature vector f and the pattern center vector ci have the same d-dimensionality, f = [f1, f2, …, fd], and ci = [ci1, ci2, …, cid], then the Euclidean distance between f and ci is the square root of the sum of the squared differences in each dimension: That is, first calculate (f1-ci1)²+(f2-ci2)²+…+(fd-cid)², and then take the square root of the result. After calculating the distances between f and all pattern center vectors, assign f to the category corresponding to the pattern center vector with the closest distance.

[0066] After all feature vectors in the hierarchical feature subsets have been assigned, the mean of all feature vectors in each category is recalculated and used as the new pattern center vector. For example, for category j, which contains feature vectors f1_j, f2_j, ..., fm_j, the new pattern center vector cj_new has each dimension value equal to the average of the corresponding dimension values ​​of all feature vectors in that category. That is, the k-th dimension value of cj_new is (f1_j[k]+f2_j[k]+...+fm_j[k]) / m (assuming there are m feature vectors in that category).

[0067] Repeat the above process of assigning feature vectors and updating pattern center vectors until the change in pattern center vector is less than a preset convergence threshold or the preset maximum number of iterations is reached. After these steps, K pattern center vectors are obtained, each corresponding to a cluster category.

[0068] For each cluster category, calculate the distance distribution between the feature vectors in that category and the pattern center vector to determine the membership range. For example, you can calculate the maximum and minimum distances between all feature vectors in that category and the pattern center vector, and use this distance range as the membership range for the feature vectors in that category. In this way, each initial candidate pattern consists of the pattern center vector and the corresponding membership range, and all initial candidate patterns constitute the initial candidate pattern set InitialCandidatePatternSet.

[0069] Step S135: Based on the distance distribution between the pattern center vector and each feature vector in the associated feature set, calculate the feature vector distribution density of each initial candidate pattern within a preset neighborhood range as the information density value, and determine the coverage radius according to the maximum distance threshold.

[0070] For each initial candidate pattern in the initial candidate pattern set InitialCandidatePatternSet, its pattern center vector is denoted as cp. In the associated feature set AssociatedFeatures, the distance between each feature vector af and the pattern center vector cp is calculated. Here, the Euclidean distance is still used as an example for calculation, as described above.

[0071] Set a preset neighborhood, such as a spherical region with a radius of r (where r is the preset neighborhood radius) centered around the pattern center vector cp. Count the number of feature vectors in the associated feature set within this preset neighborhood, denoted by N. Also, calculate the volume of this neighborhood (for a spherical neighborhood, the volume formula is 4 / 3πr³), denoted by V. The density of feature vectors for the initial candidate pattern within the preset neighborhood, i.e., the information density, InfoDensity = N / V.

[0072] In addition, the coverage radius is determined based on a maximum distance threshold. A maximum distance threshold, MaxDistance, is set. The feature vector from the associated feature set that is farthest from the pattern center vector cp and less than or equal to MaxDistance is found. The distance between this farthest feature vector and the pattern center vector is the coverage radius, CoverageRadius. By calculating the information density value and determining the coverage radius, we provide a basis for subsequent redundant merging of the initial candidate pattern set.

[0073] Step S136: performing redundant merging processing on the initial candidate pattern set according to the information density value and the coverage radius, merging the initial candidate patterns whose information density value difference within the overlapping coverage radius is less than a preset merging threshold, and generating an optimized pattern set.

[0074] Traverse every two initial candidate patterns in the initial candidate pattern set InitialCandidatePatternSet, assuming they are pattern i and pattern j, whose pattern center vectors are cpi and cpj respectively, whose information density values ​​are InfoDensity_i and InfoDensity_j respectively, and whose coverage radii are CoverageRadius_i and CoverageRadius_j respectively.

[0075] First, determine whether the coverage radiuses of the two patterns overlap. Specifically, determine whether the area centered on cpi with a radius of CoverageRadius_i intersects with the area centered on cpj with a radius of CoverageRadius_j. If so, further compare their information density values. Calculate the absolute value of the difference between InfoDensity_i and InfoDensity_j, denoted as |InfoDensity_i - InfoDensity_j|.

[0076] The absolute value of this difference is compared with the preset merge threshold (MergeThreshold). If |InfoDensity_i - InfoDensity_j| is less than MergeThreshold, the two initial candidate patterns have a small difference in information density and overlap in coverage radius. In this case, the two patterns are merged. This merging can be done by recalculating the pattern center vector of the merged pattern. For example, by calculating the mean of all feature vectors contained in the two patterns (feature vectors from corresponding regions in the associated feature set) to obtain a new pattern center vector. The merged pattern's coverage radius, information density, and other related parameters are also recalculated.

[0077] In this way, all pattern pairs in the initial candidate pattern set are judged and processed, and patterns that meet the merging conditions are merged to finally generate the optimized pattern set OptimizedPatternSet. This optimized pattern set reduces redundant patterns, making the patterns more refined and more accurately representing the underlying information patterns.

[0078] Step S137: extracting the feature unit identifier corresponding to each pattern center vector from the optimization pattern set, associating the resource attribute labels and path topology relationships in the semantic association features, and generating the attribute identifier and distribution path of the significant value information resource.

[0079] For each pattern in the optimized pattern set OptimizedPatternSet, its pattern center vector corresponds to certain eigenvectors in the associated feature set. By recording these correspondences, the feature unit identifiers corresponding to the pattern center vectors can be found. Assuming that the pattern center vector of pattern p is cp, the feature units to which these eigenvectors belong are determined by searching the eigenvectors in the associated feature set that are closest to cp or that meet the specified matching conditions, thereby obtaining the feature unit identifiers FeatureUnitIDs_p.

[0080] Then, in the semantic association feature set, the resource attribute labels and path topology relationships associated with these feature unit identifiers are found. The semantic association feature set records the semantic associations between different types of data units. The feature unit identifiers can be used to index the corresponding resource attribute labels (ResourceAttributeLabels) and path topology relationships (PathTopologyRelations).

[0081] The feature unit identifier FeatureUnitIDs_p, resource attribute labels ResourceAttributeLabels, and path topology relations PathTopologyRelations are associated and combined to generate the attribute identifiers and distribution paths of significant value information resources. For example, using the feature unit identifier as an index, the corresponding resource attribute labels and path topology relations are combined to form an information pair (AttributeID, DistributionPath) containing the attribute identifier and distribution path, where the AttributeID is composed of the resource attribute labels and the DistributionPath is determined by the path topology relations. This process is repeated for all patterns in the optimization pattern set to obtain a series of attribute identifiers and distribution paths of significant value information resources.

[0082] Step S138: Aggregate the attribute identifiers and distribution paths of all significant value information resources to construct the potential information identification result set.

[0083] Aggregate the attribute identifiers and distribution paths of all significant value information resources obtained through the above steps. For example, collect the (AttributeID, DistributionPath) information pairs generated by each pattern and form a set, recorded as PotentialInfoRecognitionResultSet. This potential information recognition result set contains the attribute identifiers and distribution paths of multiple significant value information resources. It is obtained through hierarchical pattern matching of associated feature sets and provides key information for the subsequent generation of dynamic resource integration strategies.

[0084] Step S140: generating a dynamic resource integration strategy according to the potential information identification result set, wherein the dynamic resource integration strategy is used to adjust resource scheduling rules and storage path mapping relationships in a heterogeneous network environment.

[0085] In this embodiment, a dynamic resource integration strategy is generated based on a potential information identification result set to optimize resource scheduling and storage paths in a heterogeneous network environment.

[0086] Step S141: parsing the attribute identifier of each significant value information resource in the potential information identification result set to determine the data category and access permission level to which it belongs.

[0087] In the PotentialInfoRecognitionResultSet, for each significant value information resource's attribute ID, pre-defined parsing rules are used to determine its data category. Assuming different data category IDs are pre-defined, such as DataCategory1, DataCategory2, and so on, these IDs may contain specific fields or codes to indicate the data category. By extracting and analyzing these fields or codes, the data category of the significant value information resource is determined, for example, DataCategoryX.

[0088] At the same time, information related to the access permission level is parsed from the attribute identifier. The access permission level may be represented by a code or numerical range, such as AccessLevel1, AccessLevel2, etc. By parsing the attribute identifier, the access permission level of the significant value information resource is determined to be AccessLevelY. This parsing is performed on the attribute identifiers of all significant value information resources in the potential information identification result set to determine the data category and access permission level of each resource.

[0089] Step S142: extracting the storage location identifier and transmission path topology relationship of each significant value information resource according to the distribution path.

[0090] The distribution path (DistributionPath) of each significant value information resource in the potential information identification result set is parsed according to predefined formats and rules. The distribution path may exist in a form that includes a description of the storage location and transmission path, for example, it may be a representation of a series of nodes and connection relationships. By parsing the distribution path, the storage location identifiers (StorageLocationIDs) are extracted. The storage location identifier may be the storage device number, address, or other information that uniquely identifies the storage location.

[0091] At the same time, the transmission path topology (TransmissionPathTopology) is extracted from the distribution path. The transmission path topology describes the structure of the path along which data is transmitted within the network, potentially including information such as the order and direction of connections between nodes. For example, the transmission path topology might be represented as a directed connection from node A through node B to node C. This extraction operation is performed on the distribution paths of all significant information resources, resulting in the storage location identifier and transmission path topology for each resource.

[0092] Step S143: Generate resource scheduling priority rules based on the data category and access permission level, and the resource scheduling priority rules define the response order and concurrent processing quantity of different categories of resources.

[0093] Generate resource scheduling priority rules based on the previously determined data categories and access permission levels. First, classify data according to their categories. Different data categories may have different characteristics such as importance or usage frequency. For example, for data category DataCategory1, if it is a data category that is critical to the business and frequently accessed, then assign it a higher priority; for data category DataCategory2, if its importance is relatively low, then assign it a lower priority. Here, the priority can be represented by a number or some priority code, assuming PriorityCode1, PriorityCode2, ..., and assign corresponding priority codes to different data categories.

[0094] Also, consider access rights levels. Resources with higher access rights, such as those with Access Level 1, may have higher scheduling priority. The final priority for each significant value information resource is determined based on the data category and access rights level.

[0095] Based on these priorities, the response order of resource scheduling is defined. For example, resources with higher priorities are processed first during scheduling. That is, when there are multiple resources waiting to be scheduled, resources with priority code 1 are processed first, followed by resources with priority code 2, and so on.

[0096] In addition, the number of concurrent processes must be determined. For resources of different priorities, different numbers of concurrent processes are set based on the system's processing capabilities and resource characteristics. For example, for the highest-priority resource, the number of concurrent processes allowed may be ConcurrentNum1; for lower-priority resources, the number of concurrent processes allowed may be ConcurrentNum2 (ConcurrentNum1 and ConcurrentNum2 are examples here only; the actual number depends on the system's conditions). In this way, resource scheduling priority rules are generated that clarify the response order and concurrent processing number for different categories of resources.

[0097] Step S144: generating a storage path optimization mapping table based on the storage location identifier and the transmission path topology relationship, wherein the storage path optimization mapping table includes a correspondence between resource identifiers and optimal storage nodes.

[0098] First, for each significant value information resource, the optimal storage node is determined by combining its storage location identifier and transmission path topology. The storage location identifier may contain multiple pieces of relevant information, such as the storage area number and storage device type. Assuming the storage location identifier consists of multiple fields, such as LocationField1 and LocationField2, potential storage nodes are initially screened by analyzing these fields and applying specific rules.

[0099] The transmission path topology provides structural information about data transmission within the network. For example, it indicates characteristics such as the transmission path length and transmission delay from the source node to different potential storage nodes. Assume that the transmission path topology can be represented by the connection weights between nodes. The weights may represent factors such as transmission delay or bandwidth. For each potential storage node, the comprehensive transmission cost from the data source (the data source's related nodes can be determined based on the transmission path topology) to that storage node is calculated. This comprehensive transmission cost calculation may involve some combination of the weights of the connections along the transmission path, such as summing the weights of all connections along the path (this is just an example logic).

[0100] By comparing the comprehensive transmission costs of all potential storage nodes, the storage node with the lowest comprehensive transmission cost is selected as the optimal storage node for the significant value information resource. Assume that for the significant value information resource identified by ResourceID, the optimal storage node determined by the above calculation is OptimalNode. The ResourceID and OptimalNode form a corresponding relationship pair (ResourceID, OptimalNode).

[0101] This process is repeated for all significant information resources in the potential information identification result set, and all corresponding relationship pairs are summarized to form a storage path optimization mapping table, StoragePathOptimizationMapTable. This storage path optimization mapping table clearly records the relationship between each resource identifier and the corresponding optimal storage node.

[0102] Step S145: performing policy fusion processing on the resource scheduling priority rule and the storage path optimization mapping table to generate the dynamic resource integration policy.

[0103] This step aims to integrate resource scheduling priority rules and storage path optimization mapping tables to generate a comprehensive dynamic resource integration strategy.

[0104] Step S1451: Call the node monitoring interface of the target network environment to obtain the real-time load status data of the storage node, combine the response order in the resource scheduling priority rule, perform load balancing analysis and processing on the storage nodes in the storage path optimization mapping table, and determine the real-time load status of each storage node.

[0105] By calling the node monitoring interface provided by the target network environment, you can obtain real-time load status data for each storage node. Assume that this data includes information on multiple dimensions, such as the storage node's current storage usage rate, data transfer rate, and number of processing tasks, which are recorded as StorageUsageRate, DataTransferRate, and TaskCount, respectively.

[0106] Combined with the response order in the resource scheduling priority rules, a load balancing analysis is performed on the storage nodes in the storage path optimization mapping table. For resources with higher priorities, their corresponding storage nodes have higher weights in the load balancing analysis. For example, for resources with a priority of PriorityHigh, the load of the corresponding storage node has a weight of WeightHigh in the overall analysis; for resources with a priority of PriorityLow, the weight of the corresponding storage node load is WeightLow, and WeightHigh is greater than WeightLow.

[0107] A load balancing analysis algorithm, such as one that considers factors like storage utilization, data transfer rate, and the number of processing tasks, calculates the composite load index for each storage node. Assume the composite load index is calculated as follows: multiply the storage utilization rate by a weight coefficient, WeightUsage, add the data transfer rate by a weight coefficient, WeightTransfer, and add the number of processing tasks by a weight coefficient, WeightTask. Specifically, CompositeLoadIndex = StorageUsageRate × WeightUsage + DataTransferRate × WeightTransfer + TaskCount × WeightTask. This calculation determines the real-time load status of each storage node.

[0108] Step S1452: If the real-time load status of the target storage node exceeds the capacity threshold, search for a backup storage node in the storage path optimization mapping table based on the business priority weight of the resource, prioritize migrating resources whose business priority weight is lower than the preset migration threshold, and update the corresponding relationship.

[0109] Set a capacity threshold, CapacityThreshold, to determine whether a storage node is overloaded. For each storage node, compare its calculated composite load index, CompositeLoadIndex, with the capacity threshold, CapacityThreshold. If the CompositeLoadIndex of a target storage node exceeds CapacityThreshold, the storage node is overloaded and requires resource migration.

[0110] In the resource scheduling priority rules, each resource is assigned a business priority weight (BusinessPriorityWeight). In the storage path optimization mapping table, a search is performed on alternative storage nodes other than the currently overloaded storage node. The resources' business priority weights are checked, in ascending order, to see if they are below the preset migration threshold (MigrationThreshold). Resources with a business priority weight below the MigrationThreshold are migrated from the overloaded target storage node to an alternative storage node.

[0111] Assume that resource ResourceA was originally stored on the overloaded target storage node TargetNode. When searching for a backup storage node, it is found that the backup storage node BackupNode has sufficient space and performance to accommodate ResourceA. After migrating ResourceA to BackupNode, the corresponding relationship in the storage path optimization mapping table is updated, forming a new corresponding relationship pair (ResourceA, BackupNode) between ResourceA and BackupNode, and deleting the original corresponding relationship (ResourceA, TargetNode).

[0112] Step S1453: adjusting the concurrent processing quantity in the resource scheduling priority rule according to the updated storage path optimization mapping table so that the concurrent processing quantity of the preset first priority resource matches the capacity upper limit of the storage node with sufficient remaining available capacity.

[0113] After completing resource migration and updating the storage path optimization mapping table, adjust the concurrent processing capacity in the resource scheduling priority rules. First, identify the preset first-priority resources, assuming they are PriorityFirstResources. For these resources, adjust their concurrent processing capacity based on the capacity caps of storage nodes with sufficient remaining available capacity.

[0114] Traverse the updated storage path optimization mapping table and count the remaining available capacity of each storage node. Assume that the remaining available capacity of storage node Node1 is AvailableCapacity1, the remaining available capacity of Node2 is AvailableCapacity2, and so on. Identify storage nodes with sufficient remaining available capacity and set a sufficient capacity criterion. For example, a storage node with a remaining available capacity greater than a certain threshold, AdequateCapacityThreshold, is considered a node with sufficient available capacity.

[0115] For these storage nodes with sufficient available capacity, determine their capacity upper limit CapacityUpperLimit. For example, for a group of storage nodes with sufficient available capacity, their capacity upper limits are CapacityUpperLimit1, CapacityUpperLimit2, and so on. Based on these capacity upper limits, calculate the number of concurrent processing of the preset first-priority resources that can be supported. Assume that the calculation logic is: add the capacity upper limits of all storage nodes with sufficient available capacity to obtain the total capacity upper limit TotalCapacityUpperLimit, and then determine the number of concurrent processing of the preset first-priority resources AdjustedConcurrentNum based on the preset resource allocation strategy (such as average distribution or allocation based on resource characteristics). For example, if the average distribution strategy is adopted and the number of preset first-priority resources is NumPriorityFirstResources, then AdjustedConcurrentNum = TotalCapacityUpperLimit / NumPriorityFirstResources.

[0116] Then, in the resource scheduling priority rule, the concurrent processing quantity of the preset first priority resource is adjusted to AdjustedConcurrentNum.

[0117] Step S1454: Combining the adjusted resource scheduling priority rule with the updated storage path optimization mapping table to generate a fused dynamic resource integration strategy;

[0118] Combine the adjusted resource scheduling priority rules and the updated storage path optimization mapping table. The resource scheduling priority rules define the response order and concurrent processing capacity of different resource categories, while the storage path optimization mapping table determines the correspondence between resources and optimal storage nodes.

[0119] These two components are integrated to form a unified policy document or data structure, which serves as the fused dynamic resource integration strategy, DynamicResourceIntegrationStrategy. For example, resource scheduling priority rules can be represented in a data format (such as JSON), containing information such as resource category, priority, response order, and number of concurrent processes. The storage path optimization mapping table can also be represented in a corresponding data format (such as a key-value pair, with the resource identifier as the key and the optimal storage node as the value). These two data structures are then combined into a larger data structure that contains policy information for both resource scheduling and storage paths, thereby generating a complete dynamic resource integration strategy.

[0120] Step S1455: Conflict detection is performed on the dynamic resource integration strategy to ensure that there is no logical contradiction in the execution order of different rules.

[0121] Perform conflict detection on the generated dynamic resource integration strategy DynamicResourceIntegrationStrategy. Check whether there are logical conflicts between the resource scheduling priority rules and the storage path optimization mapping table. For example, the resource scheduling priority rules may specify that certain high-priority resources require priority processing. However, in the storage path optimization mapping table, the storage nodes corresponding to these high-priority resources may not be able to provide services in a timely manner due to other reasons (such as maintenance status or incomplete overload processing), which may lead to conflicts.

[0122] To address this situation, a series of detection rules can be set. First, check whether the response order in the resource scheduling priority rules matches the availability of storage nodes. For each priority level of resources, check whether the corresponding storage node is available (storage node information is obtained through the storage path optimization mapping table and combined with real-time status data obtained from the node monitoring interface). If the storage node corresponding to a high-priority resource is unavailable, and the resource scheduling rules require that resource be processed first, this will cause a conflict.

[0123] Secondly, check whether the number of concurrent processing requests matches the capacity of the storage node. Based on the storage node capacity information in the storage path optimization mapping table and the concurrent processing request setting in the resource scheduling priority rules, determine whether the number of resources being processed simultaneously exceeds the storage node's processing power or storage capacity. For example, if a storage node's remaining available capacity can only support the simultaneous storage and processing of a certain number of resources, and the concurrent processing request set in the resource scheduling rules exceeds this limit, a conflict will occur.

[0124] If a conflict is detected, it is handled according to the pre-set conflict resolution strategy. For example, if the conflict is caused by an unavailable storage node, the storage path optimization mapping table can be re-searched for a backup storage node and the relevant mapping can be updated. If the conflict is caused by a mismatch between the number of concurrent processing and the storage node capacity, the number of concurrent processing in the resource scheduling priority rule can be further adjusted to align with the actual capacity of the storage node. This conflict detection and resolution ensures that the execution order of different rules in the dynamic resource integration strategy is logically consistent, ensuring the effectiveness and feasibility of the strategy.

[0125] Step S150: Feedback the dynamic resource integration strategy to the data management system of the target network environment, triggering the data management system to execute a resource directory update operation.

[0126] The dynamic resource integration strategy DynamicResourceIntegrationStrategy after conflict detection and processing is fed back to the data management system of the target network environment. The data management system is responsible for managing various resource information in the network environment, including the storage location and attributes of the resources.

[0127] After receiving a dynamic resource consolidation policy, the data management system first parses it. Based on the storage path optimization mapping table in the policy, it determines the resource storage location information that needs to be updated. For example, if the mapping table indicates that resource ResourceX needs to be migrated from storage node NodeA to storage node NodeB, the data management system records the change.

[0128] At the same time, according to the resource scheduling priority rules, the data management system updates the resource scheduling priority information. For example, it raises the priority of some resources from a lower level to a higher level, or adjusts the number of concurrent processing.

[0129] After parsing the dynamic resource integration policy, the data management system begins updating the resource directory. It traverses the system's resource directory and updates each resource accordingly based on the information in the dynamic resource integration policy. If a resource's storage location changes, the data management system modifies the resource's storage location field in the resource directory to point to the new storage node. If a resource's scheduling priority changes, the data management system updates the resource's priority field in the resource directory.

[0130] In this way, the data management system completes the update operation of the resource directory, so that the resource management in the system is consistent with the dynamic resource integration strategy, thereby realizing the optimized scheduling of resources and reasonable adjustment of storage paths in heterogeneous network environments, and improving the resource utilization efficiency and operating performance of the entire network environment.

[0131] The above embodiments involve a pre-trained information pattern analysis model. The following details the construction and training of this artificial intelligence model and its integration with the overall technical solution.

[0132] The information pattern analysis model primarily consists of a feature encoding layer and a pattern matching layer. The feature encoding layer performs a layer-by-layer nonlinear transformation on the input feature set to generate a multi-layer sequence of abstract feature representations. This layer comprises multiple neural network layers, each tightly connected. Assume that starting from the input layer, the layers are layer 1, layer 2, and so on. The input layer receives the feature set as input, and its number of neurons corresponds to the dimensionality of the feature set to ensure complete reception of the input information.

[0133] The first layer is fully connected to the input layer and performs the first nonlinear transformation on the input correlation features. Common activation functions, such as the ReLU function, can be used here to introduce nonlinearity. At this layer, the input features are weighted and summed using a weight matrix. Then, the activation function processes the correlation features, mapping them from their original dimensions to a new dimension. Assume that the original dimension of the correlation feature set is d0, and the feature dimension of the first layer output is d1, with d1 and d0 satisfying a preset dimensionality reduction ratio r, i.e., d1 = r * d0. This dimensionality reduction process helps extract more critical feature information and reduces data redundancy.

[0134] The second layer is also fully connected to the first layer and performs further processing based on the features output by the first layer. Similar to the first layer, the second layer performs a weighted summation of the input features using a weight matrix, then processes them through an activation function (which can also be a ReLU function), outputting features of dimension d2 = r * d1. Similarly, each layer takes the output of the previous layer as input, performs a nonlinear transformation using the weight matrix and activation function, and then performs a dimensionality reduction according to a preset ratio, ultimately generating a multi-layered sequence of abstract feature representations [L1, L2, …, Ln].

[0135] The pattern matching layer is responsible for calculating the cross-layer correlation between the multi-layer abstract feature representation sequence to determine the pattern similarity index between the abstract feature representations of adjacent layers. This layer mainly operates based on the multi-layer abstract feature representation sequence output by the feature encoding layer.

[0136] When it comes to model training, the first step is to prepare a large amount of training data. This training data should have similar characteristics to the heterogeneous network environment data in actual application scenarios, contain raw information resources in multiple data formats, and be annotated with potential information recognition results, that is, the attribute identification and distribution path of significant valuable information resources.

[0137] The training steps are as follows:

[0138] 1. Initialize model parameters: Randomly initialize the weight matrices and bias terms of each feature encoding layer. These initial values ​​will be continuously adjusted during training through the backpropagation algorithm to optimize model performance.

[0139] 2. Forward propagation: The set of correlated features in the training data is input into the information pattern analysis model. After layer-by-layer nonlinear transformations in the feature encoding layer, a multi-layer sequence of abstract feature representations is generated. This sequence then enters the pattern matching layer, where cross-layer correlation calculations are performed to obtain a pattern similarity index, which is then used to generate potential information recognition results.

[0140] 3. Loss Calculation: Compare the potential information recognition results generated by the model with the annotated true results. Use an appropriate loss function (such as the cross-entropy loss function) to calculate the difference between the two and obtain a loss value. The loss value reflects the degree of deviation between the model's current prediction and the true result.

[0141] 4. Backpropagation: Based on the loss value, the backpropagation algorithm calculates the gradient of the loss value with respect to the parameters of each layer of the model (weight matrix and bias terms). The gradient indicates the direction and degree of influence of a small change in the parameter on the loss value.

[0142] 5. Parameter Update: Optimizers (such as stochastic gradient descent (SGD), Adagrad, and Adadelta) are used to update the model's parameters based on the calculated gradients. The optimizer adjusts the parameters to reduce the loss. For example, in stochastic gradient descent (SGD), the parameters are updated at a set learning rate, in the opposite direction of the gradient. The learning rate is a pre-set parameter that controls the step size of each parameter update.

[0143] 6. Repeated training: Repeat the steps of forward propagation, loss calculation, backpropagation, and parameter update described above, performing multiple iterations of training on a large amount of training data. With each iteration, the model parameters are continuously adjusted, the loss gradually decreases, and the model performance gradually improves. Training ends when the loss converges to a smaller value or the preset maximum number of iterations is reached.

[0144] In practical applications, the information pattern analysis model is closely integrated with the overall potential information identification method. The model input is a set of associated features obtained after feature extraction. This set contains the semantic association features and dynamic interaction features between the original information resources in the heterogeneous network environment data set. Through processing at the feature encoding layer and pattern matching layer, the model outputs a set of potential information identification results, including the attribute identifiers and distribution paths of multiple significant information resources. These outputs are directly used to subsequently generate dynamic resource integration strategies to adjust resource scheduling rules and storage path mapping relationships in the heterogeneous network environment.

[0145] During the above data collection, label management, and rule setting processes, we must strictly adhere to laws, regulations, and ethical standards to ensure that there are no violations of laws, injustices, or discrimination or bias. For example, during the data collection phase, we must clearly inform data providers of the purpose, scope, and protection measures of the data to ensure the legality and transparency of data collection. In terms of label management, we must strictly review the results of labeled potential information identification to avoid unfair decisions due to label errors or bias. In terms of rule setting, we must ensure that the formulation of rules such as resource scheduling priority rules and storage path optimization mapping tables are based on objective resource characteristics and needs, rather than discriminatory factors.

[0146] Figure 2 The following diagram illustrates exemplary hardware and software components of a system 100 for identifying potential information in a heterogeneous network environment, which can implement the concepts of the present application, as provided in some embodiments of the present application. For example, the processor 120 can be used in the system 100 for identifying potential information in a heterogeneous network environment and perform the functions of the present application.

[0147] The system 100 for identifying potential information in a heterogeneous network environment can be a general-purpose server or a special-purpose server, both of which can be used to implement the method for identifying potential information in a heterogeneous network environment of the present application. Although only one server is shown in this application, for convenience, the functions described in this application can be implemented in a distributed manner on multiple similar platforms to balance the processing load.

[0148] For example, the potential information identification system 100 in combination with a heterogeneous network environment may include a network port 110 connected to the network, one or more processors 120 for executing program instructions, a communication bus 130, and storage media 140 in different forms, such as a disk, ROM, or RAM, or any combination thereof. Exemplarily, the potential information identification system 100 in combination with a heterogeneous network environment may also include program instructions stored in ROM, RAM, or other types of non-transitory storage media, or any combination thereof. The method of the present application can be implemented according to these program instructions. The potential information identification system 100 in combination with a heterogeneous network environment also includes an I / O interface 150 between the computer and other input and output devices.

[0149] For ease of explanation, only one processor is described in the potential information identification system 100 in combination with a heterogeneous network environment. However, it should be noted that the potential information identification system 100 in combination with a heterogeneous network environment in the present application may also include multiple processors, so the steps performed by one processor described in the present application may also be performed jointly or individually by multiple processors. For example, if the processor of the potential information identification system 100 in combination with a heterogeneous network environment executes step A and step B, it should be understood that step A and step B may also be performed jointly by two different processors or individually in one processor. For example, the first processor executes step A, the second processor executes step B, or the first processor and the second processor execute steps A and B together.

[0150] In addition, an embodiment of the present invention further provides a readable storage medium, in which computer-executable instructions are preset. When a processor executes the computer-executable instructions, the above-mentioned potential information identification method in combination with a heterogeneous network environment is implemented.

[0151] It should be noted that in order to simplify the description of the present invention and thus help understand one or more embodiments of the invention, in the foregoing description of the embodiments of the present invention, multiple features are sometimes combined into one embodiment, figure or description thereof.

Claims

1. A potential information identification method in a heterogeneous network environment, characterized in that: The method comprises: Acquire a heterogeneous network environment data set of a target network environment, wherein the heterogeneous network environment data set includes original information resources in multiple data formats; Extracting features from the heterogeneous network environment data set to obtain a correlation feature set, wherein the correlation feature set includes semantic correlation features and dynamic interaction features between the original information resources; Based on a pre-trained information pattern analysis model, hierarchical pattern matching processing is performed on the associated feature set to generate a potential information recognition result set, wherein the potential information recognition result set includes attribute identifiers and distribution paths of multiple significant value information resources; Generating a dynamic resource integration strategy based on the potential information identification result set, wherein the dynamic resource integration strategy is used to adjust resource scheduling rules and storage path mapping relationships in a heterogeneous network environment; Feedback the dynamic resource integration strategy to the data management system of the target network environment, triggering the data management system to perform a resource directory update operation; The extracting features from the heterogeneous network environment data set to obtain a related feature set includes: Performing semantic vectorization processing on the text data units in the heterogeneous network environment data set to generate a text semantic feature set, wherein each text semantic feature in the text semantic feature set corresponds to a semantic vector representation of a text data unit; Performing spatial feature encoding processing on image data units in the heterogeneous network environment data set to generate an image spatial feature set, wherein each image spatial feature in the image spatial feature set includes object distribution information and contextual relationship information in the image data unit; Performing dynamic pattern analysis on the time series data units in the heterogeneous network environment data set to generate a time series dynamic feature set, wherein each time series dynamic feature in the time series dynamic feature set includes data change trend and periodic fluctuation information; Performing cross-modal alignment processing on the text semantic feature set, the image spatial feature set, and the temporal dynamic feature set to determine semantic association features between features of different modalities, and generating dynamic interaction features based on feature interaction frequencies; Aggregating the semantic association features and the dynamic interaction features to construct the association feature set; The cross-modal alignment of the text semantic feature set, the image spatial feature set, and the temporal dynamic feature set to determine semantic association features between features of different modalities includes: For each text data unit, the text semantic feature corresponding to the text data unit is matched with the image data unit and the time series data unit having the synchronized time stamp within a preset time window according to the time stamp thereof; After mapping text semantic features, image spatial features, and temporal dynamic features to a unified dimensional space through a projection layer, a cross-modal alignment model is called to perform joint encoding on the mapped features to generate cross-modal joint encoding features. Performing similarity calculation processing on the cross-modal joint encoding features to determine a first correlation score between the text semantic features and the image spatial features, and a second correlation score between the text semantic features and the temporal dynamic features; Constructing a semantic association matrix according to the first association score and the second association score, wherein the rows of the semantic association matrix represent text data unit identifiers, the columns represent image data unit identifiers or time series data unit identifiers, and the matrix elements represent corresponding association scores; Extracting feature combinations corresponding to association scores exceeding a preset threshold in the semantic association matrix to generate the semantic association features; Generating dynamic interaction features based on feature interaction frequency includes: Divide each data unit in the heterogeneous network environment data set into blocks according to data category, count the number of interaction events between data units in each block, and construct a block interaction frequency matrix, where the rows and columns of the interaction frequency matrix represent different data unit identifiers, and the matrix elements represent the number of interactions between corresponding data units; Decomposing the interaction frequency matrix in a time dimension to obtain short-term interaction frequency characteristics and long-term interaction trend characteristics; Performing difference calculation on the short-term interaction frequency feature and the long-term interaction trend feature to generate a dynamic fluctuation feature; Determining the dynamic interaction intensity between the data units based on the change direction and amplitude of the dynamic fluctuation characteristics; Performing weighted correction processing on the semantic association feature based on the dynamic interaction strength to generate the dynamic interaction feature; The pre-trained information pattern analysis model is used to perform hierarchical pattern matching processing on the associated feature set to generate a potential information recognition result set, including: Inputting the associated feature set into the feature coding layer of the information pattern analysis model, performing layer-by-layer nonlinear transformation processing to generate a multi-layer abstract feature representation sequence, wherein the dimension of the n-th layer abstract feature representation is in a preset dimensionality reduction ratio with the dimension of the n-1-th layer abstract feature representation; In the pattern matching layer of the information pattern analysis model, cross-layer correlation calculation is performed on the multi-layer abstract feature representation sequence to determine the pattern similarity index between the abstract feature representations of adjacent layers; According to the comparison result of the pattern similarity index and the preset dynamic threshold, the abstract feature representation that meets the hierarchical transition condition is selected from the multi-layer abstract feature representation sequence to generate a hierarchical feature subset; Performing pattern clustering processing on the hierarchical feature subset to generate an initial candidate pattern set, wherein each initial candidate pattern includes a pattern center vector and a membership range; Based on the distance distribution between the pattern center vector and each feature vector in the associated feature set, the feature vector distribution density of each initial candidate pattern within a preset neighborhood is calculated as the information density value, and the coverage radius is determined according to the maximum distance threshold; Performing redundant merging processing on the initial candidate pattern set according to the information density value and the coverage radius, merging the initial candidate patterns whose information density value difference within the overlapping coverage radius is less than a preset merging threshold, and generating an optimized pattern set; Extracting the feature unit identifier corresponding to each pattern center vector from the optimization pattern set, associating the resource attribute labels and path topology relationships in the semantic association features, and generating the attribute identifier and distribution path of the significant value information resource; Aggregate the attribute identifiers and distribution paths of all significant value information resources to construct the potential information identification result set.

2. The method for identifying potential information in a heterogeneous network environment according to claim 1, characterized in that: The performing cross-layer correlation calculation on the multi-layer abstract feature representation sequence to determine the pattern similarity index between the abstract feature representations of adjacent layers includes: For the k-th layer abstract feature representation and the k+1-th layer abstract feature representation, calculate the cosine similarity of the two at the same feature unit position to generate a local similarity matrix; Performing a sliding window mean pooling process on the local similarity matrix to generate a windowed similarity distribution map, wherein the sliding window size matches the dimensionality scaling factor of the k-th layer abstract feature representation; Extracting regions exceeding a preset activation threshold in the windowed similarity distribution map to generate a target associated region set; Counting the coverage ratio of the target associated region set in the windowed similarity distribution map, and calculating the global similarity weight between the k-th layer and the k+1-th layer; Generate the pattern similarity index according to the product of the global similarity weight and the preset level attenuation coefficient; If the pattern similarity index is greater than a preset level transition threshold, the k-th level and the k+1-th level abstract feature representations are simultaneously added to the hierarchical feature subset; If the pattern similarity index is less than or equal to the hierarchical transition threshold and the k+1th layer is the lowest layer, only the k+1th layer abstract feature representation is added to the hierarchical feature subset.

3. The method for identifying potential information in a heterogeneous network environment according to claim 1, characterized in that: Generating a dynamic resource integration strategy based on the potential information identification result set includes: Analyze the attribute identifiers of each significant value information resource in the potential information identification result set to determine the data category and access permission level to which it belongs; Extracting the storage location identifier and transmission path topology relationship of each significant value information resource according to the distribution path; Generate resource scheduling priority rules based on the data category and access permission level, wherein the resource scheduling priority rules define the response order and concurrent processing quantity of different categories of resources; Generate a storage path optimization mapping table based on the storage location identifier and the transmission path topology relationship, wherein the storage path optimization mapping table includes a correspondence between the resource identifier and the optimal storage node; The resource scheduling priority rule is subjected to policy fusion processing with the storage path optimization mapping table to generate the dynamic resource integration strategy.

4. The method for identifying potential information in a heterogeneous network environment according to claim 3, characterized in that: The step of performing policy fusion processing on the resource scheduling priority rule and the storage path optimization mapping table to generate the dynamic resource integration strategy includes: Calling the node monitoring interface of the target network environment to obtain real-time load status data of the storage nodes, combining the response order in the resource scheduling priority rule, performing load balancing analysis on the storage nodes in the storage path optimization mapping table, and determining the real-time load status of each storage node; If the real-time load status of the target storage node exceeds the capacity threshold, the backup storage node is searched in the storage path optimization mapping table according to the service priority weight of the resource, and the resources with service priority weight lower than the preset migration threshold are preferentially migrated and the corresponding relationship is updated; Adjust the concurrent processing quantity in the resource scheduling priority rule according to the updated storage path optimization mapping table so that the concurrent processing quantity of the preset first priority resource matches the capacity upper limit of the storage node with sufficient remaining available capacity; Combine the adjusted resource scheduling priority rules with the updated storage path optimization mapping table to generate a fused dynamic resource integration strategy. Conflict detection is performed on the dynamic resource integration strategy to ensure that there is no logical contradiction in the execution order of different rules.

5. The method for identifying potential information in a heterogeneous network environment according to claim 3, characterized in that: Before feeding back the dynamic resource integration strategy to the data management system of the target network environment, the method further includes: Performing feasibility verification on the dynamic resource integration strategy includes: Deploy the dynamic resource integration strategy in a simulation environment and monitor the response time and error rate of the resource scheduling process by injecting multi-scenario stress test data; If the response time exceeds a preset time threshold or the error rate exceeds a preset fault tolerance threshold, the resource scheduling rules in the dynamic resource integration strategy are iteratively optimized; Re-execute the feasibility verification process until the response time and error rate meet the preset conditions; Mark the verified dynamic resource integration strategy as an executable strategy, and generate a strategy version identifier and execution log template; The iterative optimization of the resource scheduling rules in the dynamic resource integration strategy includes: Analyze scheduling tasks with excessive response times and determine their associated resource categories and path nodes; Lowering the priority of the resource category in the resource scheduling priority rule and allocating additional concurrent processing channels to it; For scheduling tasks with an excessive error rate, check the node connection status in their storage path mapping relationship and replace unstable nodes with backup nodes. Update the node health status identifier in the storage path optimization mapping table and recalculate the optimal storage node; The optimized resource scheduling rules and storage path mapping relationship are recombined to generate a new dynamic resource integration strategy version.

6. A potential information identification system combined with a heterogeneous network environment, characterized in that: It includes a processor and a memory, the memory is connected to the processor, the memory is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the memory to implement the potential information identification method combined with a heterogeneous network environment as described in any one of claims 1 to 5 above.

Images