Data resource migration risk prediction method, system, terminal device and storage medium

Abstract

The present invention relates to the field of computer system technology, and specifically discloses a data resource migration risk prediction method, system, terminal device and storage medium, the method comprising: constructing explicit dependency features and implicit dependency features of various types of resources respectively; obtaining comprehensive dependency features of various types of resources, and then calculating the association strength between various resources; constructing a dynamic semantic association graph based on the association strength between various resources, and updating the affected subgraphs in response to resource changes and operation log updates; obtaining neighborhood aggregation features of various resources based on the topological features of the dynamic semantic association graph and the risk features of the resources, combined with the association strength between various resources; and calculating the risk score of data resource migration using the neighborhood aggregation features after feature enhancement. The present invention deeply integrates the graph topology features with the resource characteristics themselves, transforming risk assessment from qualitative judgment to quantitative prediction, and realizing quantitative assessment of dependency strength and risk.

Description

Technical Field

[0001] The present invention relates to the technical field of computer systems, and in particular to a data resource migration risk prediction method, system, terminal device and storage medium. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] As the core carrier of enterprise data assets, the data center platform contains a vast and diverse array of data resources, including data source connections, subject domains / topics, dimension tables, models, ETL transformations, ETL jobs, task scheduling, data quality rules, and information resource catalogs. These diverse resources possess complex dependencies, both explicit structural dependencies and implicit business logic dependencies, making migration of data resources between development and testing environments and production environments a significant challenge.

[0004] In existing technologies, risk assessment of data resource migration mainly relies on manual inspection and rule-based tools. Manual inspection involves technicians manually combing through database table structures, ETL job dependencies, and task scheduling relationships to extract explicit dependencies and form a static dependency graph. Rule-based tools, on the other hand, parse SQL scripts, ETL configuration files, etc. to extract explicit dependencies such as foreign keys and task trigger conditions and generate a dependency graph.

[0005] However, the above methods lack the ability to perform semantic analysis and dynamic mining of runtime logs. They can only discover explicit dependencies such as database table foreign keys and task scheduling configurations, while implicit dependencies in business logic are often ignored. In addition, the static dependency graph formed cannot adapt to the frequent changes and modifications of data resources, making migration risk assessment disconnected from the actual environment. Moreover, the update of static dependency graphs mostly relies on static configuration or manual maintenance, which cannot capture resource changes in a timely manner, resulting in delayed updates of dependency association graphs and failure to reflect the latest status. Summary of the Invention

[0006] In order to solve the above problems, the present invention proposes a data resource migration risk prediction method, system, terminal device and storage medium. By constructing a dynamic semantic association graph, explicit and implicit dependencies are captured simultaneously; by introducing an incremental propagation algorithm to locally update the subgraph part affected by the change, dynamic maintenance of the dependency is achieved.

[0007] In some embodiments, the following technical solutions are adopted:

[0008] A data resource migration risk prediction method, comprising:

[0009] Construct explicit dependency features and implicit dependency features for each type of resource respectively;

[0010] The explicit dependency features and implicit dependency features are integrated to obtain comprehensive dependency features of various types of resources, and then the association strength between the resources is calculated;

[0011] Build a dynamic semantic association graph based on the association strength between resources, and update the affected subgraphs in response to resource changes and operation log updates;

[0012] Based on the topological characteristics of the dynamic semantic association graph and the risk characteristics of resources, combined with the association strength between resources, the neighborhood aggregation characteristics of each resource are obtained;

[0013] The risk score of data resource migration is calculated using the neighborhood aggregation features after feature enhancement.

[0014] As a further solution, the method of constructing implicit dependency features is:

[0015] Read the logs generated by the data center during operation within the set time, and match the operation records of various types of resources; based on the operation records of the resources, build an operation event sequence in chronological order, and build a co-occurrence matrix based on the operation time sequence; use the co-occurrence matrix as the weight and combine it with the resource name to generate implicit dependency features.

[0016] As a further solution, the correlation strength between resources is calculated as follows:

[0017] resource and resources The strength of the relationship between resources is equal to Comprehensive dependency characteristics and resources The comprehensive dependency features are used to calculate the cosine similarity, and then the results are mapped to .

[0018] As a further solution, after obtaining the dynamic semantic association graph, it also includes:

[0019] By integrating the resource dependencies, historical change records and load monitoring data of the ETL running server in the dynamic semantic association graph, four numerical features that can reflect the essence of the risk are extracted: impact scope, association strength, historical failure rate and load status.

[0020] As a further solution, the neighborhood aggregation characteristics of each resource are specifically as follows:

[0021] ;

[0022] in, is the graph edge weight matrix, composed of resources and resources The correlation strength is obtained; is the graph node feature matrix, including resources Historical failure rates and load conditions; Represents a resource The collection of all neighbor resources.

[0023] As a further solution, feature enhancement is performed on the neighborhood aggregation features, specifically:

[0024] Splice the resource's own features with the neighborhood aggregation features to obtain a spliced ​​feature matrix;

[0025] The transpose of the concatenated feature matrix is ​​multiplied by the trainable parameter matrix, and the obtained result is nonlinearly activated to obtain the neighborhood aggregation feature after feature enhancement.

[0026] As a further solution, after obtaining the risk score of data resource migration, it also includes:

[0027] Identify resources based on the risk score range The risk level of the resource is determined by using feature contribution analysis based on gradient back propagation to obtain the feature importance of the resource itself.

[0028] By extracting resources from the association graph The key propagation paths and the weights on the paths are used to obtain the impact of associated resources on the risk score.

[0029] In other embodiments, the following technical solutions are adopted:

[0030] A data resource migration risk prediction system, comprising:

[0031] A dependency feature building module is configured to respectively build explicit dependency features and implicit dependency features of various types of resources;

[0032] an association strength calculation module configured to fuse the explicit dependency feature vector and the implicit dependency feature vector to obtain comprehensive dependency features of various types of resources, and then calculate the association strength between the resources;

[0033] A dynamic semantic association graph construction module is configured to construct a dynamic semantic association graph based on the association strength between resources, and update the affected subgraph in response to resource changes and operation log updates;

[0034] A neighborhood aggregation feature calculation module is configured to obtain neighborhood aggregation features of each resource based on the topological features of the dynamic semantic association graph and the risk features of the resources, combined with the association strength between the resources;

[0035] The resource risk scoring module is configured to calculate the risk score of data resource migration by using the neighborhood aggregation features after feature enhancement.

[0036] In other embodiments, the following technical solutions are adopted:

[0037] A terminal device includes a processor and a memory, wherein the processor is used to implement instructions; the memory is used to store multiple instructions, and the instructions are suitable for being loaded by the processor and executing the above-mentioned data resource migration risk prediction method.

[0038] In other embodiments, the following technical solutions are adopted:

[0039] A computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor of a terminal device to perform the above-mentioned data resource migration risk prediction method.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] (1) This invention innovatively proposes a fusion modeling method for explicit and implicit dependencies. Through the collaborative work of metadata extractors and log analyzers, it realizes the capture of complete dependency relationships of various types of resources in the data center for the first time. Based on the neighborhood feature aggregation technology and combined with the impact risk prediction model, through the collaborative design of feature decoupling and feature enhancement, the graph topology features and the resource characteristics themselves are deeply integrated, so that risk assessment is transformed from qualitative judgment to quantitative prediction, and the quantitative evaluation of dependency strength and risk is realized.

[0042] (2) The present invention constructs a co-occurrence matrix based on the running time series, and extracts the implicit dependency features between different types of resources based on the co-occurrence matrix, which solves the problem that traditional methods cannot identify business logic dependencies, achieves complete coverage of the full amount of dependency relationships in the data center, and improves the scope and accuracy of dependency identification.

[0043] (3) The present invention uses comprehensive dependency features to calculate the association strength between resources, and aggregates neighborhood features based on the association strength, so that each resource node integrates information from its associated resources, thereby better reflecting the local information of the resource in the graph; at the same time, the neighborhood aggregation features are integrated with the resource's own features to ensure that the resource's own features are not completely overwhelmed by the neighborhood information, avoiding oversmoothing caused by neighborhood aggregation, which causes all node features to tend to the mean, and enhancing the generalization ability of the model in different resource scenarios.

[0044] (4) In order to adapt to the dynamic changes of business operation data, the present invention constructs a dynamic semantic association graph. When resources change and operation logs are updated, the subgraph affected by the change is locally updated, which solves the problem that the existing static graph cannot adapt to frequent changes, realizes the dynamic maintenance of dependency relationships, reduces the lag of the dependency association graph, and improves the accuracy of risk identification and prediction.

[0045] Other features and advantages of additional aspects of the present invention will be given in part in the following description and in part will become obvious from the following description or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flow chart of a data resource migration risk prediction method according to an embodiment of the present invention;

[0047] Figure 2 A schematic diagram of the process of constructing a dynamic semantic association graph in an embodiment of the present invention;

[0048] Figure 3 Schematic diagram of the process of constructing an impact risk prediction model in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] It should be noted that the following detailed description is illustrative and is intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0050] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0051] Example 1

[0052] In one or more embodiments, a data resource migration risk prediction method is disclosed, combining Figure 1 , specifically including the following process:

[0053] S101: Construct explicit dependency features and implicit dependency features of each type of resource respectively.

[0054] Specifically, data resources include data source connections, subject domains / topics, dimension tables, models, ETL transformations, ETL jobs, task scheduling, data quality rules, and information resource catalogs. There are explicit and implicit dependencies between different types of data resources. Explicit dependencies are reflected in direct foreign key associations of database tables, while implicit dependencies refer to indirect associations that are not explicitly declared for resources but are manifested through actual business operations. Their core characteristics are dynamism, unstructuredness, and strong business relevance. They change with business operation modes, cannot be directly obtained through static metadata, and can affect system stability. For example, in a data middle-office financial management scenario, when performing data verification operations, the original transaction data and summary reports are usually viewed simultaneously. Although there is no clear explicit dependency definition between these two tasks, their operational behaviors are highly correlated, indicating the existence of a potential business relationship.

[0055] Implicit dependencies are indirect, informal dependencies between resources. Capturing them through analysis and mining of actual business processes is crucial for the smooth operation of the business. However, existing tools primarily rely on structured metadata parsing and predefined rule engines. This static analysis-based approach is unable to capture implicit dependencies at the business level.

[0056] In this embodiment, an innovative fusion modeling method of explicit dependencies and implicit dependencies is proposed. Through the collaborative work of the metadata extractor and the log analyzer, the complete dependency relationship capture of various types of resources in the data center is achieved for the first time.

[0057] Specifically, the explicit dependency feature extraction process is as follows:

[0058] Build a metadata extractor to directly read the table structure definition, field information and foreign key relationship corresponding to each type of resource from the database table; for each specific resource, extract its name, creator, creation time, last modifier, and last modification time; in particular, for task scheduling, extract its scheduling execution logic and trigger conditions.

[0059] Convert the extracted explicit dependency features into explicit feature vectors , the mathematical expression is:

[0060] ;

[0061] in, For the The extraction results of explicit dependencies, For the The weight of an explicit dependency is derived based on the analysis of historical data. is the number of explicit dependent features.

[0062] The implicit dependency feature extraction process is as follows:

[0063] Build a log analyzer to read the logs generated by the data center during operation over a period of time. Use pre-made matching rules to identify the operation records related to various types of resources and standardize them. The standardized form is:

[0064] ;

[0065] in, is the runtime timestamp, The type of running resource. The primary key of the running resource in the database table to which it belongs. The name of the running resource.

[0066] After obtaining the standardized resource operation records, the operation event sequence is constructed according to the time sequence, and the co-occurrence matrix is ​​constructed based on the operation time sequence. , each element in the matrix The mathematical expression is as follows:

[0067] ;

[0068] in, Represents resources in a sequence of running events In unit time The operating frequency within Indicates the length of the time window for running the event sequence, Represents a resource and resources The number of times it appears in the same set of running records.

[0069] In the data center, business-related resources often have similar resource names, so resource names contain information that can reflect implicit dependencies at the business level. For the above reasons, the co-occurrence matrix is ​​used as the weight, combined with the resource name in the resource metadata to generate the implicit feature vector , the mathematical expression is as follows:

[0070] ;

[0071] in, Represents resource-based Generated resources The implicit eigenvector of Represents a resource and resources The number of times it appears in the same set of running records, Encode character-level word embeddings for resource names.

[0072] S102: Explicit dependency features and implicit dependency features are integrated to obtain comprehensive dependency features of various types of resources, and then the association strength between the resources is calculated.

[0073] In this embodiment, after obtaining the explicit feature vector and the implicit feature vector, the explicit dependency feature vector and the implicit dependency feature vector of each resource are obtained. Weighted fusion to obtain comprehensive resource dependency characteristics , the mathematical expression is as follows:

[0074] ;

[0075] in, represents explicit feature weights, represents the implicit feature weight, and is a hyperparameter, the default is , can be customized by users according to business characteristics. For example, when emphasizing data lineage, the explicit feature weight should be higher. If it is an unstructured interaction under the microservice architecture, the implicit feature weight should be higher. Represents the feature Find the L2 norm to scalar the importance of the feature.

[0076] After obtaining the dependency characteristics of each resource, the association strength between the resources is calculated based on the dependency characteristics. , the mathematical expression is as follows:

[0077] ;

[0078] in, For resources The comprehensive dependence characteristics of For resources The comprehensive dependence characteristics of As the activation function, the calculated cosine similarity between features is mapped to It can not only reflect the implicit dependency between resources, but also reflect the strength of the association between resources, providing a numerical basis for quantifying and evaluating the specific risk conditions of resources.

[0079] S103: Construct a dynamic semantic association graph based on the association strength between resources, and update the affected subgraph in response to resource changes and operation log updates.

[0080] Existing technologies often rely on static configuration or manual maintenance, which is unable to capture frequent changes and modifications to resources in a timely manner, resulting in delayed updates of dependency graphs and a failure to reflect the latest status. This embodiment constructs a dynamic semantic association graph to identify and record explicit and implicit dependencies between resources, ensuring the logical and business integrity of resource migration. This solves the problem of static graphs being unable to adapt to frequent changes, enables dynamic maintenance of dependency relationships, reduces the lag of dependency graphs, and improves the accuracy of risk identification and prediction.

[0081] Specifically, the construction process of the dynamic semantic association graph is as follows: Figure 2 As shown; First, the dynamic semantic association graph is initialized based on all resources and their association strengths , is the set of all nodes in the graph, that is, all resources, The set of all edges, i.e., the relationships between resources. After building the initial association graph, we regularly check new resources and operation logs, and update the association graph with new resources and association strengths.

[0082] To ensure efficient graph maintenance, this embodiment uses an incremental propagation method to update only the affected subgraphs, avoiding global reconstruction. The mathematical expression of the update process is as follows:

[0083] ;

[0084] in, Indicates the current graph. represents the change caused by updating resources and association strength, Represents the updated association graph. The mathematical expression is:

[0085] ;

[0086] in, Indicates newly added resources. Represents any two resources in the graph, Indicates The corresponding parameter combination when the maximum result is achieved , Indicates the resource node pair that will eventually be updated in the graph. Indicates the update threshold. New resources are added to the graph only when the change in association strength exceeds this threshold.

[0087] S104: Based on the topological features of the dynamic semantic association graph and the risk features of the resources, combined with the association strength between the resources, the neighborhood aggregation features of each resource are obtained.

[0088] While automatically capturing dependencies accurately, change risk assessment remains an essential step. Only through risk prediction and assessment can we further quantify the importance of these dependencies in real-world business scenarios and determine the scope and severity of the impact of a resource change. Furthermore, further risk prediction and assessment can identify highly sensitive resources, critical paths, and potential vulnerabilities, providing decision support and reference for operations and maintenance personnel, thereby preventing and avoiding potential failures.

[0089] Based on the above objectives, this embodiment builds an impact risk prediction model by integrating resource dependencies, historical change records and load monitoring data of ETL running servers in the dynamic semantic association graph. The specific process is as follows: Figure 3 As shown in the figure, the risk level that may be caused by the change is quantified by the impact risk prediction model, ensuring system stability and change controllability.

[0090] Specific, combined Figure 3 By analyzing various factors that may lead to risks, this embodiment constructs feature engineering based on four dimensions: impact scope, dependency strength, historical stability, and system status. A total of four numerical features that can reflect the essence of risks are extracted: impact scope, correlation strength, historical failure rate, and load status. The description of each feature is shown in Table 1.

[0091] Table 1 Risk assessment model characteristics and description

[0092]

[0093] Among them, the impact scope and association strength are obtained by retrieving the dynamic semantic association graph, the historical failure rate is obtained by analyzing historical change records, and the load status is specifically for the two types of data resources, ETL transformation and ETL operation, which can be directly obtained by reading the JVM memory monitoring interface of the ETL server in the development and testing environment.

[0094] Decouple the topological features (i.e., graph structure) and risk features (i.e., historical failure rate and load status) of the associated graph, and define the graph node feature matrix and the graph edge weight matrix , the mathematical expression is as follows:

[0095] ;

[0096] in, Indicates the The historical failure rate of each resource, Indicates the The load status of each resource, Indicates the total number of resources.

[0097] ;

[0098] in, Represents resources in the association graph and resources The strength of association.

[0099] This embodiment constructs a graph node feature matrix H based on historical failure rates and load status. The historical failure rates directly reflect the reliability of the nodes, while the load status reflects the real-time pressure. Encoding the historical failure rates and load status as node features can directly reflect the impact of the resources themselves, enabling the model to quickly identify high-risk individuals and facilitating operation and maintenance personnel to intuitively interpret the source of risk. This embodiment constructs a graph edge weight matrix W based on association strength, which can accurately quantify the degree of association between explicit and implicit dependency features between nodes, and stably characterize the association topology between resources, so as to accurately extract key propagation paths and precisely locate the scope of impact.

[0100] In order to enable each resource node to integrate information from its associated resources so that it can better reflect the local information in the graph and optimize the prediction results, neighborhood feature aggregation is performed to obtain neighborhood aggregation features. The mathematical expression is as follows:

[0101] ;

[0102] in, Represents a resource Neighborhood aggregation features, Represents the graph edge weight matrix Elements of Represents a resource The feature vector of contains the load status and historical failure rate, Represents a resource The total number of neighbor resources is the influence range.

[0103] This embodiment combines the dependency relationship with risk factors such as historical failure rates and system load status, solving the problem of insufficient accuracy and refinement of risk assessment due to reliance solely on empirical judgment and lack of quantitative modeling.

[0104] S105: Calculate the risk score of data resource migration using the neighborhood aggregation features after feature enhancement.

[0105] To ensure that the resource's own characteristics are not completely overwhelmed by the neighborhood information, to avoid oversmoothing caused by neighborhood aggregation, which causes all node characteristics to tend to the mean, and to enhance the model's generalization ability in different resource scenarios, it is necessary to enhance the characteristics of the resource itself. The mathematical expression is as follows:

[0106] ;

[0107] in, Represents a resource Enhanced aggregation features, Indicates the combination of resource characteristics and aggregated characteristics. Represents a trainable parameter matrix, which is used to ensure that the resource's own characteristics are not overwhelmed by the neighborhood information, and can enhance the model's generalization ability in different scenarios by training weights. Represents nonlinear activation, which is used to ensure the non-negativity of the output features.

[0108] The enhanced aggregation feature not only retains the characteristics of the resource itself, but also integrates the characteristics from neighboring resources. By activating it, the risk score of the resource change can be obtained. , the mathematical expression is as follows:

[0109] ;

[0110] in, Represents a resource Enhanced aggregation features, is the activation function, mapping the risk score to .

[0111] Compared with the limitations of existing technologies that can only provide binary judgments of "risk or not" and rely on the subjectivity of experience-based judgments, this embodiment calculates risk scores through enhanced neighborhood aggregation features, realizing the transition from simple "dependence or not" to quantitative risk assessment in the continuous interval of 0-1, thereby improving the granularity of risk detection and assessment and improving accuracy.

[0112] S106: Classify risk levels and generate a risk assessment report.

[0113] Specifically, this embodiment classifies risk scores. If , then it is considered that resources The changes can continue to be executed if , then it is considered that resources Changes to the , then it is considered that resources The above-mentioned range division is only an example, and those skilled in the art can divide the risk level interval according to actual needs.

[0114] Then, the feature contribution analysis based on gradient back propagation is used to obtain the feature importance of the resource itself (i.e., resource feature importance). The mathematical expression is as follows:

[0115] ;

[0116] in, Indicates the The feature importance of each feature, Indicates the risk score for The partial derivative of a feature reflects the impact of the change of the feature on the score. Indicates the The eigenvalue of a feature.

[0117] Resource feature importance reflects the importance of a resource due to its own attributes. For example, if a resource has a high historical failure rate or has a significant impact on load conditions, then changes to it may bring greater risks to the overall resource migration.

[0118] Finally, by extracting the resources in the association graph The key propagation paths and weights on the paths are calculated, and the sum of the weights on all paths is calculated to obtain the impact of the associated resources on the risk score. The key communication path refers to the The path with the largest weight among all associated resource paths; the weight is the strength of the association between resources.

[0119] The impact of associated resources can reflect the importance of a resource due to its association with other resources. For example, in extreme cases, if a resource is directly associated with all other resources, then changes to this resource are considered to be risky. If a resource has a high degree of association with several other resources, then the risk level of this resource is relatively high.

[0120] The resulting risk scores and ratings, resource feature importance, and associated resource impact are summarized and presented in a risk report. The resource feature importance and associated resource impact clearly demonstrate the risky resources and risk profile within the migration resource package, providing a reference for targeted handling and prevention.

[0121] After the resource package that has passed the risk prediction is imported into the production environment, the import record is written into the training sample library for feedback on the training impact risk prediction model, forming an "import->monitor->feedback->optimization" process. The specific steps are as follows:

[0122] (1) Obtaining resource changes: The production environment reads the resource package but before writing data, compares the differences between the imported data resources and the corresponding existing resources in the production environment, which is the resource change situation, and records the resource change situation in the historical change record.

[0123] (2) Monitor system status: Continuously monitor system performance within 48 hours after import, observing and recording any operational failures such as errors and failures. Record the operational status in the historical change log 48 hours after import. If a failure occurs, record the relevant resources and the type of failure.

[0124] (3) Analyze prediction errors: Compare the prediction results of the risk prediction model with the actual performance results of the production environment, perform attribution analysis on the parts with larger errors, and adjust the corresponding edge weights in the association graph or the weights during model training.

[0125] (4) Update the association map: If a new implicit dependency between resources is found during fault analysis, a new association is added to the association map for the corresponding resource.

[0126] (5) Iterative prediction model: After a certain time interval (such as 7 days) or a certain number of unprocessed historical change records, the impact risk prediction model is retrained, and the model parameters are iterated and optimized to improve the model's prediction accuracy.

[0127] This embodiment designs a feedback training mechanism to reversely optimize the associated graph and model parameters through the actual performance of the production environment, thereby solving the problem of lagging graph and model iterations and achieving continuous improvement in predictive capabilities.

[0128] After training the impact risk prediction model, input the required data into the impact risk prediction model to perform risk prediction. The specific steps are as follows:

[0129] (1) The dynamic semantic association graph structured data constructed by the latest resources of the development and test environment data resources, the historical change records of the target environment, and the load monitoring data of the test and development environment are input into the impact risk prediction model to obtain the risk score of the current resource migration and the importance of the associated resources.

[0130] (2) Within the model, the risk score and the historical failure rate data obtained from the historical change records of the target environment and the load status data obtained from the load monitoring data of the test and development environment are analyzed to obtain the feature importance of each resource node in the graph.

[0131] (3) Summarize the risk scores, resource feature importance, and associated resource importance to generate a migration risk report for the current development and testing environment data resources.

[0132] This embodiment integrates the topological features of the association graph and the characteristics of the resources themselves through a deep learning model. Through the collaborative design of feature decoupling and feature enhancement, the graph topological features and the characteristics of the resources themselves are deeply integrated, so that risk assessment is transformed from qualitative judgment to quantitative prediction, realizing the quantitative assessment of dependency intensity and risk.

[0133] Example 2

[0134] In one or more embodiments, a data resource migration risk prediction system is disclosed, including:

[0135] A dependency feature building module is configured to respectively build explicit dependency features and implicit dependency features of various types of resources;

[0136] an association strength calculation module configured to fuse the explicit dependency feature vector and the implicit dependency feature vector to obtain comprehensive dependency features of various types of resources, and then calculate the association strength between the resources;

[0137] A dynamic semantic association graph construction module is configured to construct a dynamic semantic association graph based on the association strength between resources, and update the affected subgraph in response to resource changes and operation log updates;

[0138] A neighborhood aggregation feature calculation module is configured to obtain neighborhood aggregation features of each resource based on the topological features of the dynamic semantic association graph and the risk features of the resources, combined with the association strength between the resources;

[0139] The resource risk scoring module is configured to calculate the risk score of data resource migration by using the neighborhood aggregation features after feature enhancement.

[0140] The risk classification module is configured to determine the resource based on the interval range to which the risk score belongs. The risk level of the resource itself is analyzed using the feature contribution analysis based on gradient back propagation to obtain the feature importance of the resource itself; by extracting the resource in the associated graph The key propagation paths and the weights on the paths are used to obtain the impact of associated resources on the risk score.

[0141] It should be noted that the specific implementation of the above modules is exactly the same as that in Example 1 and will not be described in detail.

[0142] Example 3

[0143] In one or more embodiments, a terminal device is disclosed, which includes a processor and a memory, the processor being used to implement instructions; the memory being used to store multiple instructions, the instructions being suitable for being loaded by the processor and executing the data resource migration risk prediction method described in Example 1.

[0144] It should be understood that in this embodiment, the processor may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), off-the-shelf field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0145] The memory may include a read-only memory and a random access memory, and provides instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0146] During implementation, each step of the above method may be completed by an integrated logic circuit of hardware in a processor or by instructions in the form of software.

[0147] Example 4

[0148] In one or more embodiments, a computer-readable storage medium is disclosed, in which a plurality of instructions are stored. The instructions are suitable for being loaded by a processor of a terminal device and executing the data resource migration risk prediction method described in Example 1.

[0149] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. A data resource migration risk prediction method, characterized in that: include: Construct explicit dependency features and implicit dependency features for each type of resource respectively; Build a metadata extractor to read the table structure definition, field information, and foreign key relationships corresponding to each type of resource from the database table; for each specific resource, extract the name, creator, creation time, last modifier, and last modification time; for task scheduling, extract the scheduling execution logic and trigger conditions to generate explicit dependency features; Read the logs generated by the data center during operation within the set time, and match the operation records of various types of resources; based on the operation records of the resources, build the operation event sequence in chronological order, and build the co-occurrence matrix based on the operation time sequence; Using the co-occurrence matrix as a weight and combining it with the resource name, an implicit dependency feature is generated; The explicit dependency features and implicit dependency features are integrated to obtain comprehensive dependency features of various types of resources, and then the association strength between the resources is calculated; Build a dynamic semantic association graph based on the association strength between resources, and update the affected subgraphs in response to resource changes and operation log updates; Based on the topological characteristics of the dynamic semantic association graph and the risk characteristics of resources, combined with the association strength between resources, the neighborhood aggregation characteristics of each resource are obtained; The specific neighborhood aggregation characteristics of each resource are: ； in, is the graph edge weight matrix, composed of resources and resources The correlation strength is obtained; is the graph node feature matrix, including resources Historical failure rates and load conditions; Represents a resource The set of all neighbor resources; The risk score of data resource migration is calculated using the neighborhood aggregation features after feature enhancement.

2. A data resource migration risk prediction method according to claim 1, characterized in that: Calculate the association strength between resources, specifically: resource and resources The strength of the relationship between resources is equal to Comprehensive dependency characteristics and resources The comprehensive dependency features are used to calculate the cosine similarity, and then the results are mapped to .

3. A data resource migration risk prediction method according to claim 1, characterized in that: After obtaining the dynamic semantic association graph, it also includes: By integrating the resource dependencies, historical change records and load monitoring data of the ETL running server in the dynamic semantic association graph, four numerical features that can reflect the essence of the risk are extracted: impact scope, association strength, historical failure rate and load status.

4. A data resource migration risk prediction method according to claim 1, characterized in that: Feature enhancement is performed on the neighborhood aggregation features, specifically: Splice the resource's own features with the neighborhood aggregation features to obtain a spliced ​​feature matrix; The transpose of the concatenated feature matrix is ​​multiplied by the trainable parameter matrix, and the obtained result is nonlinearly activated to obtain the neighborhood aggregation feature after feature enhancement.

5. A data resource migration risk prediction method according to claim 1, characterized in that: After obtaining the risk score for data resource migration, the following is also included: Identify resources based on the risk score range The risk level of the resource is determined by using feature contribution analysis based on gradient back propagation to obtain the feature importance of the resource itself. By extracting resources from the association graph The key propagation paths and the weights on the paths are used to obtain the impact of associated resources on the risk score.

6. A data resource migration risk prediction system, executing the data resource migration risk prediction method according to any one of claims 1 to 5, characterized in that: include: A dependency feature building module is configured to respectively build explicit dependency features and implicit dependency features of various types of resources; an association strength calculation module configured to fuse the explicit dependency feature vector and the implicit dependency feature vector to obtain comprehensive dependency features of various types of resources, and then calculate the association strength between the resources; A dynamic semantic association graph construction module is configured to construct a dynamic semantic association graph based on the association strength between resources, and update the affected subgraph in response to resource changes and operation log updates; A neighborhood aggregation feature calculation module is configured to obtain neighborhood aggregation features of each resource based on the topological features of the dynamic semantic association graph and the risk features of the resources, combined with the association strength between the resources; The resource risk scoring module is configured to calculate the risk score of data resource migration by using the neighborhood aggregation features after feature enhancement.

7. A terminal device comprising a processor and a memory, wherein the processor is used to implement instructions; the memory is used to store multiple instructions, characterized in that: The instructions are suitable for being loaded by a processor and executing the data resource migration risk prediction method described in any one of claims 1-5.

8. A computer-readable storage medium storing a plurality of instructions, characterized in that: The instructions are suitable for being loaded by a processor of a terminal device and executing the data resource migration risk prediction method according to any one of claims 1-5.

Images