A data warehouse change risk assessment method and device and a storage medium

Abstract

The application provides a data warehouse change risk assessment method and device and a storage medium, relates to the technical field of data processing, and comprises the following steps: obtaining change information of a data warehouse, determining first processing information of the data warehouse before change and second processing information of the data warehouse after change according to the change information; comparing data processing apertures before and after change of a target field according to the first processing information and the second processing information to obtain a processing aperture comparison result; performing influence analysis on a downstream associated object corresponding to the target field according to an aperture change relationship in the processing aperture comparison result and a field blood relationship and a downstream task dependency relationship corresponding to the target field to obtain an influence object analysis result; and performing risk assessment on the change information according to the processing aperture comparison result and the influence object analysis result to obtain a change risk assessment result of the data warehouse. In this way, the affected range and the risk can be accurately determined before the data warehouse change is released.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus and storage medium for risk assessment of data warehouse changes. Background Technology

[0002] During the change release process of an enterprise data warehouse, a change in a single field or data processing logic can affect multiple downstream data objects and business metrics along data lineage and task dependencies. Therefore, accurately determining the impact of a change on downstream tasks and business operations before release is extremely important. In existing data warehouse data service processes, the impact of a change on downstream operations is typically determined based on table-level or field-level lineage relationships or task runtime status.

[0003] However, when the fields and table structures remain unchanged, but the SQL processing logic corresponding to the fields and tables changes, the data tasks can still run normally and the lineage relationship has not changed significantly. However, the data processing scope of the fields has changed, making it difficult for traditional data lineage and task operation monitoring methods to identify the affected downstream business objects. As a result, it is difficult to determine the scope of impact and the degree of risk before the change is released.

[0004] Therefore, improving the accuracy of the scope and degree of impact of changes before data warehouse changes are released is an urgent issue to be addressed. Summary of the Invention

[0005] This application provides a risk assessment method, apparatus, and storage medium for data warehouse changes, aiming to solve the problem that existing methods for determining the impact of changes on downstream entities based on table, field-level lineage, or task running status are difficult to identify affected downstream business objects, thus making it difficult to determine the scope and degree of impact before the change is released.

[0006] To address the problems of this application, the following technical solution is provided: Firstly, this application provides a risk assessment method for data warehouse changes, applied to a server in an intelligent assessment system for data warehouses, the method comprising: Obtain change information from the data warehouse; Based on the change information, the data warehouse is determined to have first processing information before the change and second processing information after the change; the first processing information and the second processing information respectively include at least one of the following: the field expression corresponding to the target field, the source field, the filtering condition, and the aggregation granularity; Based on the first processing information and the second processing information, the data processing caliber of the target field before and after the change is compared to obtain the processing caliber comparison result. Based on the diameter change relationship in the comparison results of the processing diameter, as well as the field lineage relationship and downstream task dependency relationship corresponding to the target field, an impact analysis is performed on the downstream related objects corresponding to the target field to obtain the impact object analysis results; Based on the comparison results of the processing caliber and the analysis results of the affected objects, a risk assessment is performed on the change information to obtain the change risk assessment results of the data warehouse.

[0007] Secondly, this application provides a risk assessment device for data warehouse changes, applied to a server in an intelligent assessment system for data warehouses, the device comprising: The data acquisition module is used to acquire change information from the data warehouse. The change determination module is used to determine, based on the change information, the first processing information of the data warehouse before the change and the second processing information after the change; the first processing information and the second processing information respectively include at least one of the following: the field expression corresponding to the target field, the source field, the filtering condition, and the aggregation granularity; The impact analysis module is used to compare the data processing scope of the target field before and after the change based on the first processing information and the second processing information to obtain the processing scope comparison result; and to perform impact analysis on the downstream related objects corresponding to the target field based on the scope change relationship in the processing scope comparison result, as well as the field lineage relationship and downstream task dependency relationship corresponding to the target field to obtain the impact object analysis result. The risk assessment module is used to assess the risk of the change information based on the processing caliber comparison results and the impact object analysis results, and obtain the risk assessment results of the data warehouse change.

[0008] Thirdly, this application provides a server including a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the programs include instructions for performing steps in any of the methods of the first aspect of the embodiments of this application.

[0009] Fourthly, this application provides a computer-readable storage medium storing a computer program for electronic data interchange, wherein the computer program causes a computer to perform some or all of the steps described in any of the methods of the first aspect of the embodiments of this application.

[0010] Compared to most current methods that determine the impact of data warehouse changes through table and field-level lineage relationships and task execution status, this approach struggles to identify changes in field data processing standards and their downstream impacts when the data processing logic changes while the field or table structure remains unchanged. This application proposes a data warehouse change risk assessment method, apparatus, and storage medium. First, it acquires the data warehouse change information. Second, it determines the first processing information and the second processing information of the data warehouse before and after the change based on the change information. Then, it compares the data processing standards of the target field before and after the change based on the first and second processing information to obtain the processing standard comparison results. Further, based on the standard change relationships in the processing standard comparison results, as well as the field lineage relationships and downstream task dependencies of the target field, it performs an impact analysis on the downstream related objects corresponding to the target field to obtain the impact object analysis results. Finally, it conducts a risk assessment of the change information based on the processing standard comparison results and the impact object analysis results to obtain the data warehouse change risk assessment results. Since the first and second processing information respectively represent at least one of the following: field expression, source field, filtering conditions, and aggregation granularity of the target field before and after the change, by comparing the data processing scope of the first and second processing information before and after the change, it is possible to identify changes in the data processing scope of the target field when the field or table structure has not changed. Furthermore, based on the relationship of the scope change, combined with the field lineage of the target field and the downstream task dependency relationship, the impact analysis of downstream related objects is carried out, so that the change in data processing scope can determine its downstream affected objects along the data association relationship and task dependency relationship corresponding to the target field. Then, based on the processing scope comparison results and the results of the affected object analysis, a risk assessment is carried out, so that the risk assessment simultaneously links the change in data processing scope and the scope of downstream impact. This is beneficial for identifying scope changes and their downstream impacts that are difficult to detect by traditional table, field-level lineage relationships or task running status before the data warehouse change is released, thereby improving the accuracy of data warehouse change risk assessment. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is an architecture diagram of an intelligent evaluation system for a data warehouse provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a server provided in an embodiment of this application; Figure 3This is a flowchart illustrating a risk assessment method for data warehouse changes provided in an embodiment of this application; Figure 4 This is a schematic diagram of an AI semantic-assisted analysis process provided in an embodiment of this application; Figure 5 This is a schematic diagram illustrating the downstream impact of data warehouse changes, provided in an embodiment of this application. Figure 6 This is a flowchart illustrating an access control process for publishing information changes, provided in an embodiment of this application. Figure 7 This is a diagram illustrating the results of a data warehouse change risk assessment, provided in an embodiment of this application. Figure 8 This application provides a flowchart for data warehouse change risk assessment and access control deployment. Figure 9 This is a block diagram of the functional modules of a risk assessment device for data warehouse changes provided in an embodiment of this application. Detailed Implementation

[0013] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0014] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0015] It should be understood that the character " / " in this document indicates that the objects before and after it are in an "or" relationship. In the embodiments of this application, "multiple" refers to two or more.

[0016] In the embodiments of this application, "at least one item" or its similar expression refers to any combination of these items, including any combination of a single item or a plurality of items. "One or more" means one or more, while "multiple" means two or more. For example, "at least one item" of a, b, or c can represent the following seven cases: a, b, c; a and b; a and c; b and c; a, b, and c. Each of a, b, and c can be an element or a set containing one or more elements.

[0017] In this application, the term "connection" refers to various connection methods, such as direct connection or indirect connection, to achieve communication between devices. This application does not impose any limitations on this.

[0018] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0019] The following is combined with Figure 1 The architecture of a data warehouse change risk assessment method according to an embodiment of this application is described. Figure 1 This is an architecture diagram of an intelligent evaluation system for a data warehouse provided in an embodiment of this application. The intelligent evaluation system 100 for the data warehouse includes a change acquisition and preprocessing layer 110, a change analysis layer 120, a release control layer 130, and an early warning analysis layer 140.

[0020] The change acquisition and preprocessing layer 110 is used to acquire data warehouse change information to be published, and to parse and complete the metadata of the data warehouse change information to provide structured change data for subsequent change analysis. In some embodiments, the change acquisition and preprocessing layer 110 includes a change acquisition module 111, an SQL parsing module 112, and a metadata completion module 113. The change acquisition module 111 is used to receive change information in the data warehouse, which may include Git commits, changes to Structured Query Language (SQL) files, changes to table creation statements, changes to scheduling processes, changes to data quality rules, and changes to table structures. The SQL parsing module 112 is used to parse the SQL statements before and after the change, and extract information such as the source table, target table, field expressions, filtering conditions, association conditions, aggregation granularity, partition fields, and write methods. The metadata completion module 113 is used to acquire table hierarchy, field comments, business terms, responsible persons, task names, scheduling cycles, data quality rules, and historical output information from the metadata platform, and associate the acquired metadata with the parsed change information.

[0021] The change analysis layer 120 is used to analyze the SQL semantics and business definitions before and after the change based on the preprocessed change information, and to determine the downstream impact scope and risk level corresponding to the data warehouse change. In some embodiments, the change analysis layer 120 includes a semantic definition comparison module 121, a downstream impact diffusion module 122, a risk scoring module 123, and an AI semantic assistance module 124. The semantic definition comparison module 121 is used to compare the processing expressions, source fields, filtering conditions, aggregation methods, dimension granularity, and field annotations before and after the change to identify field additions, field deletions, changes in filtering conditions, changes in association conditions, changes in aggregation granularity, and changes in indicator definitions. The downstream impact diffusion module 122 is used to diffuse the impact layer by layer downstream based on field lineage, task dependencies, BI query logs, interface access logs, and data quality rule reference relationships to determine the affected downstream table fields, indicators, reports, interfaces, scheduling tasks, and quality rules. The risk scoring module 123 is used to determine the risk level based on factors such as the intensity of definition changes, downstream consumption importance, scheduling timeliness requirements, data sensitivity, historical issues, and compatibility measures. The AI ​​semantic assistance module 124 combines requirement descriptions, field annotations, SQL differences, lineage fragments, downstream object names, and historical cases to assist in the analysis of the meaning of changes and potential business impacts, and outputs candidate types of changes in scope, descriptions of business impacts, and suggested verification points. It is evident that the change analysis layer 120 can further identify changes in business scope caused by changes in SQL filtering conditions, correlation conditions, or aggregation granularity, even when field names have not changed, and determine the scope and degree of impact based on actual downstream consumption.

[0022] The release control layer 130 is used to execute corresponding release controls based on the risk scoring results and output processing results corresponding to the risk level. In some embodiments, the release control layer 130 includes a release access control module 131, an early warning output module 132, and a feedback learning module 133. The release access control module 131 is used to determine the release strategy based on the risk level, including allowing release, reminding release, requiring manual confirmation, or blocking release. The early warning output module 132 is used to generate early warning information such as change summary, affected objects and responsible persons, risk reasons, suggested verification SQL, suggested notification content, and suggested action. The feedback learning module 133 is used to record the results of manual confirmation, false alarms, repair results, and online incident results, and feeds the relevant results back to the subsequent risk assessment and historical case analysis process to optimize subsequent risk identification rules. The early warning analysis layer 140 is used to perform auxiliary analysis on historical change cases and current early warning results, and generate auxiliary content that is easy for technical and business personnel to understand and handle. The early warning analysis layer 140 includes a historical case retrieval module 141 and an AI copywriting and SQL auxiliary generation module 142. The historical case retrieval module 141 is used to retrieve historical changes, false alarms, online incidents, remediation plans, and review conclusions, and provides similar case references for the risk analysis of current changes. The AI ​​copywriting and SQL-assisted generation module 142 is used to generate business-readable impact statements, draft notices to responsible persons, draft verification SQL, and draft handling steps based on the current warning results, and provides evidence support by combining the corresponding SQL fragments, field lineage paths, downstream objects, and risk causes.

[0023] As can be seen, through the architecture of the intelligent evaluation system 100 for data warehouses provided in the above embodiments, changes can be collected and preprocessed, semantic and business interpretation of changes can be analyzed, downstream impact can be disseminated, risk scoring can be performed, release access control can be established, and early warning assistance can be generated in sequence before the data warehouse changes are officially released. This transforms the data warehouse change process from "discovering problems after release" to "identifying risks before release", improving the completeness of downstream impact identification and the accuracy of risk warning.

[0024] The following is combined with Figure 2 The server in the embodiments of this application will be described. Figure 2 This is a schematic diagram of the structure of a server provided in an embodiment of this application, such as... Figure 2 As shown, the server 200 includes one or more processors 210, a memory 220, a communication interface 230, and one or more programs 221. The processor 210 is communicatively connected to the memory 220 and the communication interface 230 via an internal communication bus.

[0025] The one or more programs 221 are stored in the memory 220 and configured to be executed by the processor 210. The one or more programs 221 include instructions for performing any step in the above method embodiments.

[0026] The processor 210 can be a central processing unit (CPU), a general-purpose processor, or a digital signal processor (DSP).

[0027] The memory 220 can be volatile memory or non-volatile memory, or it can include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory.

[0028] It is understood that server 200 may include more or fewer structural elements than those shown in the block diagram above, such as a power module, physical buttons, a Wi-Fi module, a speaker, a Bluetooth module, sensors, a display module, etc., without limitation. It is understood that server 200 may be equipped with... Figure 1 The architecture of the intelligent evaluation system for the data warehouse.

[0029] After understanding the software and hardware architecture of this application, the following will be combined with... Figure 3 This application describes a risk assessment method for data warehouse changes as described in its embodiments. Figure 3 This is a flowchart illustrating a risk assessment method for data warehouse changes provided in an embodiment of this application. The method is applied to a server in an intelligent assessment system for a data warehouse and specifically includes the following steps: Step S301: Obtain change information for the data warehouse.

[0030] The change information represents the data changes made to the data warehouse before its release. This may include changes to the corresponding SQL files, table creation statements, table structure, scheduling processes, and data quality rules, without any limitations.

[0031] Specifically, the system receives a pending change request for the data warehouse and obtains the corresponding change file or configuration. Based on the change file or configuration, it determines the content of the data warehouse changes to be released and identifies the target data warehouse object corresponding to the change content. Then, it parses the change content to determine the corresponding change object and change item. The change object may include data tables, fields, SQL processing logic, scheduling tasks, and data quality rules. Finally, it associates the determined change object, change item, and corresponding change content to generate change information for the data warehouse, and uses this change information as input for subsequent processing caliber comparison and downstream impact analysis.

[0032] It should be noted that in this embodiment, the data warehouse change information is obtained before the data warehouse is actually published. That is, after receiving the data warehouse changes submitted by the developers, the server analyzes the changes first, rather than performing impact detection after the changes are published and data is generated. The change information can correspond to one or more changed objects in the data warehouse, so that the processing information before and after the change can be determined, and the impact of the change on downstream related objects can be further analyzed.

[0033] Step S302: Determine the first processing information of the data warehouse before the change and the second processing information after the change based on the change information; the first processing information and the second processing information respectively include at least one of the following: the field expression corresponding to the target field, the source field, the filtering condition, and the aggregation granularity.

[0034] The first processing information represents the data processing logic of the target field before the implementation of this change, and the second processing information represents the data processing logic of the target field after the implementation of this change. The target field is the field to be analyzed corresponding to the change information. The field expression represents the data calculation or processing method of the target field, the source field represents the data field referenced when generating the target field, the filtering condition represents the rules or conditions used to filter the input data when generating the target field, and the aggregation granularity represents the data dimension used when performing statistics or aggregation on the input data.

[0035] Specifically, first, the server determines the target field corresponding to this change based on the change information and obtains the corresponding SQL or configuration file for the target field in the pre-change and post-change versions. Next, the pre-change and post-change versions are parsed respectively to extract the field expression corresponding to the target field and the source field referenced by the field expression. Further, the filter conditions and aggregation granularity corresponding to the target field are extracted from the pre-change and post-change versions respectively, and the extraction results are associated with the target field respectively. Finally, at least one of the field expression, source field, filter condition, and aggregation granularity extracted from the pre-change version is determined as the first processing information, and the corresponding information extracted from the post-change version is determined as the second processing information.

[0036] To facilitate understanding, let's illustrate with an example: In the monthly summary table of customers scanning QR codes, the target field can be the transaction success amount field `succ_amt`. The server retrieves the SQL processing logic before and after the change of this field, and extracts the transaction amount source field, transaction status filter conditions, and aggregation granularity corresponding to `succ_amt`. Before the change, the filter condition could be `operation_type=5`, used to filter transaction data with successful transactions. When the developers modify the transaction success recognition logic, the server further retrieves the changed filter conditions and forms the corresponding processing information before and after the change into first processing information and second processing information, respectively, for subsequent data processing comparison. The server can read the SQL or configuration file from the previous and new versions and parse it to obtain information such as the source table, target table, field expressions, filter conditions, Join conditions, and Group By granularity.

[0037] It can be seen that by obtaining the processing information of the target field before and after the change, the processing logic of the data warehouse before and after the change can be converted into comparable structured information. This enables the identification of changes in the processing logic of the target field from dimensions such as field expression, source field, filtering conditions, and aggregation granularity, providing a basis for subsequently determining the relationship of changes in data processing standards and analyzing the impact of these changes on downstream objects.

[0038] Step S303: Based on the first processing information and the second processing information, compare the data processing caliber of the target field before and after the change to obtain the processing caliber comparison result.

[0039] Among them, the data processing caliber represents the data calculation rules and business statistics rules corresponding to the target field, and the processing caliber comparison result represents whether the data processing logic of the target field has changed before and after the change, and the specific content of the change. The processing caliber comparison result may include at least one of the following: changes in field source, changes in filtering conditions, changes in aggregation method, changes in aggregation granularity, and changes in field expression.

[0040] Specifically, firstly, the field expressions, source fields, filtering conditions, and aggregation granularities in the first and second processing information are matched to determine the correspondence between the processing information before and after the change. Next, the processing information with corresponding relationships is compared to determine whether the source field, filtering conditions, aggregation method, or aggregation granularity of the target field has changed before and after the change. Then, the processing logic of the target field is classified according to the determined differences to determine the changes in the data processing scope corresponding to the target field. Finally, the changes in the data processing scope are correlated to generate a comparison result of the processing scope corresponding to the target field, representing the change in the data processing scope of the target field caused by this change.

[0041] In an optional embodiment, the step of comparing the data processing caliber of the target field before and after the change based on the first processing information and the second processing information to obtain a processing caliber comparison result includes: performing semantic parsing on the processing logic corresponding to the target field based on the first processing information to obtain first semantic parsing information before the change; and performing semantic parsing on the processing logic corresponding to the target field based on the second processing information to obtain second semantic parsing information after the change; determining the semantic change type corresponding to the target field based on the first semantic parsing information and the second semantic parsing information; the semantic change type includes at least one of field source field change, filter condition change, and aggregation granularity change; determining the data processing caliber change relationship corresponding to the target field based on the semantic change type; and comparing the data processing caliber of the target field before and after the change based on the data processing caliber change relationship to obtain the processing caliber comparison result.

[0042] Semantic parsing is used to convert the processing logic corresponding to the target field into semantic parsing information that can characterize the meaning of data processing. The first semantic parsing information characterizes the semantic content of the target field's processing logic before the change, and the second semantic parsing information characterizes the semantic content of the target field's processing logic after the change. The semantic change type characterizes the semantic dimensions that have changed in the target field's processing logic before and after the change, including at least one of changes in the field source field, changes in filtering conditions, and changes in aggregation granularity. The data processing scope change relationship characterizes the change relationship of the data processing rules corresponding to the target field under different semantic change types.

[0043] Specifically, firstly, semantic parsing is performed on the field expressions, source fields, filtering conditions, and aggregation granularity in the first processed information, converting the processing logic corresponding to the target field into the first semantic parsing information before the change. Simultaneously, the second processed information is parsed using the same semantic parsing method as the first processed information, resulting in the second semantic parsing information after the change. Then, the first and second semantic parsing information are compared to determine the differences in the processing logic before and after the change across different semantic dimensions, and the semantic change type corresponding to the target field is determined based on these differences. For example, when the source field referenced by the target field changes before and after the change, the semantic change type is determined to include a change in the field source field; when the filtering condition corresponding to the target field changes, the semantic change type is determined to include a change in the filtering condition; when the statistical dimension or Group By granularity corresponding to the target field changes, the semantic change type is determined to include a change in the aggregation granularity. Next, the data processing scope change relationship corresponding to the target field is determined based on the semantic change type, and the data processing rules before and after the change are further analyzed based on the data processing scope change relationship. For example, changes in filtering conditions indicate a change in the filtering range; changes in aggregation granularity indicate a change in the statistical object or dimension corresponding to the target field; and changes in the source field indicate a change in the data source of the target field. Finally, based on the determined data processing caliber changes, a comparison is made between the target field's data processing caliber before and after the change to determine whether the target field's data processing caliber has changed, and the specific changes corresponding to this change. The comparison results are then used to generate the processing caliber comparison results. For example, in the implementation scenario of a monthly summary table for scanned customers, the processing logic of the target field `succ_amt` before the change could filter successful transaction data and calculate transaction amounts based on `operation_type=5`. When developers modify the successful transaction recognition logic, the server performs semantic parsing on the processing logic before and after the change, and identifies a difference between the changed filtering conditions and the original filtering conditions. This determines that the target field has changed its filtering conditions and further confirms that its data processing caliber has changed.

[0044] It can be seen that by performing semantic analysis on the processing logic before and after the target field change, and identifying the change type from semantic dimensions such as field source, filtering conditions and aggregation granularity, the apparent SQL syntax change can be further converted into a change relationship at the data processing rule level. This allows for accurate identification of changes in data processing scope when the field name remains unchanged, thereby improving the accuracy of the change risk assessment.

[0045] In an optional embodiment, determining the data processing caliber change relationship corresponding to the target field based on the semantic change type includes: obtaining field annotations and business terms corresponding to the target field; determining business semantic information corresponding to the target field based on the field annotations and business terms; determining the semantic change information of the target field before and after the change based on the first semantic parsing information, the second semantic parsing information, and the semantic change type; determining the business semantic change relationship of the target field before and after the change based on the business semantic information and the semantic change information; and determining the data processing caliber change relationship corresponding to the target field based on the business semantic change relationship.

[0046] Among them, field annotations describe the business meaning of the target field in the data warehouse; business terms represent the business concepts or business objects corresponding to the target field; business semantic information represents the actual meaning of the target field at the business level; semantic change information represents the specific semantic changes that occurred in the processing logic corresponding to the target field before and after the change; business semantic change relationship represents the association between the business meaning of the target field and the changes in its processing logic; and data processing caliber change relationship represents whether and how the processing rules used to determine the field data changed before and after the change.

[0047] Specifically, firstly, the field annotations and business terms of the target field are obtained from the metadata corresponding to the data warehouse. For example, field annotations can explain the specific business data meaning represented by the target field, while business terms can explain the business metrics, business objects, or business status corresponding to the target field. After obtaining the field annotations and business terms, correlation analysis can be performed between the two to determine the business semantic information corresponding to the target field. For example, when the field annotation of the target field indicates that it represents "successful transaction amount," and the business term associated with this field is "successful transaction," the business semantic information corresponding to the target field can be determined as the statistical result of the amount corresponding to a successful transaction based on the field annotation and business term. Further, based on the first semantic parsing information, the second semantic parsing information, and the already determined semantic change type, the semantic change information of the target field before and after the change is determined, respectively. For example, information such as the source field, filtering conditions, and aggregation granularity of the target field can be extracted from the first and second semantic parsing information, and the specific semantic elements that have changed can be determined based on the semantic change type. If the semantic change type is a change in filter conditions, the filter conditions before and after the change and their differences can be identified as the corresponding semantic change information. If the semantic change type is a change in field source, the source fields before and after the change and their differences can be identified as the corresponding semantic change information. If the semantic change type is a change in aggregation granularity, the aggregation granularity before and after the change and their differences can be identified as the corresponding semantic change information. After obtaining the semantic change information, it can be further associated with the business semantic information corresponding to the target field to determine the business semantic change relationship of the target field before and after the change. Specifically, it can be determined whether changes in processing logic such as the source field, filter conditions, or aggregation granularity corresponding to the semantic change information have changed the business object, business scope, or statistical meaning represented by the target field. When the processing logic changes but does not change the business meaning corresponding to the target field, the change can be identified as a logical adjustment that does not involve business semantic changes. When the processing logic changes, causing changes in the actual business object, statistical scope, or statistical granularity corresponding to the target field, it can be determined that the target field has business semantic changes. For example, the field annotation of the target field succ_amt is "transaction success amount", and the corresponding business term is "transaction success". Before the change, the target field used the filter condition `operation_type=5` to calculate the transaction amount. After the change, the filter condition was expanded to include multiple transaction states. If the expanded transaction states include closed, refunded, or failed retry states, then although the field names, field types, and target table structure of `succ_amt` remain unchanged, the actual range of data counted by the field has changed.At this point, the business semantics of the field can be determined based on the field annotations and the business term "transaction successful." Combined with the filtering conditions before and after the change, it can be determined that the business semantics of the field has changed from the original "transaction successful amount" to a transaction amount that includes other transaction statuses, thus determining the corresponding change in data processing scope. Further, after determining the business semantic change relationship, the data processing scope change relationship corresponding to the target field can be determined based on this relationship. Specifically, changes in the source field, filtering conditions, aggregation method, or aggregation granularity corresponding to the business semantic change can be used as specific representations of the data processing scope change, and the data processing scope change relationship can be determined based on the correlation between different semantic changes. In addition, during actual implementation, the semantic parsing results can be used to further verify the business semantics determined by the field annotations and business terms.

[0048] In one possible implementation, requirement descriptions, field annotations, SQL differences, lineage fragments, downstream object names, and historical case summaries can be input into the AI ​​semantic assistance module to generate candidate results such as categories of caliber changes and explanations of business impact. However, the final results need to be supported by rule validation, confidence thresholds, and metadata or SQL evidence. For easier understanding, please refer to [link to relevant documentation]. Figure 4 , Figure 4 This is a schematic diagram of an AI semantic-assisted analysis process provided in an embodiment of this application. As can be seen, the AI ​​semantic-assisted module takes requirement description, SQL differences, field annotations, lineage paths, and historical cases as inputs, performs semantic analysis on data warehouse changes through controlled input, and outputs candidate analysis results in structured JSON. After obtaining the candidate analysis results, the candidate analysis results are further constrained and verified by combining rule validation and manual confirmation, and candidate outputs and auxiliary generation content are generated accordingly.

[0049] Specifically, the requirement description represents the business requirements or purpose of this data warehouse change; the SQL difference represents the differences between the SQL before and after the change; field annotations represent the business meaning of the relevant data table fields; the lineage path represents the data association between the changed data object and its upstream data objects and downstream consumer objects; historical cases represent historical processing information such as historical changes, false alarms, online incidents, remediation plans, and review conclusions. Furthermore, the AI ​​semantic assistance module processes the above input information using a controlled input method and generates structured JSON output according to a preset data structure to avoid generating unconstrained risk judgments directly from natural language. Candidate outputs include: type of change in scope, omitted downstream keywords, and suggested verification dimensions. In some embodiments, the structured JSON output may include candidate results such as type of change in scope, omitted downstream keywords, and suggested verification dimensions, along with corresponding confidence levels and evidence citations. Specifically, type of change in scope represents the business scope change type that the current change may correspond to; omitted downstream keywords represent downstream object keywords that may not be identified based on existing lineage information but may be related to the current change; and suggested verification dimensions represent the direction of further data verification recommended for the current change. Furthermore, candidate results need to undergo rule validation. Specifically, rule validation can include metadata evidence matching, SQL fragment validation, and confidence threshold validation. Metadata evidence matching determines whether candidate results are supported by metadata such as field annotations, lineage paths, and downstream objects; SQL fragment validation determines whether candidate results are corroborated by actually changed SQL fragments; and confidence threshold validation determines whether the credibility of the AI ​​output reaches a preset threshold. Only when a candidate result is supported by corresponding evidence and its confidence level meets preset conditions can it be used as a valid auxiliary result for subsequent analysis. Furthermore, for high-risk or insufficiently supported candidate results, a manual confirmation stage can be initiated, where the responsible person reviews the candidate results output by the AI ​​semantic assistance module to prevent the AI ​​semantic analysis results from directly determining the risk level or the published result. After rule validation and manual confirmation, the AI ​​semantic assistance module can also perform auxiliary generation, generating a verification SQL draft, risk statement, and notification content. Among them, the SQL draft verification is used to assist data verification personnel in comparing the data results before and after the change; the risk statement is used to explain the reasons for the change and the possible impact to developers or business leaders; and the notification content is used to assist in generating impact notifications for relevant responsible persons.

[0050] It can be seen that by introducing the field annotations and business terms of the target field into the semantic parsing process before and after the change, and further associating the changes in processing logic obtained from semantic parsing with the business semantics of the target field, the changes in field processing logic can be explained and judged from the perspective of business meaning. This enables us not only to identify changes in technical aspects such as source fields, filtering conditions, and aggregation granularity, but also to determine whether the above changes actually change the business scope represented by the target field; thereby improving the accuracy of identifying changes in data processing scope.

[0051] In one optional embodiment, determining the data processing caliber change relationship corresponding to the target field based on the business semantic change relationship includes: determining the semantic change difference corresponding to the target field based on the business semantic change relationship; the semantic change difference includes at least one of field source difference, filtering condition difference, aggregation method difference, and granularity difference; determining the processing caliber change type corresponding to the target field based on the semantic change difference; determining the data processing caliber change intensity corresponding to the target field based on the processing caliber change type; the data processing caliber change intensity characterizes the degree of difference in data processing caliber before and after the change of the target field; and determining the data processing caliber change relationship corresponding to the target field based on the processing caliber change type and the data processing caliber change intensity.

[0052] Semantic change difference characterizes the differences in business semantics of the target field before and after the change across different processing dimensions, including at least one of the following: differences in field source, differences in filtering conditions, differences in aggregation methods, and differences in granularity. Data processing caliber change intensity characterizes the degree of difference in data processing caliber of the target field before and after the change. This can be determined based on the type of processing caliber change and the corresponding semantic change difference; the greater the degree of data processing caliber difference, the higher the corresponding data processing caliber change intensity can be. Data processing caliber change relationship is used to associate the type and degree of processing caliber change corresponding to the target field.

[0053] Specifically, firstly, the semantic relationship of the business changes is analyzed to determine the semantic differences of the target field before and after the change. Then, these semantic differences are categorized according to dimensions such as field source, filtering conditions, aggregation methods, and data processing granularity to determine the semantic change differences corresponding to the target field. For example, if the source field referenced by the target field changes before and after the change, a difference in field source is determined; if the data filtering range changes before and after the change, a difference in filtering conditions is determined; if the statistical function changes before and after the change, a difference in aggregation methods is determined; and if the statistical dimensions change before and after the change, a difference in granularity is determined. Next, the processing scope change type corresponding to the target field is determined based on the semantic change differences, and this processing scope change type is associated with the corresponding business semantic change. For example, if the semantic change difference includes a difference in filtering conditions, the processing scope change type can be determined as a change in filtering range; if the semantic change difference includes a difference in aggregation methods, the processing scope change type can be determined as a change in aggregation methods. Finally, the intensity of the data processing scope change of the target field is determined based on the processing scope change type and the corresponding semantic change differences. In some embodiments, the server can pre-set corresponding change levels for different processing caliber change types and determine the intensity of the data processing caliber change based on the actual change content. Finally, the processing caliber change type and the intensity of the data processing caliber change are correlated to obtain the data processing caliber change relationship corresponding to the target field, and the data processing caliber change relationship is used as input for subsequent processing caliber comparison and risk assessment.

[0054] It can be seen that by further refining the semantic changes of business into differences in semantic changes such as field source, filtering conditions, aggregation method and granularity, and determining the specific processing caliber change type and its intensity based on the semantic change differences, the semantic changes of the target field can be further quantified into data processing caliber change relationships that can be used for risk assessment, thereby more accurately distinguishing different types and degrees of data caliber changes.

[0055] Step S304: Based on the diameter change relationship in the processing diameter comparison result, as well as the field lineage relationship and downstream task dependency relationship corresponding to the target field, perform an impact analysis on the downstream associated objects corresponding to the target field to obtain the impact object analysis results.

[0056] Among them, field lineage represents the data transmission relationship between the target field and the downstream field, and downstream task dependency represents the data dependency between different data processing tasks.

[0057] Specifically, based on the field lineage of the target field, downstream fields with direct data transmission relationships can be identified layer by layer along the downstream data transmission direction. After identifying multiple downstream fields, a field propagation path can be established between the target field and these downstream fields based on their field lineage. This field propagation path records the downstream fields the target field passes through sequentially and the data transmission relationships between them, allowing changes in the processing scope of the target field to be tracked according to the actual data processing order. Based on this, multiple downstream tasks associated with the field propagation path are identified based on downstream task dependencies, and further, an impact propagation path corresponding to the target field is established based on the task dependencies between these downstream tasks. This impact propagation path describes the propagation process of changes in the target field's scope from the field level to the task level. Furthermore, based on the scope change relationships in the processing scope comparison results, the specific changes in processing rules that occurred before and after the change in the target field can be analyzed, such as changes in source field, filtering conditions, aggregation methods, or aggregation granularity. This, combined with the impact propagation path, can determine whether these changes will propagate to downstream fields and downstream tasks. When a change in the scope of a target field leads to changes in the data value, calculation method, aggregation result, or data filtering result of a downstream field, it can be determined that there is an impact relationship between the downstream field and the target field, and the downstream task corresponding to the downstream field is identified as the affected object. Finally, the impact object analysis results can be generated based on the identified affected objects and the impact types corresponding to each affected object. The impact type characterizes the nature of the impact of the change in scope on the corresponding downstream object, such as changes in data value, calculation method, aggregation result, or data filtering result. Because it can further identify indirect impacts formed through multi-level data processing and task dependencies, it helps improve the completeness and accuracy of identifying downstream impacts of data warehouse changes.

[0058] In an optional embodiment, the step of performing an impact analysis on downstream related objects corresponding to the target field based on the caliber change relationship in the processing caliber comparison result, as well as the field lineage relationship and downstream task dependency relationship corresponding to the target field, to obtain the impact object analysis result, includes: determining downstream fields that have a direct data transmission relationship with the previous layer field by layer based on the field lineage relationship, thus obtaining multiple downstream fields; establishing a field propagation path between the target field and the multiple downstream fields based on the field lineage relationship; determining multiple downstream tasks associated with the field propagation path based on the downstream task dependency relationship; establishing an impact propagation path corresponding to the target field based on the multiple downstream task dependencies corresponding to the multiple downstream tasks; determining the affected objects among the multiple downstream tasks that have an impact relationship with the target field based on the caliber change relationship and the impact propagation path, and determining the impact type corresponding to the affected objects; and generating the impact object analysis result based on the affected objects and the impact type.

[0059] Among them, the field propagation path represents the data transmission path of the change in the data processing caliber corresponding to the target field among multiple downstream fields, and the influence propagation path represents the path of the change in the data processing caliber corresponding to the target field propagating downstream through multiple downstream fields and multiple downstream tasks.

[0060] Specifically, when determining the downstream associated objects corresponding to the target field, the downstream fields that have a direct data transmission relationship with the target field are first identified based on field lineage, and these identified downstream fields are designated as the first-level downstream fields of the target field. Further, each downstream field in the first-level downstream field is used as the upper-level field, and the next-level downstream fields that have a direct data transmission relationship with the upper-level fields are determined based on field lineage. This process is repeated layer by layer downstream until no new downstream fields exist or a preset downstream expansion condition is met, resulting in multiple downstream fields. For example, the target field can be referenced by downstream fields as the result of the first-level data processing, and downstream fields can be further processed as input to another data processing logic, thus forming a multi-level field data transmission relationship. Through this layer-by-layer determination method, the field propagation relationship between the target field and multiple downstream fields can be obtained. Further, based on the field lineage relationship between each level of downstream fields, the target field and the multiple downstream fields are associated to obtain the field propagation path. After obtaining the field propagation path, multiple downstream tasks corresponding to the downstream fields in the field propagation path can be determined based on downstream task dependencies. For example, when a data processing task reads a downstream field in a field propagation path and generates further data processing results based on that downstream field, the data processing task can be identified as a downstream task associated with the field propagation path. Further, based on the task dependencies between multiple downstream tasks, multiple downstream tasks can be associated according to the data processing order of the tasks to establish an influence propagation path corresponding to the target field. The influence propagation path represents the process by which a change in the scope of the target field is transmitted from the field level to the task level, and then continues to propagate downstream through multiple dependent downstream tasks. In some embodiments, the target field in the data warehouse is the transaction success amount field, and its downstream fields can be used to generate risk characteristics such as transaction amounts for top customers, transaction time periods, and amount ranges. When the processing filtering conditions of the transaction success amount field change, the downstream fields that directly use the transaction success amount field can be determined first through field lineage, and then other downstream fields that use the downstream field for further calculations can be determined; subsequently, the tasks that generate the aforementioned downstream data and other tasks that continue to depend on the output results of the aforementioned tasks are determined based on scheduling dependencies, thereby forming an influence propagation path that propagates from the target field to the downstream fields, and then from the downstream fields to the downstream tasks.

[0061] Furthermore, based on the caliber change relationship in the processing caliber comparison results, the specific processing rule changes of the target field before and after the change are determined, and the changes in processing rules are mapped to the influence propagation path. For example, when the caliber change relationship represents a change in the filtering conditions of the target field, it can be determined whether the change in filtering conditions will be passed to the corresponding downstream field along the influence propagation path; when the calculation logic of the downstream field references the target field, and the change in the filtering range of the target field will cause changes in the data value, calculation caliber, aggregation result, or data filtering result of the downstream field, it can be determined that there is an influence association between the downstream field and the target field, and the downstream tasks that generate or use the downstream field are further identified as affected objects. After identifying the affected objects, the influence type corresponding to the affected objects can be further determined. The influence type can characterize the specific impact of the change in the processing caliber of the target field on the affected objects. Finally, the influence object analysis results are generated based on the identified affected objects and the influence types corresponding to each affected object. The results of the impact object analysis can at least record the object identifier, object type, and corresponding impact type of the affected object. It can also further associate the field propagation path and impact propagation path between the target field and the affected object, so that the subsequent risk scoring module can determine the downstream impact risk parameters based on the number, type, and related metadata information of the affected object.

[0062] For easier understanding, please refer to Figure 5 , Figure 5 This is a schematic diagram illustrating the downstream impact of data warehouse changes, provided in an embodiment of this application. As can be seen, Figure 5 Centered on the Data Warehouse Summary (DWS) target table, this diagram illustrates the data transfer and impact relationships between the actual upstream sources, the DWS target table, and multiple downstream related objects. Specifically, the actual upstream sources process data to form the DWS target table, which is further linked to the monthly width table of scanned ID cards, risk characteristics / model variables, and scheduling links. Additionally, the DWS target table is associated with sensitive field examples, which are further linked to pre-release alerts.

[0063] Specifically, firstly, the actual upstream source provides raw or processed data to the DWS target table. The server can determine the data transfer relationship between the actual upstream source and the DWS target table based on field lineage, and use the DWS target table as the data warehouse object to be analyzed. Then, when the data processing method changes for a target field in the DWS target table, the server can analyze downstream fields and downstream data objects that have direct or indirect data transfer relationships with that target field, layer by layer, based on the field lineage corresponding to the target field. Specifically, the data corresponding to the DWS target table is further transferred to the QR code ID card monthly width table, enabling the QR code ID card monthly width table to generate corresponding downstream data based on the relevant fields in the DWS target table. Simultaneously, the DWS target table is also associated with risk features / model variables, providing a data foundation for downstream risk features or model variables. Next, the server can determine the scheduling links associated with the target field based on the task dependencies corresponding to the DWS target table, to identify data processing tasks and their execution relationships that may be affected by changes in the target field. Furthermore, when a target field in the DWS target table falls within the data range corresponding to a sensitive field example, the data sensitivity level of that target field can be determined based on pre-acquired data sensitivity information, and this sensitivity level can be used as a reference factor for subsequent risk assessment. Finally, by comprehensively considering information such as changes in the data processing scope of the target field, downstream objects, scheduling links, and sensitive field examples, a pre-release warning can be generated, and corresponding risk warnings or release handling results can be output before the actual release of data warehouse changes.

[0064] As can be seen, by displaying the relationships between the real upstream source, DWS target table, QR code ID card monthly width table, risk characteristics / model variables, scheduling links, sensitive field examples, and pre-release warnings, it is possible not only to identify the impact of data warehouse changes on downstream data objects, but also to further combine scheduling links and data sensitivity information to determine change risks and generate pre-release warnings before the change is released, providing a basis for the subsequent implementation of differentiated release access control strategies.

[0065] In an optional embodiment, determining the affected objects among the plurality of downstream tasks that have an impact association with the target field based on the caliber change relationship and the impact propagation path includes: determining the caliber change item corresponding to the target field based on the caliber change relationship; determining the propagation relationship of the caliber change item among the plurality of downstream fields based on the propagation order in the impact propagation path; determining the impact relationship of the caliber change item on each of the plurality of downstream fields based on the field expression corresponding to the plurality of downstream fields and the propagation relationship; determining whether the caliber change item causes a change in the data value, calculation caliber, aggregation result, or data filtering result of the corresponding downstream field based on the impact relationship; and determining the downstream field and the corresponding downstream task as affected objects when it is determined that the caliber change item causes a change in at least one of the data value, calculation caliber, aggregation result, or data filtering result of the downstream field.

[0066] The change in scope represents the specific data processing rules that changed the target field before and after the change. Changes in scope can include changes in source field, filtering conditions, aggregation methods, and aggregation granularity. The propagation relationship represents the association between the change in scope and its propagation path from the target field to downstream fields at each level. The impact relationship represents the influence of the change in scope on the data processing results of the corresponding downstream fields after processing by the field expressions.

[0067] Specifically, firstly, one or more caliber change items corresponding to the target field are determined based on the caliber change relationship. By further breaking down the processing caliber change into specific caliber change items, an analytical object can be provided for subsequent analysis of the impact of each change along the downstream data processing link. In some embodiments, changes in filtering conditions, join conditions, aggregation granularity, and indicator formulas can be considered as identifiable change types, and semantic caliber changes can be further determined based on field annotations, source fields, filtering conditions, aggregation functions, and dimensional granularity. After determining the caliber change items, the caliber change items can be propagated layer by layer according to the propagation order of each field and task in the impact propagation path. Specifically, it can be first determined whether the caliber change item can propagate from the target field to the first-level downstream field; if it is determined that it can propagate to the first-level downstream field, then based on the field propagation relationship between the first-level downstream field and the second-level downstream field, it can be further determined whether the caliber change item can propagate to the second-level downstream field, and so on, until the analysis of the relevant downstream fields in the impact propagation path is completed. Therefore, the propagation relationship of the caliber change item among multiple downstream fields can be obtained, avoiding the inclusion of downstream objects without actual data transmission relationships in the scope of influence. Furthermore, after determining the propagation relationship of the caliber change item among multiple downstream fields, the specific impact of the caliber change item can be analyzed by combining the field expression corresponding to each downstream field. Then, it can be further determined whether the caliber change item will cause changes in the data processing results of the corresponding downstream field. When the caliber change item changes the input data range of the downstream field and causes a change in the data value of the downstream field, it can be determined that the downstream field is affected; when the caliber change item causes a change in the calculation rules used by the downstream field, it can be determined that the calculation caliber of the downstream field has changed; when the downstream field performs summation, counting, or other aggregation processing on the target field, and the caliber change of the target field causes a change in the aggregation input, it can be determined that the aggregation result of the downstream field has changed; when the downstream field further performs conditional filtering based on the target field, and the caliber change causes a change in the data range that meets the filtering conditions, it can be determined that the data filtering result of the downstream field has changed. When a change in the scope of a term causes a change in at least one of the following in a downstream field: its data value, calculation scope, aggregation result, or data filtering result, that downstream field can be identified as an affected object. Based on the downstream task dependencies corresponding to that field, downstream tasks that generate, process, or use that downstream field can also be identified as affected objects. For fields and tasks that continue to propagate downstream along the impact propagation path and produce actual changes in results, the same method can be used to further identify them, thus obtaining a set of affected objects that are actually associated with the target field.Compared to directly considering all downstream objects as affected based on lineage, this approach further establishes a criterion of "whether there is an actual change in data processing results." This reduces invalid affected objects arising from mere field references that are not actually affected by changes in caliber. Finally, the identified affected fields, affected tasks, and corresponding impact types can be associated to form affected objects and their impact types. The impact type can be determined based on the actual assessment results as at least one of the following: changes in data values, changes in calculation caliber, changes in aggregation results, or changes in data filtering results. The affected objects and impact types are incorporated into the impact object analysis results, providing a data foundation for the subsequent calculation of downstream impact risk parameters.

[0068] It can be seen that by breaking down the caliber change relationship into specific caliber change items, analyzing its propagation relationship among multiple downstream fields according to the impact propagation path, and then combining the field expressions of each downstream field to determine the actual impact relationship, it is possible to further distinguish between downstream objects that "have field reference relationships" and those that are "actually affected by the caliber change". After determining that the caliber change has caused changes in the data values, calculation calibers, aggregation results, or data filtering results of downstream fields, the corresponding fields and downstream tasks are then identified as affected objects. This achieves a complete closed loop from caliber change identification, change propagation, impact determination to the identification of affected objects, thereby improving the accuracy of downstream impact analysis.

[0069] Step S305: Based on the processing diameter comparison results and the impact object analysis results, a risk assessment is performed on the change information to obtain the change risk assessment result of the data warehouse.

[0070] The change risk assessment is used to indicate the degree of risk of changes in data processing standards corresponding to the change information and the impact of the change on downstream related objects. The change risk assessment result may include at least one of the following: risk assessment parameters corresponding to the change information, risk level, and risk description corresponding to the risk level.

[0071] Specifically, firstly, based on the comparison results of processing calibers, the changes in data processing calibers corresponding to the changed information are determined, and the caliber change risk parameters corresponding to the changed information are determined based on the changes in data processing calibers. Next, based on the analysis results of the affected objects, the affected objects corresponding to the changed information are determined, and the metadata information corresponding to the affected objects is obtained to determine the downstream impact risk parameters corresponding to the changed information. Then, based on the caliber change risk parameters and the downstream impact risk parameters, a comprehensive calculation is performed to obtain the risk assessment parameters corresponding to the changed information, and the risk assessment parameters are matched with preset risk level information to determine the risk level corresponding to the changed information. Finally, based on the risk level, the change risk assessment result corresponding to the data warehouse is generated. In one possible embodiment, the caliber change risk parameters can be determined based on the type of caliber change and the intensity of the data processing caliber change in the processing caliber comparison results; the downstream impact risk parameters can be determined based on information such as the number of affected objects, object type, consumption importance, and corresponding scheduling timeliness requirements.

[0072] It can be seen that by simultaneously considering changes in data processing standards and the impact on downstream related objects when conducting risk assessments of data warehouse changes, we can avoid judging risks based solely on the content of the change or the lineage of a single field. This allows the risk assessment results to simultaneously reflect the degree of change in data processing rules and the actual scope of downstream impact, thereby improving the accuracy of hierarchical identification of data warehouse change risks.

[0073] In one optional embodiment, the step of performing a risk assessment on the change information based on the processing caliber comparison result and the impact object analysis result to obtain the change risk assessment result of the data warehouse includes: determining the caliber change risk parameter corresponding to the change information based on the processing caliber comparison result; determining the downstream impact risk parameter corresponding to the change information based on the impact object analysis result and the metadata information corresponding to the impact object; calculating the risk assessment parameter corresponding to the change information based on the caliber change risk parameter and the downstream impact risk parameter; determining the risk level corresponding to the change information based on preset risk level information and the risk assessment parameter; and generating the change risk assessment result of the data warehouse based on the risk level.

[0074] The downstream impact risk parameter characterizes the degree of impact on related objects after the change in processing caliber corresponding to the change information propagates downstream. It can be determined based on the number of affected objects, the type of affected objects, and the metadata information corresponding to the affected objects in the analysis results. The metadata information includes the business attributes and data usage attributes corresponding to the affected objects, and may include at least one of the following: business level, consumption status, task scheduling information, data sensitivity information, and responsible person information. The risk assessment parameter comprehensively characterizes the overall change risk level corresponding to the change information. The risk level information establishes the correspondence between the risk assessment parameter and the risk level. For example, the risk assessment parameter can be divided into different risk ranges, corresponding to low risk, medium risk, high risk, or severe risk levels, respectively.

[0075] Specifically, firstly, based on the comparison results of processing calibers, the changes in the data processing caliber of the target field before and after the change are determined. Then, based on the type and degree of change in the data processing caliber, the corresponding caliber change risk parameters are determined. Next, based on the analysis results of the affected objects, the affected objects corresponding to the change information are determined. Then, the metadata information corresponding to each affected object is obtained, and the downstream impact risk parameters corresponding to the change information are determined based on the number of affected objects, object type, and corresponding business level, consumption status, scheduling timeliness requirements, and data sensitivity. For example, when a target field with a changed processing caliber is referenced by multiple downstream tasks, risk indicators, or data reports, and the downstream objects have high business importance or high timeliness requirements, the downstream impact risk parameters corresponding to the change information can be determined to be high. Next, the caliber change risk parameters and the downstream impact risk parameters are comprehensively calculated to obtain the risk assessment parameters corresponding to the change information. In some embodiments, the caliber change risk parameters and the downstream impact risk parameters can be weighted to obtain the risk assessment parameters; wherein the weights corresponding to each parameter can be pre-configured according to the actual business needs of the data warehouse. Furthermore, the risk assessment parameters are matched with preset risk level information, and the risk level corresponding to the change information is determined based on the risk range in which the risk assessment parameters fall. Finally, a change risk assessment result for the data warehouse is generated based on the determined risk level, and the change risk assessment result is associated with and stored with the change information, so that the access control module can determine the corresponding release processing strategy based on the risk level.

[0076] In an optional embodiment, the step of calculating the risk assessment parameters corresponding to the change information based on the caliber change risk parameter and the downstream impact risk parameter includes: determining the caliber change score corresponding to the change information based on the caliber change risk parameter; determining the number and type of downstream objects corresponding to the change information based on the impact object analysis results; determining the consumption importance score corresponding to the change information based on the number and type of downstream objects; determining the timeliness requirement score corresponding to the change information based on the task scheduling information corresponding to the impact objects; determining the data sensitivity score corresponding to the change information based on the data sensitivity information corresponding to the impact objects; determining the historical problem score corresponding to the change information based on the historical problem records corresponding to the change information; and determining the risk assessment parameters corresponding to the change information based on the caliber change score, the consumption importance score, the timeliness requirement score, the data sensitivity score, and the historical problem score.

[0077] Specifically, the change in scope represents the degree of change in the data processing scope corresponding to the change information; the importance of consumption represents the importance of the downstream objects corresponding to the change information to the consumption of data in the data warehouse; the timeliness requirement represents the timeliness requirement of data processing or data output corresponding to the affected objects; the data sensitivity represents the sensitivity of the data used or processed by the affected objects; and the historical issues represent the impact of historical change issues or historical data issues related to the change information on the risk of this change. Downstream object types may include at least one of downstream fields, data indicators, data reports, data interfaces, data quality rules, and scheduled tasks. Task scheduling information may include at least one of task scheduling cycle, task execution time, and data output time limit corresponding to the task; data sensitivity information may include data sensitivity level or data security classification information.

[0078] Specifically, firstly, the change score corresponding to the change information is determined based on the risk parameters of the change in processing caliber. When the filtering conditions, source fields, aggregation methods, or aggregation granularity of the target field representing the processing caliber comparison results change, the server determines the change score based on the corresponding data processing caliber change type and intensity; the greater the degree of change in data processing caliber, the higher the corresponding change score can be. Then, based on the impact object analysis results, the number of downstream objects corresponding to the change information is counted, and the downstream objects are type-identified to obtain the downstream object type; next, the consumption importance score is determined based on the number and type of downstream objects. Next, the task scheduling information corresponding to the affected objects is obtained, and the timeliness requirement score is determined based on the task scheduling information; for downstream tasks with shorter data production cycles or higher scheduling timeliness requirements, a higher timeliness requirement score can be determined. Simultaneously, the data sensitivity information corresponding to the affected objects is obtained, and the data sensitivity score is determined based on the data sensitivity information; when the affected objects involve highly sensitive data, a higher data sensitivity score can be determined. Finally, the corresponding historical problem records are retrieved based on the change information, and the historical problem score is determined based on the historical problem records. Finally, the risk assessment parameters corresponding to the changed information are obtained by comprehensively calculating the scores for changes in scope, consumption importance, timeliness requirements, data sensitivity, and historical issues. In one specific implementation, corresponding weights can be assigned to each of the above risk scores, and a weighted calculation can be performed based on each risk score and its corresponding weight to obtain the risk assessment parameters; alternatively, the risk scores can be accumulated according to preset risk scoring rules to obtain the risk assessment parameters. The risk assessment parameters are further used to match with preset risk level information to determine the risk level corresponding to the changed information.

[0079] It can be seen that by quantifying the degree of change in data processing standards, the consumption importance of downstream objects, the timeliness requirements of tasks, the sensitivity of data, and the historical issues into corresponding risk scores, and by comprehensively calculating multiple risk scores, we can comprehensively evaluate data warehouse changes from multiple dimensions such as the data change itself, the scope of downstream consumption, business timeliness, data sensitivity, and historical risks. This improves the ability of risk assessment results to represent the impact on actual business and provides a quantitative basis for implementing differentiated access control strategies according to different risk levels.

[0080] With the above Figure 3 For embodiments consistent with those described, please refer to [link / reference]. Figure 6 , Figure 6This is a flowchart illustrating the access control process for changing published information, provided in an embodiment of this application. After performing a risk assessment on the changed information based on the processing caliber comparison results and the impact object analysis results to obtain the change risk assessment results for the data warehouse, the process further includes the following steps: S601. Match the risk level in the change risk assessment result with the preset access control release conditions to obtain the access control release matching result; S602. Determine the release access control policy corresponding to the changed information based on the release access control matching result; the release access control policy includes an automatic release policy, a developer confirmation policy, a responsible person approval policy, and a release blocking policy. S603. When the access control policy is published as the automatic release policy, the publishing operation corresponding to the change information is allowed to be executed. S604. When the access control policy is published and confirmed by the developer, the publishing operation corresponding to the change information is paused, and the change risk assessment result is output. S605. When the access control policy is published as the approval policy of the person in charge, the publishing operation corresponding to the change information is suspended, and the change risk assessment result is output to the person in charge. S606. When the access control policy is the blocking policy, prevent the execution of the publishing operation corresponding to the change information, and output the risk warning information corresponding to the change risk assessment result.

[0081] The access control conditions are used to establish the correspondence between risk levels and access control policies. These conditions can be pre-configured according to the business requirements of the data warehouse. The access control matching results represent the matching status between the risk levels in the change risk assessment results and each access control condition. The access control policies are used to determine, based on the matching status, whether the change information can be directly published, whether it needs to be confirmed by developers or approved by the person in charge, and whether it needs to be blocked from publication.

[0082] Specifically, after obtaining the change risk assessment results, the risk level corresponding to the change information can be obtained from the risk assessment results, and the risk level can be matched with the pre-configured release access control conditions. Release access control conditions can be divided into multiple risk ranges according to the degree of risk, and corresponding release processing requirements can be configured for different risk ranges. For example, changes corresponding to lower risk levels can be configured to allow direct release, changes corresponding to medium risk levels can be configured to require developer confirmation, changes corresponding to higher risk levels can be configured to require manager approval, and changes corresponding to severe risk levels can be configured to block release. Thus, the release access control matching result can be determined based on the matching result between the risk level and the release access control conditions. Further, the release access control policy corresponding to the change information is determined based on the release access control matching result. When the risk level meets the release access control conditions corresponding to the automatic release policy, the automatic release policy can be determined as the release access control policy corresponding to the change information; when the risk level meets the release access control conditions corresponding to the developer confirmation policy, the developer confirmation policy can be determined; when the risk level meets the release access control conditions corresponding to the manager approval policy, the manager approval policy can be determined; and when the risk level meets the release access control conditions corresponding to the blocking release policy, the blocking release policy can be determined. The different strategies described above correspond to different levels of release control, allowing risk assessment results to be directly translated into control conditions in the release process. When the release access control strategy is an automatic release strategy, it indicates that the risk level of the changed information meets the preset low-risk release conditions. In this case, the release operation corresponding to the changed information can be allowed, enabling changes that meet the low-risk conditions to enter the release process without additional manual intervention. When the release access control strategy is a developer confirmation strategy, it indicates that although the changed information is not a high-risk change that needs to be directly blocked, its risk level has reached the point where further manual verification is required. Therefore, the release operation corresponding to the changed information can be suspended, and the change risk assessment result can be output to the developer who submitted or processed the change, allowing the developer to confirm the change based on the risk level, changes in definition, and downstream affected objects. After the developer completes the confirmation and the corresponding conditions for continuing release are met, the current release suspension can be lifted, and subsequent release operations can continue. When the release access control strategy is a responsible person approval strategy, it indicates that the risk level of the changed information is further increased, requiring review of the change risk by a responsible person with corresponding business or data responsibility authority. Therefore, the release operation corresponding to the changed information can be suspended, and the responsible person can be identified based on the affected objects in the impact object analysis results. The change risk assessment results can be output to the responsible person. The change risk assessment results may include information such as risk level, risk cause, and affected objects, enabling the responsible person to judge whether to continue the release based on the actual business impact.When the access control policy is set to block publication, it indicates that the risk level corresponding to the changed information has reached the preset severe risk condition. In this case, the publication operation for the changed information is not allowed to continue; instead, the publication process corresponding to the changed information is blocked, and a risk warning message corresponding to the change risk assessment result is output. The risk warning message can be used to explain the risk reason that triggered the publication block, the affected objects, and the risk matters requiring further handling. For example, for a change where the field name has not changed but the processing caliber has changed significantly, and this field is used by multiple important downstream risk characteristics, it can be identified as a high-risk or severe-risk change. Developers should be required to supplement the comparison results of the old and new calibers and the confirmation results of the downstream impact before the publication access control is lifted based on the confirmation results.

[0083] Furthermore, when implementing the aforementioned access control policy, information such as the risk level, risk cause, affected objects, and impact type from the change risk assessment results can be used as the basis for access control processing, and the corresponding processing results can be recorded. For example, for changes requiring developer confirmation or manager approval, the confirmation or approval result can be recorded; for blocked changes, the reason for blocking and subsequent processing results can be recorded. This allows for a correlation between the risk assessment result of a change and the actual release and handling results, providing a data foundation for subsequent risk rule adjustments and historical case analysis.

[0084] For easier understanding, please refer to Figure 7 , Figure 7 This is a diagram illustrating the results of a data warehouse change risk assessment provided in an embodiment of this application. As can be seen, Figure 7The presentation shows a summary of the changes. Specifically, the server determines the risk assessment parameters corresponding to the changes based on the comparison results of the processing caliber corresponding to the data warehouse changes and the analysis results of the affected objects. It then determines the risk level corresponding to the changes based on preset risk level information. A risk level of "high risk, release blocked" indicates that the current change may have a significant impact on related business data in the data warehouse. Next, a corresponding "risk reason" is generated based on the identified changes in data processing caliber. The corresponding risk reason is "Field name and type unchanged, but filter conditions changed, potentially altering the amount and number of successful transactions." This means that although the field name and type of the target field have not changed, the filter conditions corresponding to the target field have changed, which may lead to a change in the scope of successful transaction data filtering, thereby affecting the amount and number of successful transactions. Following this, "evidence sources" are generated based on the relevant analytical information used in the risk assessment process to provide corresponding analytical basis for the current risk level and risk reasons. Simultaneously, the "real downstream objects" corresponding to the changes are determined based on field lineage and downstream task dependencies to show the actual downstream objects associated with this change and their scope of impact. Then, based on the identified change risks, an "AI-assisted SQL draft for verification" can be generated to assist relevant personnel in performing data verification for this change. Finally, based on the determined risk level, risk cause, and downstream impact, corresponding "suggested actions" are generated, including: "1. Retain the original successful transaction criteria; 2. If the status needs to be expanded, add an independent field to carry it; 3. Layered comparison; 4. Notify the scanning wide table and personnel with complex risk control characteristics to confirm," so that relevant personnel can take appropriate change handling measures based on the risk assessment results.

[0085] As can be seen, by matching the risk levels in the change risk assessment results with preset release access conditions, and adopting differentiated strategies such as automatic release, developer confirmation, responsible person approval, and release blocking for different risk levels, the risk identification results of data warehouse changes can be further transformed into specific release control actions. This allows low-risk changes to be released quickly, while changes with higher downstream impact risks undergo corresponding levels of manual confirmation or approval before release, and release is blocked for serious risk changes. In this way, the risk assessment, risk classification, and release control of data warehouse changes can form a closed loop, reducing the risk of erroneous data entering downstream tables, indicators, reports, or risk control tasks due to the failure to identify changes in processing methods in a timely manner.

[0086] Please see Figure 8 , Figure 8This application provides a flowchart for data warehouse change risk assessment and release access control. First, in the "S1, Change Submission" stage, developers submit the data warehouse changes to be released to the intelligent assessment system. These changes may include "SQL, DDL, Flow, quality rules entering the release process," etc. The system obtains the corresponding change information and uses this information as the object of subsequent risk analysis. Then, in the "S2, Difference Parsing" stage, the server obtains the data processing logic before and after the change, and parses it to extract processing information such as "source table, field expression, filtering conditions, aggregation granularity," and determines the differences between the processing information before and after the change. Based on this, in the "S3, Metadata Completion" stage, further metadata information related to the changed object is obtained, supplementing the parsed processing information, including "completing field comments, responsible persons, SLAs, consumption records," etc., to provide a more complete information foundation for subsequent business semantic understanding and risk assessment. Next, in the "S4, AI Semantic Understanding" stage, business semantic analysis is performed on the changed content by combining the processing information, field annotations, and business-related information before and after the change. This analysis aims to "summarize changes in caliber, business impact, and verification points," determining the changes in data processing caliber and potential business impact corresponding to the data warehouse change from a business perspective, and providing a basis for subsequent verification. Then, in the "S5, Calibration Comparison" stage, the data processing caliber before and after the change is compared based on the aforementioned analysis and semantic understanding results to "identify fields that remain unchanged but whose business meaning has changed." For example, even if the field name and field type of the target field have not changed, if the filter conditions, source fields, or aggregation granularity corresponding to the target field change, the server can still identify that the business meaning or data processing caliber of the target field has changed, thus avoiding judging the impact of the change solely based on the field name or table structure. Next, in the "S6, Impact Diffusion" stage, based on the field lineage and downstream task dependencies of the target field, a layer-by-layer propagation analysis of the data processing caliber change is conducted. This is combined with a method of "tracing downstream along lineage, scheduling, BI logs, and interface logs" to identify downstream objects with an impact related to the change. By analyzing field propagation relationships and task dependencies, the downstream fields, data metrics, reports, interfaces, and related scheduling tasks that may be affected by the target field's caliber change can be identified. Then, in the "S7, Risk Scoring" stage, considering both the data processing caliber change and the downstream impact, the risk of this change is quantified, and "low risk, medium risk, high risk, and severe risk" are calculated. Specifically, risk assessment parameters can be determined based on factors such as the degree of caliber change, the importance of downstream consumption, task timeliness requirements, data sensitivity, and historical issues, and the corresponding risk level is determined based on these parameters.Next, in the "S8, Release Access Control" stage, the determined risk level is matched with the preset release access control conditions to execute the corresponding release control in "Pass, Reminder, Approval, or Block". Finally, in the "S9, Early Warning Feedback" stage, corresponding "Output Description, Verification SQL, Notification, and Feedback" are output based on the risk assessment results and release access control processing results. The output description explains the risk level and cause of the change to relevant personnel; the verification SQL assists relevant personnel in verifying the data results before and after the change; the notification sends the risk information to the corresponding developers or responsible persons; and the feedback records the change processing process and results for reference in subsequent change risk analysis. Further, after the above processing is completed, the server generates the corresponding "S10, Access Control Result", which includes "Low Risk: Automatic Pass", "Medium Risk: Reminder Confirmation", "High Risk: Responsible Person Approval", and "Severe Risk: Release Blocked". Thus, differentiated release control is executed according to different risk levels, achieving a closed-loop data warehouse change risk management system from change identification, definition analysis, impact assessment to release decision-making and feedback.

[0087] As can be seen, by implementing the above-described risk assessment method for data warehouse changes, the following steps are taken: First, change information of the data warehouse is obtained. Second, the structured query statements before and after the change are parsed and normalized to obtain change difference information. Third, the change difference information is supplemented according to the metadata base table to obtain the business metadata corresponding to the target field. Fourth, the semantic definition of the target field is compared with the change difference information and the business metadata to obtain the definition change type corresponding to the target field. Fifth, the diffusion analysis is performed downstream along the field lineage, task dependency relationship and data consumption record according to the target field to obtain the list of affected objects. Sixth, the risk score of the change is performed according to the definition change type and the list of affected objects to obtain the risk level of the change. Fifth, the release access control process is performed on the change according to the risk level, and the warning record corresponding to the change is generated. In this way, because the semantic definition of the structured query statements before and after the change is compared, hidden changes such as changes in field names and table structure but changes in processing definition can be identified. Even if the task is running normally and the field lineage has not changed, the risk of definition change can still be detected before release, thereby reducing the risk of definition change flowing into the production environment and causing deviation of downstream indicators. Furthermore, risk scoring is performed based on the type of change in scope and the list of affected objects, and release access control is executed to improve the accuracy of data warehouse change release control.

[0088] The above primarily describes the solutions of the embodiments of this application from the perspective of the method execution process. It is understood that, in order to achieve the above functions, the server includes the corresponding hardware structure and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in the embodiments provided herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0089] This application embodiment can divide the server into functional units according to the above method example. For example, each function can be divided into a separate functional unit, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0090] When dividing each function into modules according to its corresponding function. Figure 9 This is a functional module block diagram of a data warehouse change risk assessment device 900 provided in an embodiment of this application. The data warehouse change risk assessment device 900 includes: Data acquisition module 910 is used to acquire change information from the data warehouse; The change determination module 920 is used to determine, based on the change information, the first processing information of the data warehouse before the change and the second processing information after the change; the first processing information and the second processing information respectively include at least one of the following: the field expression corresponding to the target field, the source field, the filtering condition, and the aggregation granularity; The impact analysis module 930 is used to compare the data processing scope before and after the change of the target field based on the first processing information and the second processing information to obtain the processing scope comparison result; and to perform impact analysis on the downstream related objects corresponding to the target field based on the scope change relationship in the processing scope comparison result, as well as the field lineage relationship and downstream task dependency relationship corresponding to the target field to obtain the impact object analysis result. The risk assessment module 940 is used to conduct a risk assessment on the change information based on the processing caliber comparison results and the impact object analysis results, and obtain the change risk assessment results of the data warehouse.

[0091] In an optional embodiment, the impact analysis module 930, in comparing the data processing calibers of the target field before and after the change based on the first processing information and the second processing information to obtain a processing caliber comparison result, is specifically configured to: perform semantic parsing on the processing logic corresponding to the target field based on the first processing information to obtain first semantic parsing information before the change; and perform semantic parsing on the processing logic corresponding to the target field based on the second processing information to obtain second semantic parsing information after the change; determine the semantic change type corresponding to the target field based on the first semantic parsing information and the second semantic parsing information; the semantic change type includes at least one of field source field change, filter condition change, and aggregation granularity change; determine the data processing caliber change relationship corresponding to the target field based on the semantic change type; and compare the data processing calibers of the target field before and after the change based on the data processing caliber change relationship to obtain the processing caliber comparison result.

[0092] In an optional embodiment, the impact analysis module 930, in determining the data processing caliber change relationship corresponding to the target field based on the semantic change type, is specifically configured to: obtain field annotations and business terms corresponding to the target field; determine the business semantic information corresponding to the target field based on the field annotations and the business terms; determine the semantic change information of the target field before and after the change based on the first semantic parsing information, the second semantic parsing information, and the semantic change type; determine the business semantic change relationship of the target field before and after the change based on the business semantic information and the semantic change information; and determine the data processing caliber change relationship corresponding to the target field based on the business semantic change relationship.

[0093] In an optional embodiment, the impact analysis module 930, in determining the data processing caliber change relationship corresponding to the target field based on the business semantic change relationship, is specifically configured to: determine the semantic change difference corresponding to the target field based on the business semantic change relationship; the semantic change difference includes at least one of field source difference, filtering condition difference, aggregation method difference, and granularity difference; determine the processing caliber change type corresponding to the target field based on the semantic change difference; determine the data processing caliber change intensity corresponding to the target field based on the processing caliber change type; the data processing caliber change intensity characterizes the degree of difference in data processing caliber before and after the change of the target field; and determine the data processing caliber change relationship corresponding to the target field based on the processing caliber change type and the data processing caliber change intensity.

[0094] In an optional embodiment, the impact analysis module 930, in performing impact analysis on downstream related objects corresponding to the target field based on the caliber change relationship in the processing caliber comparison result, and the field lineage relationship and downstream task dependency relationship corresponding to the target field, to obtain the impact object analysis result, is specifically used for: determining downstream fields that have a direct data transmission relationship with the previous layer field by layer based on the field lineage relationship, thus obtaining multiple downstream fields; establishing a field propagation path between the target field and the multiple downstream fields based on the field lineage relationship; determining multiple downstream tasks associated with the field propagation path based on the downstream task dependency relationship; establishing an impact propagation path corresponding to the target field based on the multiple downstream task dependencies corresponding to the multiple downstream tasks; determining the affected objects among the multiple downstream tasks that have an impact relationship with the target field based on the caliber change relationship and the impact propagation path, and determining the impact type corresponding to the affected objects; and generating the impact object analysis result based on the affected objects and the impact type.

[0095] In an optional embodiment, the impact analysis module 930, in determining the affected objects among the plurality of downstream tasks that have an impact association with the target field based on the caliber change relationship and the impact propagation path, is specifically configured to: determine the caliber change item corresponding to the target field based on the caliber change relationship; determine the propagation relationship of the caliber change item among the plurality of downstream fields based on the propagation order in the impact propagation path; determine the impact relationship of the caliber change item on each of the plurality of downstream fields based on the field expression corresponding to the plurality of downstream fields and the propagation relationship; determine whether the caliber change item causes a change in the data value, calculation caliber, aggregation result, or data filtering result of the corresponding downstream field based on the impact relationship; and when it is determined that the caliber change item causes a change in at least one of the data value, calculation caliber, aggregation result, or data filtering result of the downstream field, identify the downstream field and the corresponding downstream task as affected objects.

[0096] In an optional embodiment, the risk assessment module 940, in performing a risk assessment on the change information based on the processing caliber comparison result and the impact object analysis result to obtain the change risk assessment result of the data warehouse, is specifically configured to: determine the caliber change risk parameter corresponding to the change information based on the processing caliber comparison result; and determine the downstream impact risk parameter corresponding to the change information based on the impact object analysis result and the metadata information corresponding to the impact object; calculate the risk assessment parameter corresponding to the change information based on the caliber change risk parameter and the downstream impact risk parameter; determine the risk level corresponding to the change information based on preset risk level information and the risk assessment parameter; and generate the change risk assessment result of the data warehouse based on the risk level.

[0097] In an optional embodiment, the risk assessment module 940, in calculating the risk assessment parameters corresponding to the change information based on the caliber change risk parameter and the downstream impact risk parameter, is specifically configured to: determine the caliber change score corresponding to the change information based on the caliber change risk parameter; determine the number and type of downstream objects corresponding to the change information based on the impact object analysis results; determine the consumption importance score corresponding to the change information based on the number and type of downstream objects; determine the timeliness requirement score corresponding to the change information based on the task scheduling information corresponding to the impact objects; determine the data sensitivity score corresponding to the change information based on the data sensitivity information corresponding to the impact objects; determine the historical problem score corresponding to the change information based on the historical problem records corresponding to the change information; and determine the risk assessment parameters corresponding to the change information based on the caliber change score, the consumption importance score, the timeliness requirement score, the data sensitivity score, and the historical problem score.

[0098] In an optional embodiment, after the risk assessment module 940 performs a risk assessment on the change information based on the processing caliber comparison result and the impact object analysis result to obtain the change risk assessment result of the data warehouse, it is further configured to: match the risk level in the change risk assessment result with preset release access control conditions to obtain a release access control matching result; determine the release access control policy corresponding to the change information based on the release access control matching result; the release access control policy includes an automatic release policy, a developer confirmation policy, a responsible person approval policy, and a blocking release policy; when the release access control policy is the automatic release policy, the release operation corresponding to the change information is allowed to be executed; when the release access control policy is the developer confirmation policy, the release operation corresponding to the change information is suspended, and the change risk assessment result is output; when the release access control policy is the responsible person approval policy, the release operation corresponding to the change information is suspended, and the change risk assessment result is output to the responsible person; when the release access control policy is the blocking release policy, the release operation corresponding to the change information is blocked, and risk warning information corresponding to the change risk assessment result is output.

[0099] As can be seen, by executing the data warehouse change risk assessment device provided in this application embodiment, the change information of the data warehouse is obtained, and the first processing information and the second processing information of the data warehouse before the change are determined based on the change information; the data processing scope of the target field before and after the change is compared based on the first processing information and the second processing information to obtain the processing scope comparison result; the impact analysis of the downstream related objects corresponding to the target field is performed based on the scope change relationship in the processing scope comparison result, as well as the field lineage relationship and downstream task dependency relationship corresponding to the target field to obtain the impact object analysis result; the change information is risk-assessed based on the processing scope comparison result and the impact object analysis result to obtain the data warehouse change risk assessment result. Thus, the scope and risk of the impact can be accurately determined before the data warehouse change is released.

[0100] This application also provides a computer-readable storage medium storing a computer program for electronic data interchange, which causes a computer to perform some or all of the steps of any of the methods described in the above method embodiments, wherein the computer includes a server.

[0101] It should be noted that, for the sake of simplicity, the above embodiments are all described as a series of actions. Those skilled in the art should understand that this application is not limited to the described order of actions, as some steps in the embodiments of this application can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions, steps, modules, or units involved are not necessarily essential to the embodiments of this application.

[0102] In the above embodiments, the descriptions of each embodiment in this application have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0103] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0104] The steps of the methods or algorithms described in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in RAM, flash memory, ROM, EPROM, electrically erasable programmable read-only memory (EEPROM), registers, hard disk, portable hard disk, read-only optical disk (CD-ROM), or any other form of storage medium well known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Furthermore, the ASIC can reside in a terminal device or management device. Alternatively, the processor and storage medium can exist as discrete components in the terminal device or management device.

[0105] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in the embodiments of this application can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, in whole or in part, in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of this application are generated, in whole or in part. The modules / units included in the various devices and products described in the above embodiments can be software modules / units, hardware modules / units, or a combination of both. For example, for various devices and products applied to or integrated into terminal devices, the modules / units included can all be implemented using hardware such as circuits. Different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or different components within the terminal device. Alternatively, at least some modules / units can be implemented using software programs that run on a processor integrated within the terminal device, while the remaining (if any) modules / units can be implemented using hardware such as circuits.

[0106] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the embodiments of this application. It should be understood that the above descriptions are merely specific embodiments of the embodiments of this application and are not intended to limit the protection scope of the embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solutions of the embodiments of this application should be included within the protection scope of the embodiments of this application.

Claims

1. A risk assessment method for data warehouse changes, characterized in that, A server in an intelligent evaluation system applied to a data warehouse, the method comprising: Obtain change information from the data warehouse; Based on the change information, the data warehouse is determined to have first processing information before the change and second processing information after the change; the first processing information and the second processing information respectively include at least one of the following: the field expression corresponding to the target field, the source field, the filtering condition, and the aggregation granularity; Based on the first processing information and the second processing information, the data processing caliber of the target field before and after the change is compared to obtain the processing caliber comparison result. Based on the diameter change relationship in the comparison results of the processing diameter, as well as the field lineage relationship and downstream task dependency relationship corresponding to the target field, an impact analysis is performed on the downstream related objects corresponding to the target field to obtain the impact object analysis results; Based on the comparison results of the processing calibers and the analysis results of the affected objects, a risk assessment is performed on the change information to obtain the change risk assessment results of the data warehouse; The step of comparing the data processing scope of the target field before and after the change based on the first processing information and the second processing information to obtain the processing scope comparison result includes: Based on the first processing information, semantic parsing is performed on the processing logic corresponding to the target field to obtain the first semantic parsing information before the change; and based on the second processing information, semantic parsing is performed on the processing logic corresponding to the target field to obtain the second semantic parsing information after the change; The semantic change type corresponding to the target field is determined based on the first semantic parsing information and the second semantic parsing information; the semantic change type includes at least one of the following: change of field source field, change of filtering conditions, and change of aggregation granularity. Determine the data processing caliber change relationship corresponding to the target field based on the semantic change type; Based on the changes in the data processing standards and the auxiliary analysis results, the data processing standards of the target field before and after the change are compared to obtain the processing standard comparison results; the auxiliary analysis results are the results of semantic analysis of multi-source change information by the AI ​​analysis module.

2. The method as described in claim 1, characterized in that, The step of determining the data processing caliber change relationship corresponding to the target field based on the semantic change type includes: Obtain the field comments and business terms corresponding to the target field; The business semantic information corresponding to the target field is determined based on the field annotation and the business terminology; Based on the first semantic parsing information, the second semantic parsing information, and the semantic change type, determine the semantic change information of the target field before and after the change; Determine the business semantic change relationship of the target field before and after the change based on the business semantic information and the semantic change information; The data processing caliber change relationship corresponding to the target field is determined based on the business semantic change relationship.

3. The method as described in claim 2, characterized in that, Determining the data processing caliber change relationship corresponding to the target field based on the business semantic change relationship includes: The semantic change differences corresponding to the target field are determined based on the business semantic change relationship; the semantic change differences include at least one of the following: differences in field source, differences in filtering conditions, differences in aggregation methods, and differences in granularity. The processing caliber change type corresponding to the target field is determined based on the semantic change differences; The intensity of the data processing caliber change corresponding to the target field is determined based on the type of processing caliber change; the intensity of the data processing caliber change characterizes the degree of difference in the data processing caliber of the target field before and after the change; The data processing diameter change relationship corresponding to the target field is determined based on the type of change in processing diameter and the intensity of the change in data processing diameter.

4. The method according to any one of claims 1-3, characterized in that, The impact analysis is performed on the downstream related objects corresponding to the target field based on the diameter change relationship in the processing diameter comparison results, as well as the field lineage relationship and downstream task dependency relationship corresponding to the target field, to obtain the impact object analysis results, including: Based on the lineage relationship of the fields, starting from the target field, downstream fields that have a direct data transmission relationship with the field at the previous level are determined layer by layer to obtain multiple downstream fields; Establish a field propagation path between the target field and the multiple downstream fields based on the field lineage relationship; Based on the downstream task dependencies, determine multiple downstream tasks associated with the field propagation path; Establish the impact propagation path corresponding to the target field based on the dependencies between the multiple downstream tasks; Based on the changes in the caliber and the propagation path of the impact, identify the affected objects in the multiple downstream tasks that are associated with the target field, and determine the impact type corresponding to the affected objects; Based on the affected objects and the type of impact, the analysis results of the affected objects are generated.

5. The method as described in claim 4, characterized in that, The step of determining the affected objects in the multiple downstream tasks that have an impact association with the target field based on the caliber change relationship and the impact propagation path includes: Determine the caliber change item corresponding to the target field based on the caliber change relationship; The propagation relationship of the caliber change term among the multiple downstream fields is determined based on the propagation order in the propagation path. The influence relationship of the caliber change term on each of the multiple downstream fields is determined based on the field expressions corresponding to the multiple downstream fields and the propagation relationship. Based on the influence relationship, determine whether the change in the caliber causes changes in the data value, calculation caliber, aggregation result, or data filtering result of the corresponding downstream field; When it is determined that the change in the caliber causes a change in at least one of the data value, calculation caliber, aggregation result, or data filtering result of the downstream field, the downstream field and the corresponding downstream task are identified as the affected objects.

6. The method according to any one of claims 1-3, characterized in that, The step of conducting a risk assessment of the change information based on the processing caliber comparison results and the impact object analysis results to obtain the change risk assessment results of the data warehouse includes: Based on the comparison results of the processing calibers, determine the caliber change risk parameters corresponding to the change information; and based on the analysis results of the affected objects and the metadata information corresponding to the affected objects, determine the downstream impact risk parameters corresponding to the change information. Based on the risk parameters of the change in caliber and the risk parameters of the downstream impact, the risk assessment parameters corresponding to the change information are calculated. The risk level corresponding to the change information is determined based on the preset risk level information and the risk assessment parameters. Based on the risk level, generate a change risk assessment result for the data warehouse.

7. The method as described in claim 6, characterized in that, The risk assessment parameters corresponding to the change information are calculated based on the risk parameters of the change in caliber and the risk parameters of downstream impact, including: The caliber change score corresponding to the change information is determined based on the caliber change risk parameters. Based on the results of the impact object analysis, determine the number and type of downstream objects corresponding to the change information; and based on the number and type of downstream objects, determine the consumption importance score corresponding to the change information. The timeliness requirement score corresponding to the change information is determined based on the task scheduling information corresponding to the affected object; and the data sensitivity score corresponding to the change information is determined based on the data sensitivity information corresponding to the affected object; and the historical problem score corresponding to the change information is determined based on the historical problem record corresponding to the change information. The risk assessment parameters corresponding to the change information are determined based on the change in scope score, the consumption importance score, the timeliness requirement score, the data sensitivity score, and the historical issue score.

8. The method as described in claim 1, characterized in that, After performing a risk assessment on the change information based on the processing caliber comparison results and the impact object analysis results to obtain the change risk assessment results for the data warehouse, the method further includes: The risk level in the change risk assessment results is matched with the preset access control release conditions to obtain the access control release matching results; The release access control policy corresponding to the changed information is determined based on the release access control matching results; the release access control policy includes an automatic release policy, a developer confirmation policy, a responsible person approval policy, and a release blocking policy. When the access control policy is set to the automatic release policy, the release operation corresponding to the change information is allowed to be executed. When the access control policy is published and confirmed by the developer, the publication operation corresponding to the change information is paused, and the change risk assessment result is output. When the access control policy is published as an approval policy for the person in charge, the publishing operation corresponding to the change information is suspended, and the change risk assessment result is output to the person in charge. When the access control policy is the blocking policy, the release operation corresponding to the change information is prevented, and risk warning information corresponding to the change risk assessment result is output.

9. A risk assessment device for data warehouse changes, characterized in that, A server used in an intelligent evaluation system for a data warehouse, the device comprising: The data acquisition module is used to acquire change information from the data warehouse. The change determination module is used to determine, based on the change information, the first processing information of the data warehouse before the change and the second processing information after the change; the first processing information and the second processing information respectively include at least one of the following: the field expression corresponding to the target field, the source field, the filtering condition, and the aggregation granularity; The impact analysis module is used to compare the data processing scope of the target field before and after the change based on the first processing information and the second processing information to obtain the processing scope comparison result; and to perform impact analysis on the downstream related objects corresponding to the target field based on the scope change relationship in the processing scope comparison result, as well as the field lineage relationship and downstream task dependency relationship corresponding to the target field to obtain the impact object analysis result. The risk assessment module is used to assess the risk of the change information based on the processing caliber comparison results and the impact object analysis results, and obtain the change risk assessment results of the data warehouse. The step of comparing the data processing scope of the target field before and after the change based on the first processing information and the second processing information to obtain the processing scope comparison result includes: Based on the first processing information, semantic parsing is performed on the processing logic corresponding to the target field to obtain the first semantic parsing information before the change; and based on the second processing information, semantic parsing is performed on the processing logic corresponding to the target field to obtain the second semantic parsing information after the change; The semantic change type corresponding to the target field is determined based on the first semantic parsing information and the second semantic parsing information; the semantic change type includes at least one of the following: change of field source field, change of filtering conditions, and change of aggregation granularity. Determine the data processing caliber change relationship corresponding to the target field based on the semantic change type; Based on the changes in the data processing standards and the auxiliary analysis results, the data processing standards of the target field before and after the change are compared to obtain the processing standard comparison results; the auxiliary analysis results are the results of semantic analysis of multi-source change information by the AI ​​analysis module.

Images