A contract automation management method

Abstract

The present invention discloses a method for automated contract management and control, which relates to the technical field of automated management and control, including: constructing constraints on data extraction frequency according to the maximum change frequency of contract data and the load capacity of a data middle platform architecture; constructing a contract data conversion rule library, and quantifying the initial data conversion complexity according to the algorithm complexity, data type, and connection depth of the conversion rule; obtaining a plurality of conversion arrays; and applying the conversion arrays to perform ETL respectively, obtaining resource occupancy data, delay data, and conversion quality data of the ETL process, and constructing an ETL evaluation model; making the output of the ETL evaluation model correspond to the conversion array and constructing a three-dimensional effect display model, obtaining the selected conversion array, and applying it to automated contract management and control. The present application solves the problem that it is difficult to strike a balance between data extraction frequency and data conversion complexity to simultaneously meet real-time requirements and diversified data processing.

Description

Technical Field

[0001] The present invention relates to the field of automated control technology, and more specifically, to a method for automated contract control. Background Art

[0002] Existing contract automation management methods mainly rely on the contract life cycle management system, which uses information technology to achieve automation and process management of contract drafting, approval, execution, archiving and other links. These systems are usually based on preset templates, intelligent approval processes and contract data analysis functions, which significantly improve the efficiency and compliance of contract processing. However, with the diversification of corporate business and the expansion of geographical distribution, contract data is often stored in different department systems, such as ERP, CRM, financial systems and third-party cooperation platforms. This decentralized storage and independent operation have formed the so-called data islands, which severely limit the overall effect of contract automation management and control, and make it difficult to achieve cross-system data sharing and collaboration.

[0003] In order to solve the problem of data islands, existing technologies have proposed a variety of data integration methods. Among them, ETL (Extract-Transform-Load) technology extracts scattered data from the source system and loads it into the target system after unified transformation processing, thus achieving data standardization and centralization; API interface integration dynamically acquires and synchronizes data by establishing direct communication between systems, which is suitable for scenarios with high real-time requirements.

[0004] The data middle platform architecture builds a unified data processing and sharing platform to connect the data of various business systems and provide standardized data services for upper-level applications. Each technology has alleviated the problem of data silos to a certain extent, but the differences in complexity, real-time performance, and applicable scenarios are still important aspects that need to be weighed in current data integration technology.

[0005] For example, the invention patent with announcement number: CN109472442B announces an automated processing system and method for work orders, including an automated processing unit for work orders and an automated management and control platform. The automated processing unit for work orders is used to provide a work order application template, and to form an instantiation work order according to the user's input, and to parse the instantiation work order information into a machine-executable task parameter form, and to execute the work order according to the executable task parameter form; the automated management and control platform is used for users to select corresponding work order templates to create instantiation work orders, and to send the user-instantiated work orders to the automated processing unit for analysis. The system can automatically process work orders, and can be flexibly expanded for different types of work orders, to achieve rapid automated work order execution, speed up the entire work order circulation time, and reduce the workload of operation and maintenance personnel.

[0006] For example, the invention patent with announcement number: CN116341885A announces an automated production control system and method based on big data, including recording the scanned code information in automated production, marking the main beat and sub-beat of the minimum unit production line respectively, so as to simplify the intricate relationship between production lines or equipment, and conduct micro-coordinated analysis of multi-dimensional factors; analyze the efficiency anomalies and stability of the minimum unit production line, and then amplify the abnormal changes in the process flow; in the historical product production cycle, find out the associated minimum unit production lines, and calculate the impact association probability between the minimum unit production lines; mark the stability of the minimum unit production line in real time, and calculate the real-time impact probability between the minimum unit production lines; and then be able to grasp the main contradictions in automated production, conduct targeted supervision and prediction, make the monitoring scope or boundary clear, and avoid a large amount of manpower and material resources for inspection and supervision.

[0007] There are at least the following technical problems in the above-mentioned disclosed technical solutions: in the data middle-office architecture used to solve the problem of data islands in contract automation management and control, the adjustment of data extraction frequency and data conversion complexity is particularly important for the processing effect. A high frequency can ensure timely data updates, but may lead to an increase in system load. Complex conversions will improve data quality, but will also increase development and execution costs.

[0008] The higher the extraction frequency, the greater the amount of real-time data that the system needs to process. If the conversion complexity is high, the system processing time will be further increased, which may lead to the accumulation of data processing delays. When the conversion complexity is reduced, even if the extraction frequency is high, the system pressure will be small, but it may affect the data quality. If both the extraction frequency and the conversion complexity are set high, the system is prone to overload. It is necessary to balance the two based on system capabilities and real-time requirements while ensuring data quality.

[0009] Although the data middle platform architecture used in existing contract automation management and control has advantages in solving the data island problem, it lacks a flexible dynamic optimization mechanism for adjusting the frequency of data extraction and the complexity of data conversion, and it is difficult to simultaneously meet the balance between real-time requirements and diversified data processing. In view of the above problems, the present invention proposes a solution. Summary of the invention

[0010] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides a contract automation management and control method, which evaluates and screens different data extraction frequencies and data conversion complexities to solve the problem that the adjustment between data extraction frequency and data conversion complexity lacks a flexible dynamic optimization mechanism, and it is difficult to simultaneously meet the real-time requirements and the balance of diversified data processing.

[0011] To achieve the above object, the present invention provides the following technical solutions:

[0012] A contract automation management and control method comprises the following steps: constructing a constraint condition of data extraction frequency according to the maximum change frequency of contract data and the load capacity of a data middle platform architecture, and obtaining an initial data extraction frequency according to historical data; constructing a contract data conversion rule library, and quantifying the initial data conversion complexity according to the algorithm complexity, data type and connection depth of the conversion rule; performing several random perturbations on the initial data extraction frequency and the initial data conversion complexity, and randomly arranging and combining them into several conversion arrays; obtaining contract data from several scattered source systems, and applying the conversion arrays to perform ETL respectively, obtaining resource occupancy data, delay data and conversion quality data of the ETL process, and constructing an ETL evaluation model; making correspondence between the output of the ETL evaluation model and the conversion array and constructing a three-dimensional effect display model, performing surface analysis on the three-dimensional effect display model, obtaining a selected conversion array, and applying it to contract automation management and control.

[0013] In a preferred embodiment, constraints on data extraction frequency are constructed based on the maximum change frequency of contract data and the load capacity of the data middle platform architecture, specifically: analyzing the change patterns of historical contract data, counting the update frequency of contract data, testing the processing capabilities of the data middle platform architecture at different concurrent extraction frequencies, recording the maximum data processing volume, and combining the bandwidth, storage, and computing resources of the data middle platform to determine the maximum extraction frequency supported by the system; obtaining the initial data extraction frequency based on historical data, specifically: obtaining historical contract data within a preset historical time from each source system, the historical contract data includes generating timestamps and change records, classifying and recording different types of contract data changes, and calculating the change frequency of each change type; according to business needs, select the inverse of the minimum change interval of the change type closest to the needs as the initial data extraction frequency.

[0014] In a preferred embodiment, the initial data conversion complexity is quantified according to the algorithmic complexity, data type and connection depth of the conversion rule, specifically: the time complexity and space complexity of each conversion rule are calculated, and the sum is used as the algorithmic complexity of the conversion rule; the data type includes structured data, semi-structured data and unstructured data; adaptation factors are assigned to different types of data; the structured data adaptation factor defaults to 1; the semi-structured data adaptation factor defaults to 2; the unstructured data adaptation factor defaults to 3; the number of connection tables involved in each conversion rule is obtained, and the number of connection tables is used as the quantitative result of the connection depth; the algorithmic complexity, adaptation factor and connection depth of the rules in the conversion process of the preset standard ETL are summed up as the initial data conversion complexity.

[0015] In a preferred embodiment, the initial data extraction frequency and the initial data conversion complexity are randomly perturbed several times, and randomly arranged and combined into several transformation arrays, specifically: taking the initial data extraction frequency as a benchmark, and setting upper and lower limits as the perturbation range according to the data extraction frequency constraint conditions; taking the initial data conversion complexity as a benchmark, and setting the data conversion complexity of the maximum rule as the upper limit, and 0 as the lower limit as the perturbation range; within the perturbation range, randomly generating several data extraction frequencies and data conversion complexities; and randomly arranging and combining the generated data extraction frequencies and data conversion complexities to form several transformation arrays.

[0016] In a preferred embodiment, a transformation array is applied to perform ETL, specifically: the data extraction frequency in the transformation array is used as the data extraction frequency of ETL; according to the data conversion complexity in the transformation array, the ETL process is adjusted by increasing or decreasing the number of conversion rules, adjusting the data processing granularity, controlling the data transformation method, and adjusting parallel processing and single-thread processing, so that the data conversion complexity is applied to the ETL conversion process.

[0017] In a preferred embodiment, the resource occupancy data includes a resource occupancy impact coefficient, and a specific method for obtaining the resource occupancy impact coefficient is as follows: obtain the system resource occupancy rate of several time periods under different data extraction frequencies in the ETL process; calculate the standard deviation and average value of the system resource occupancy rate within the same time interval; calculate the coefficient of variation of the resource occupancy rate based on the standard deviation and average value of the system resource occupancy rate; calculate the resource occupancy impact coefficient based on the resource occupancy rate variation coefficient based on a preset resource occupancy impact coefficient calculation formula.

[0018] In a preferred embodiment, the delay data includes a synchronization timeliness influence coefficient, and a specific method for obtaining the synchronization timeliness influence coefficient is as follows: at different data extraction frequencies, the processing time required from initiating a data extraction request to completing data loading and generating a conversion result is obtained as the response time; for the same data extraction frequency, several response time calculations are performed to calculate the average response time; the data processing volume and total duration of the system at different data extraction frequencies are obtained, and the system data processing rate is calculated; an initial timeliness decreasing effect model is set according to the average response time and the system data processing rate; several response time points are set for system performance testing, and at each response time point, timeliness score data of the data processing results are collected; based on the experience score data, the undetermined parameters of the initial timeliness decreasing effect model are optimized based on the nonlinear least squares method to obtain a calculation model for the synchronization timeliness influence coefficient, and the undetermined parameters include the maximum value of the timeliness score, the rate of decrease of the score with the increase of the response time, and the sensitivity coefficient of the response time to the decreasing score.

[0019] In a preferred embodiment, the conversion quality data includes a data quality impact coefficient, and a specific method for obtaining the data quality impact coefficient is as follows: through consistency verification in the data conversion process, the data conversion deviation generated is set as a benchmark deviation value; the data conversion deviation with a deviation value less than the benchmark deviation value is set as an acceptable deviation, and the data conversion deviation with a deviation value greater than the benchmark deviation value is set as a potential hidden danger deviation; all data conversion deviation values ​​generated in the data conversion process are obtained, and these deviation values ​​are compared with the benchmark deviation value; a hidden danger deviation set is established for all potential hidden danger deviation values, and an acceptable deviation set is established for all acceptable deviation values; data analysis is performed based on the hidden danger deviation set and the acceptable deviation set, and the data quality impact coefficient is calculated.

[0020] In a preferred embodiment, the output of the ETL evaluation model is matched with the transformation array and a three-dimensional effect display model is constructed, specifically: a three-dimensional coordinate system is constructed to display the influence of different transformation arrays on the comprehensive effect in the ETL process; the three dimensions of the three-dimensional coordinates respectively represent the data extraction frequency, the data conversion complexity and the ETL evaluation coefficient; according to the output result of the ETL evaluation model, the evaluation result of each transformation array is combined with its corresponding data extraction frequency and data conversion complexity and mapped to the three-dimensional space; each transformation array corresponds to a point, and the position of the point is determined by the output of the ETL evaluation model and the transformation array; a surface consisting of points is drawn in the three-dimensional coordinate system using a surface fitting algorithm to obtain a three-dimensional effect display model.

[0021] In a preferred embodiment, a surface analysis is performed on the three-dimensional effect display model to obtain a selected conversion array, and the array is applied to contract automation management and control, specifically: the peak value of the three-dimensional effect display model is determined, and the peak value is used as a benchmark to expand the selected area according to a preset control range; according to business needs, the selected areas in the surface are weighed. If the real-time requirement is high, the conversion array corresponding to the peak value in the area close to the high-frequency extraction in the surface is selected as the selected conversion array, giving priority to meeting the synchronization timeliness impact coefficient; if the data quality requirement is higher, the conversion array corresponding to the peak value in the area with higher conversion complexity in the surface is selected as the selected conversion array to ensure the best data quality impact coefficient; if system resources are limited, the conversion array corresponding to the peak value in the entire surface area is selected as the selected conversion array.

[0022] The technical effects and advantages of a contract automation management method of the present invention are as follows:

[0023] 1. The present invention improves the efficiency and quality of the ETL (extraction, transformation, loading) process by finely managing and optimizing the frequency of contract data extraction and the complexity of data conversion. First, by analyzing the change rules and historical data of contract data, reasonable constraints on data extraction frequency are constructed to ensure that the data extraction frequency matches the system load capacity and avoid excessive pressure on system resources. Secondly, a contract data conversion rule library is constructed, and by quantifying the complexity and data type of the conversion rules, the complexity of the data conversion process is flexibly adjusted to ensure that the balance between real-time requirements and data processing requirements is met in different business scenarios. Finally, by introducing the resource occupancy influence coefficient, the synchronization timeliness influence coefficient and the conversion quality influence coefficient, the data extraction frequency and conversion complexity are dynamically optimized, the real-time, controllable and flexible nature of the ETL process are improved, and the problems of uneven resource allocation and low data processing efficiency are effectively solved. The scheme can significantly improve the comprehensive effect of the data processing process, meet diversified business needs, and ensure that data extraction and conversion are carried out efficiently and accurately.

[0024] 2. The present invention constructs a three-dimensional effect display model to effectively correspond the output of the ETL evaluation model to the transformation array, and can intuitively display the comprehensive effects of different transformation arrays in the ETL process. The technical effect of this method is to visualize the relationship between data extraction frequency, data conversion complexity and ETL evaluation coefficient through a three-dimensional coordinate system, so that the evaluation and selection of the most suitable transformation array becomes more accurate and intuitive. Through surface analysis, the peak area of ​​the system can be identified, and the optimal transformation array can be flexibly selected according to specific business needs: if real-time is emphasized, the area with higher frequency is given priority; if data quality is emphasized, the area with higher conversion complexity is selected; if faced with system resource limitations, the most suitable transformation array is selected from the full surface area. This technical solution can effectively improve the accuracy of contract automation management and control, optimize the execution efficiency of the ETL process, and dynamically adjust resource allocation according to actual needs to ensure that data processing meets real-time and quality requirements, thereby improving the overall efficiency and flexibility of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 The present invention is a schematic diagram of the structure of a contract automation management and control method. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] Embodiment 1, Figure 1 The present invention provides a contract automation management method, which includes the following steps:

[0028] S1, build constraints on data extraction frequency based on the maximum change frequency of contract data and the load capacity of the data middle platform architecture, and obtain the initial data extraction frequency based on historical data.

[0029] In this embodiment, the constraints on the data extraction frequency are constructed according to the maximum change frequency of the contract data and the load capacity of the data middle platform architecture, specifically:

[0030] Analyze the changing patterns of historical contract data, count the update frequency of contract data, test the processing capabilities of the data center architecture at different concurrent extraction frequencies, record the maximum data processing volume, and determine the maximum extraction frequency supported by the system based on the bandwidth, storage, and computing resources of the data center;

[0031] The frequency of data extraction needs to meet the following constraints:

[0032]

[0033] In the formula, is the minimum update frequency of contract data, is the data extraction frequency, is the extraction frequency determined according to the maximum change frequency, The maximum extraction frequency allowed by the load capacity of the data center architecture.

[0034] In this embodiment, the initial data extraction frequency is obtained according to historical data, specifically:

[0035] Obtain historical contract data within a preset historical period from each source system. The historical contract data includes generation timestamps and change records, classifies and records different types of contract data changes, and calculates the change frequency of each change type;

[0036] According to business needs, select the inverse of the minimum change interval of the change type closest to the needs as the initial data extraction frequency;

[0037] It should be noted that the source systems refer to different data storage or management platforms. These systems run independently and the data is not interconnected. They include relational databases, non-relational databases, file storage systems, third-party API interfaces, or dedicated systems within the enterprise. For example, a company's contract management process involves payment data in the financial system, contracted employee information in the human resources system, project contract details in the business system, and data from external supplier systems. These source systems have formed so-called data islands due to differences in technical architecture, data format, and management rules, which poses challenges to data integration and unified management.

[0038] It should be noted that different types of contract data changes are closely related to the life cycle and management needs of the contract, including the following categories: basic data changes, such as contract signing time and contract number. These changes usually occur when the contract is created, with a low frequency but stable data; key terms changes, such as contract amount, payment terms, and delivery date. These changes are highly business sensitive and usually occur during contract modification or negotiation; status flow changes, from draft, under approval to effective, completed, etc. The frequency depends on the complexity of the approval and execution process; attachments and supplementary information changes, such as adding supplementary agreements or updating contract attachments. These changes are sudden and mostly depend on business scenarios; log changes, such as audit logs that record contract operations or related change records. These changes are frequent but have limited direct impact on the business. For these types of changes, different monitoring and processing strategies should be set according to their importance and timeliness to ensure the comprehensiveness and accuracy of contract data.

[0039] It should be noted that business needs are the core driving force of contract automation management and control, and different business scenarios have different requirements for the sensitivity and timeliness of contract data changes. In scenarios with complex approval processes, businesses are more concerned about changes in status flow, such as the update of the contract status from "under approval" to "effective" to ensure real-time monitoring of key nodes. In scenarios involving dynamic negotiations or price adjustments, changes in key terms, such as changes in amount and delivery date, are the primary focus to avoid affecting the core interests of contract performance. For businesses with legal or audit compliance requirements, log changes are preferred because this type of data is directly related to the traceability and compliance verification of the operation process. Selecting the closest change type requires in-depth analysis of key indicators of business needs, such as timeliness, change frequency, and the degree of impact on contract execution, and comparing them with the data change type to ensure that the selected change type can meet business needs to the greatest extent possible and avoid waste of resources and omission of key data.

[0040] S2, build a contract data conversion rule base and quantify the initial data conversion complexity based on the algorithm complexity, data type and connection depth of the conversion rules.

[0041] In this embodiment, a contract data conversion rule base is constructed, specifically:

[0042] Combined with the characteristics of various types of contract data (such as structured, semi-structured, and unstructured data), the existing conversion rules in the current system are collected, including field mapping, data cleaning, format conversion, cross-table joins, data aggregation, and other operations, and sorted and archived in a classified manner. At the same time, according to the characteristics of different source systems and specific conversion requirements in business processes, these rules are further optimized and supplemented to form a rule set with wider coverage, and a contract data conversion rule library is obtained;

[0043] It should be noted that according to the diversified conversion scenarios of contract data, the rules in the rule base are arranged and combined according to the needs. For example, for status-type contract data, the status flow rules and field standardization rules in the rule base can be used first; for contracts involving core data such as amount and date, the amount conversion rules, date format standardization rules and related field connection rules are used first. By combining and applying multiple rules, accurate processing of complex contract data can be achieved.

[0044] In this embodiment, the initial data conversion complexity is quantified according to the algorithm complexity of the conversion rule, the data type, and the connection depth, specifically:

[0045] Calculate the time complexity and space complexity of each conversion rule, and sum them up as the algorithm complexity of the conversion rule;

[0046] Data types include structured data, semi-structured data, and unstructured data; adaptation factors are assigned to different types of data;

[0047] The default adaptation factor for structured data is 1; the default adaptation factor for semi-structured data is 2; the default adaptation factor for unstructured data is 3;

[0048] Obtain the number of join tables involved in each transformation rule, and use the number of join tables as the quantitative result of the join depth;

[0049] The algorithm complexity, adaptation factor and connection depth of the rules in the preset standard ETL transformation process are summed up as the initial data conversion complexity.

[0050] It should be noted that the adaptation factor can be adjusted according to actual needs; structured data, such as amount, date, etc.; semi-structured data, such as clause text; unstructured data, such as scanned copies; join depth refers to the complexity of data integration across multiple tables or systems during the contract data conversion process.

[0051] S3, randomly perturb the initial data extraction frequency and the initial data conversion complexity several times, and randomly arrange and combine them into several transformation arrays.

[0052] In this embodiment, the initial data extraction frequency and the initial data conversion complexity are randomly perturbed several times and randomly arranged and combined into several conversion arrays, specifically:

[0053] Taking the initial data extraction frequency as the benchmark, and setting the upper and lower limits as the disturbance range according to the data extraction frequency constraints;

[0054] Based on the initial data conversion complexity as the benchmark, the data conversion complexity of the maximum rule is set as the upper limit, and 0 is the lower limit as the perturbation range;

[0055] Within the perturbation range, several data extraction frequencies and data conversion complexities are randomly generated;

[0056] The generated data extraction frequency and data conversion complexity are randomly arranged and combined to form several transformation arrays.

[0057] Embodiment 2, S4, obtains contract data from several scattered source systems, and applies transformation arrays to perform ETL respectively, obtains resource occupancy data, delay data and conversion quality data of the ETL process, and builds an ETL evaluation model based on the bp neural network.

[0058] Apply the transformation array for ETL, specifically:

[0059] The data extraction frequency in the transformation array is used as the data extraction frequency of ETL;

[0060] According to the data conversion complexity in the conversion array, the ETL process is adjusted by increasing or decreasing the number of conversion rules, adjusting the data processing granularity, controlling the data transformation method, and adjusting parallel processing and single-threaded processing, so that the data conversion complexity is applied to the ETL conversion process.

[0061] It should be noted that there are multiple adjustment combinations for adjusting the ETL transformation process close to a certain data conversion complexity, and the adjustment combination closest to the data conversion complexity is selected as the application method.

[0062] The resource occupancy data includes a resource occupancy influence coefficient, the delay data includes a synchronization timeliness influence coefficient, and the conversion quality data includes a data quality influence coefficient.

[0063] The resource occupancy impact coefficient is used to quantify the resource occupancy during the data extraction process, reflecting the impact of different data extraction frequencies on the source system. High-frequency extraction will lead to frequent access to the source system, increase the system's computing and network load, and may affect the performance of the source system (such as database performance degradation or network bandwidth exhaustion), thereby affecting the stability and responsiveness of the system. Low-frequency extraction relatively reduces the burden on the source system, but may cause a lag in the target data, especially in scenarios with high real-time requirements, and may not be able to meet immediate data needs. By measuring these impacts, this coefficient evaluates the resource occupancy and comprehensive effect of the ETL process under different combinations of data extraction frequencies and data conversion complexities, providing a decision-making basis for ETL optimization.

[0064] Analyzing the resource occupancy impact coefficient has the following advantages for evaluating the comprehensive effect of ETL under different data extraction frequencies and data conversion complexities, and solving the problem of lack of flexible dynamic optimization mechanism for adjusting data extraction frequency and data conversion complexity, and difficulty in meeting the balance between real-time requirements and diversified data processing at the same time:

[0065] Improve the efficiency of resource allocation: By quantifying the resource occupancy impact coefficient, we can more accurately evaluate the occupancy of system resources under different data extraction frequencies and conversion complexities, ensure the reasonable allocation of resources, avoid excessive consumption of system load, and thus improve the overall efficiency of the ETL process.

[0066] Optimize the balance between real-time and data processing: The resource occupancy impact coefficient can help reasonably adjust the relationship between data extraction frequency and conversion complexity while meeting real-time requirements, ensuring timely extraction and loading of real-time data while avoiding system performance degradation due to excessively high frequency or complex conversion, helping to achieve a balance in data processing.

[0067] Flexible dynamic optimization mechanism: By introducing the resource occupancy impact coefficient, a flexible dynamic optimization mechanism can be established. According to the system load, business needs, and changes in data extraction frequency and conversion complexity, the system can adjust parameters in real time to ensure that the ETL process can run efficiently, adapt to fluctuations in business needs, and ensure that the data extraction and conversion process is executed in the optimal state.

[0068] Meeting diverse data processing needs: In different business scenarios, data processing requirements may vary. The resource usage impact coefficient provides an adjustable basis, which can flexibly adjust the data extraction frequency and conversion complexity according to the needs of different business scenarios to ensure that different types of data are processed reasonably while avoiding unnecessary waste of resources.

[0069] Improve the transparency and controllability of the ETL process: By quantifying multiple indicators such as resource usage, timeliness and data quality, we can clearly understand the performance of the ETL process in each link, so as to facilitate corresponding adjustments and optimizations, improve the transparency and controllability of the system, and help solve the balance between real-time requirements and complex data processing.

[0070] The specific method for obtaining the resource occupancy influence coefficient is as follows:

[0071] Obtain the system resource usage for several time periods at different data extraction frequencies during the ETL process;

[0072] Calculate the standard deviation and average value of system resource usage within the same time interval;

[0073] Calculate the coefficient of variation of resource utilization based on the standard deviation and average value of system resource utilization;

[0074] The resource occupancy rate variation coefficient is calculated based on a preset resource occupancy influence coefficient calculation formula to calculate the resource occupancy influence coefficient.

[0075] The specific calculation formula of the coefficient of variation of the resource occupancy rate is as follows:

[0076]

[0077] The specific calculation formula of the resource occupation impact coefficient is as follows:

[0078]

[0079] In the formula, is the coefficient of variation of resource utilization, is the system resource occupancy rate at the jth moment, is the total time, j is the time label; is the resource occupancy impact coefficient.

[0080] The synchronization timeliness impact coefficient is used to quantify the data timeliness in the ETL process, specifically evaluating the delay from extracting data from the source system to loading it into the target system, and whether the target data is available for use in accordance with business requirements. This coefficient takes into account extraction timeliness, that is, the time from data extraction from the source system to completion of loading into the target system, loading timeliness, that is, whether the data is available for business use on time, and update frequency matching, that is, the degree of fit between the frequency of data extraction and the frequency of updates required by the business. In real-time or quasi-real-time application scenarios, timeliness indicators are particularly important, helping to optimize the scheduling strategy for data extraction and loading to ensure that business needs are responded to in a timely manner. The calculation of the synchronization timeliness impact coefficient can evaluate the timeliness effect of the ETL process under different data extraction frequencies and data conversion complexities, guide the reasonable scheduling of ETL tasks, and ensure the consistency of data synchronization efficiency and business needs.

[0081] Analyzing the synchronization timeliness impact coefficient has the following advantages for evaluating the comprehensive effect of ETL under different data extraction frequencies and data conversion complexities, and solving the problem of lack of flexible dynamic optimization mechanism for adjusting data extraction frequency and data conversion complexity, and difficulty in meeting the balance between real-time requirements and diversified data processing at the same time:

[0082] Enhanced real-time optimization capabilities: The synchronization timeliness impact coefficient can accurately quantify the delay from data extraction to the completion of loading into the target system. This coefficient can be used to dynamically monitor and adjust the timeliness of the ETL process, thereby meeting the business's requirements for data real-time performance and ensuring that the system can respond in a timely manner under different loads and data complexities.

[0083] Improve the flexibility of scheduling strategies: The synchronization timeliness impact coefficient provides a quantitative basis for optimizing the data extraction frequency and transformation complexity in the ETL process. The data extraction frequency and transformation complexity can be flexibly adjusted according to the real-time changes in business needs to achieve optimal timeliness and load balance, avoid resource waste and ensure timely data processing.

[0084] Support dynamic adjustment and optimization: Synchronizing the timeliness impact coefficient helps to build a dynamic optimization mechanism. By real-time tracking and analyzing the extraction timeliness, loading timeliness, and update frequency matching, the system can automatically adjust the frequency of data extraction and the complexity of conversion to ensure that data processing meets real-time requirements without sacrificing data processing quality or system load capacity.

[0085] Optimize the trade-off between data timeliness and complexity: In the process of adjusting data extraction frequency and conversion complexity, the balance between real-time requirements and system load is crucial. The synchronization timeliness impact coefficient can help accurately evaluate and optimize this balance, so that the system can cope with different types of data processing requirements while ensuring timely data updates and avoiding real-time data processing lags or overloads.

[0086] Adapt to the needs of diverse business scenarios: Different business scenarios may have different requirements for timeliness. The synchronization timeliness impact coefficient can provide customized optimization solutions for different business scenarios, ensuring that the frequency of data extraction and the complexity of conversion can be appropriately adjusted according to changes in business needs to achieve the best timeliness and processing effect.

[0087] Improve the controllability and stability of the ETL system: By quantifying the timeliness impact coefficient, we can gain a deeper understanding of the timeliness bottlenecks and potential problems in the data processing process, making the ETL process more controllable in actual applications and ensuring that real-time data can be processed in a timely manner without affecting system stability in the case of diverse data and complex transformations.

[0088] The specific method for obtaining the synchronization timeliness influence coefficient is as follows:

[0089] At different data extraction frequencies, the processing time required from initiating a data extraction request to completing data loading and generating conversion results is obtained as the response time;

[0090] For the same data extraction frequency, perform several response time calculations and calculate the average response time;

[0091] Obtain the data processing volume and total time of the system at different data extraction frequencies, and calculate the system data processing rate;

[0092] Set the initial timeliness reduction effect model based on the average response time and system data processing rate;

[0093] Set several response time points to perform system performance testing, and collect timeliness score data of data processing results at each response time point;

[0094] According to the experience rating data, the undetermined parameters of the initial timeliness decreasing effect model are optimized based on the nonlinear least squares method to obtain a calculation model for the synchronous timeliness influence coefficient. The undetermined parameters include the maximum value of the timeliness score, the rate of decrease of the score with increasing response time, and the sensitivity coefficient of response time to the decreasing score.

[0095] The specific calculation formula of the average response time is as follows:

[0096]

[0097] The specific calculation formula of the adjustment step length of the time-effectiveness decreasing effect model is as follows:

[0098]

[0099] The specific calculation formula of the synchronization timeliness influence coefficient is as follows:

[0100]

[0101] In the formula, is the average response time, is the response time of the ith measurement, is the number of times the response time is measured at the same data extraction frequency, is the adjustment step size of the time-dependent decreasing effect model, The amount of data used to score the timeliness of the data processing results for optimization, The label of the timeliness scoring data of the data processing results, The timeliness score data value for the data processing results. is the synchronization timeliness influence coefficient.

[0102] The data quality impact coefficient is used to quantify the impact of data quality in the ETL process and evaluate the accuracy, completeness and consistency of data during the conversion process. High-complexity conversions can usually correct errors in source data, unify data standards, and enrich semantic information through more complex correction and standardization processes, thereby significantly improving data quality and ensuring the reliability and consistency of data in subsequent use. In contrast, although low-complexity conversions may maintain the originality of the data, they are prone to inconsistencies, missing data, or errors, affecting data quality. The calculation of the data quality impact coefficient helps evaluate the impact of the ETL process on data quality under different combinations of data extraction frequency and conversion complexity, providing a basis for optimizing the ETL process and ensuring that the final data meets business needs and high quality standards.

[0103] Analyzing the data quality impact coefficient has the following advantages for evaluating the comprehensive effect of ETL under different data extraction frequencies and data conversion complexities, and solving the problem of lack of flexible dynamic optimization mechanism for adjusting data extraction frequency and data conversion complexity, and difficulty in meeting the balance between real-time requirements and diversified data processing at the same time:

[0104] Optimize the balance between data quality and real-time performance: The data quality impact factor can quantify the relationship between data quality and conversion complexity, ensuring that a reasonable balance can be found between real-time requirements and data quality when adjusting data extraction frequency and conversion complexity. High-complexity conversion can improve data quality, while low-complexity conversion may sacrifice data consistency and integrity. The data quality impact factor can ensure that data quality is not affected while ensuring real-time and diversified data processing.

[0105] Improve data consistency and accuracy: Highly complex transformation rules help correct errors in source data and unify data standards, thereby improving data consistency and accuracy. The data quality impact factor can quantify this effect and ensure that data accuracy and consistency are effectively improved during the ETL process, especially when dealing with scattered data from multiple source systems, which can better avoid inconsistent or missing data problems.

[0106] Enhance the intelligent adjustment capability of the ETL process: Through the data quality impact coefficient, the system can provide real-time feedback on changes in data quality and dynamically adjust the data extraction frequency and conversion complexity accordingly. The system can flexibly select appropriate conversion rules and extraction frequency based on data quality feedback to avoid data quality problems caused by excessively high or low conversion complexity, ensuring that every step in the ETL process meets business needs.

[0107] Support adaptability to diverse data needs: Different business needs have different requirements for data quality. The data quality impact coefficient can provide a quantitative basis for different business scenarios, helping the system adapt to different data quality standards according to changes in business needs, ensuring that data integrity and quality can be guaranteed under changing business needs and data characteristics.

[0108] Improve data processing efficiency and accuracy: Through accurate assessment of data quality, the system can carefully optimize the frequency of data extraction and the complexity of conversion to avoid data quality degradation caused by excessive or insufficient conversion. For example, in the case of high-frequency extraction, overly simple conversion rules may lead to data quality degradation, while by optimizing complexity, data correction and standardization effects can be improved to ensure the generation of high-quality data.

[0109] Reduce data quality risks: By dynamically adjusting the conversion complexity, the system can evaluate the data quality performance at different frequencies based on the data quality impact coefficient, avoiding the risk caused by low-quality data flowing into the target system. Especially when dealing with cross-system data integration, the data quality impact coefficient can effectively control data quality and reduce the risk of errors or missing data caused by mismatched conversion complexity.

[0110] Enhance system adaptability: The data quality impact factor provides an adaptive mechanism for the ETL process, which can be dynamically adjusted based on real-time data quality feedback, avoiding data loss or errors due to improper conversion complexity settings, and improving the stability and reliability of the entire ETL system.

[0111] The specific method for obtaining the data quality influence coefficient is as follows:

[0112] Through consistency verification in the data conversion process, the data conversion deviation generated is set as the reference deviation value; the data conversion deviation with a deviation value less than the reference deviation value is set as an acceptable deviation, and the data conversion deviation with a deviation value greater than the reference deviation value is set as a potential hidden danger deviation;

[0113] Obtain all data conversion deviation values ​​generated during the data conversion process, and compare these deviation values ​​with the benchmark deviation values; establish a hidden danger deviation set for all potential hidden danger deviation values, and establish an acceptable deviation set for all acceptable deviation values;

[0114] Data analysis is performed based on the hidden danger deviation set and the acceptable deviation set, and the data quality impact coefficient is calculated.

[0115] Calculate the data quality impact coefficient. The specific calculation formula is as follows:

[0116]

[0117] In the formula, is the data quality influence coefficient, is the acceptable deviation value in the acceptable deviation set, h is the number of the acceptable deviation value in the acceptable deviation set, is the potential hidden danger deviation value in the hidden danger deviation set, k is the number of the potential hidden danger deviation value in the hidden danger deviation set, , The time period for obtaining acceptable deviation values ​​from the data source. , The time period for obtaining outliers from the data source.

[0118] In this embodiment, an ETL evaluation model is constructed based on the bp neural network, specifically:

[0119] The obtained resource occupancy impact coefficient, synchronization timeliness impact coefficient and data quality impact coefficient are used to construct a photovoltaic power station frequency regulation effect evaluation model to generate an ETL evaluation coefficient;

[0120] The specific calculation formula of the ETL evaluation coefficient is as follows:

[0121]

[0122] In the formula, is the ETL evaluation coefficient, is the preset proportional coefficient of the resource occupancy impact coefficient. is the preset proportional coefficient of the synchronization timeliness influence coefficient, is the preset proportional coefficient of the data quality impact coefficient, is the resource occupation influence coefficient, is the synchronization timeliness influence coefficient, is the data quality impact coefficient.

[0123] This embodiment improves the efficiency and quality of the ETL (extraction, transformation, loading) process by finely managing and optimizing the frequency of contract data extraction and the complexity of data conversion. First, by analyzing the change rules and historical data of contract data, reasonable data extraction frequency constraints are constructed to ensure that the data extraction frequency matches the system load capacity and avoid excessive pressure on system resources. Secondly, a contract data conversion rule library is constructed, and by quantifying the complexity and data type of the conversion rules, the complexity of the data conversion process is flexibly adjusted to ensure that the balance between real-time requirements and data processing requirements is met in different business scenarios. Finally, by introducing the resource occupancy influence coefficient, the synchronization timeliness influence coefficient and the conversion quality influence coefficient, the data extraction frequency and conversion complexity are dynamically optimized, the real-time, controllable and flexible nature of the ETL process are improved, and the problems of uneven resource allocation and low data processing efficiency are effectively solved. This solution can significantly improve the comprehensive effect of the data processing process, meet diversified business needs, and ensure that data extraction and conversion are carried out efficiently and accurately.

[0124] S5, correspond the output of the ETL evaluation model and the transformation array and build a three-dimensional effect display model, perform surface analysis on the three-dimensional effect display model, obtain the selected transformation array, and apply it to contract automation management and control.

[0125] In this embodiment, the output of the ETL evaluation model and the transformation array are matched and a three-dimensional effect display model is constructed, specifically:

[0126] Construct a three-dimensional coordinate system to show the impact of different transformation arrays on the overall effect in the ETL process;

[0127] The three dimensions of the three-dimensional coordinate represent the frequency of data extraction, the complexity of data conversion, and the ETL evaluation coefficient;

[0128] According to the output results of the ETL evaluation model, the evaluation results of each transformation array are mapped into a three-dimensional space with its corresponding data extraction frequency and data conversion complexity combination;

[0129] Each transformation array corresponds to a point, and the position of the point is determined by the output of the ETL evaluation model and the transformation array;

[0130] A surface fitting algorithm is used to draw a surface composed of points in a three-dimensional coordinate system to obtain a three-dimensional effect display model.

[0131] In this embodiment, a surface analysis is performed on the three-dimensional effect display model to obtain a selected transformation array, which is applied to contract automation management and control, specifically:

[0132] The peak value of the three-dimensional effect display model is determined, and the peak value is used as a benchmark to expand and obtain the selected area according to the preset control range.

[0133] According to business needs, the candidate areas in the surface are weighed. If the real-time requirement is high, the conversion array corresponding to the peak in the area close to the high-frequency extraction in the surface is selected as the selected conversion array, giving priority to satisfying the synchronization timeliness impact coefficient;

[0134] If the data quality requirement is higher, the transformation array corresponding to the peak in the area with higher transformation complexity in the surface is selected as the selected transformation array to ensure the best data quality impact coefficient;

[0135] If the system resources are limited, the conversion array corresponding to the peak value in the full surface area is selected as the selected conversion array.

[0136] This embodiment constructs a three-dimensional effect display model to effectively correspond the output of the ETL evaluation model to the transformation array, and can intuitively display the comprehensive effect of different transformation arrays in the ETL process. The technical effect of this method is to visualize the relationship between data extraction frequency, data conversion complexity and ETL evaluation coefficient through a three-dimensional coordinate system, so that the evaluation and selection of the most suitable transformation array becomes more accurate and intuitive. Through surface analysis, the peak area of ​​the system can be identified, and the optimal transformation array can be flexibly selected according to specific business needs: if real-time is emphasized, the area with higher frequency is given priority; if data quality is emphasized, the area with higher conversion complexity is selected; if faced with system resource limitations, the most suitable transformation array is selected from the full surface area. This technical solution can effectively improve the accuracy of contract automation management and control, optimize the execution efficiency of the ETL process, and dynamically adjust resource allocation according to actual needs to ensure that data processing meets real-time and quality requirements, and improve the overall efficiency and flexibility of the system.

[0137] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0138] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments may be implemented in whole or in part in the form of a computer program product.

[0139] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0140] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0141] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0142] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A contract automation management and control method, characterized in that: The steps include: Construct constraints on data extraction frequency based on the maximum change frequency of contract data and the load capacity of the data middle platform architecture, and obtain the initial data extraction frequency based on historical data; Build a contract data conversion rule library and quantify the initial data conversion complexity based on the algorithm complexity, data type, and connection depth of the conversion rules; Perform several random perturbations on the initial data extraction frequency and the initial data conversion complexity, and randomly arrange and combine them into several conversion arrays; Obtain contract data from several dispersed source systems, and apply transformation arrays to perform ETL respectively, obtain resource usage data, latency data, and transformation quality data of the ETL process, and build an ETL evaluation model; The output of the ETL evaluation model is matched with the transformation array and a three-dimensional effect display model is constructed. The three-dimensional effect display model is subjected to surface analysis to obtain the selected transformation array, which is then applied to contract automation management and control.

2. A contract automation management and control method according to claim 1, characterized in that: The constraints on data extraction frequency are constructed based on the maximum change frequency of contract data and the load capacity of the data middle platform architecture. Specifically: Analyze the changing patterns of historical contract data, count the update frequency of contract data, test the processing capabilities of the data center architecture at different concurrent extraction frequencies, record the maximum data processing volume, and determine the maximum extraction frequency supported by the system based on the bandwidth, storage, and computing resources of the data center; The frequency of data extraction needs to meet the following constraints: In the formula, is the minimum update frequency of contract data, is the data extraction frequency, is the extraction frequency determined according to the maximum change frequency, The maximum extraction frequency allowed by the load capacity of the data center architecture; Get the initial data extraction frequency based on historical data, specifically: Obtain historical contract data within a preset historical period from each source system. The historical contract data includes generation timestamps and change records, classifies and records different types of contract data changes, and calculates the change frequency of each change type; According to business requirements, select the inverse of the minimum change interval of the change type closest to the requirements as the initial data extraction frequency.

3. A contract automation management and control method according to claim 2, characterized in that: The initial data conversion complexity is quantified based on the algorithm complexity, data type and connection depth of the conversion rule, specifically: Calculate the time complexity and space complexity of each conversion rule, and sum them up as the algorithm complexity of the conversion rule; Data types include structured data, semi-structured data, and unstructured data; adaptation factors are assigned to different types of data; The default adaptation factor for structured data is 1; the default adaptation factor for semi-structured data is 2; the default adaptation factor for unstructured data is 3; Obtain the number of join tables involved in each transformation rule, and use the number of join tables as the quantitative result of the join depth; The algorithm complexity, adaptation factor and connection depth of the rules in the preset standard ETL transformation process are summed up as the initial data conversion complexity.

4. A contract automation management and control method according to claim 3, characterized in that: The initial data extraction frequency and the initial data conversion complexity are randomly perturbed several times, and randomly arranged and combined into several conversion arrays, specifically: Taking the initial data extraction frequency as the benchmark, and setting the upper and lower limits as the disturbance range according to the data extraction frequency constraints; Based on the initial data conversion complexity as the benchmark, the data conversion complexity of the maximum rule is set as the upper limit, and 0 is the lower limit as the perturbation range; Within the perturbation range, several data extraction frequencies and data conversion complexities are randomly generated; The generated data extraction frequency and data conversion complexity are randomly arranged and combined to form several transformation arrays.

5. A contract automation management and control method according to claim 4, characterized in that: Apply the transformation array for ETL, specifically: The data extraction frequency in the transformation array is used as the data extraction frequency of ETL; According to the data conversion complexity in the conversion array, the ETL process is adjusted by increasing or decreasing the number of conversion rules, adjusting the data processing granularity, controlling the data transformation method, and adjusting parallel processing and single-threaded processing, so that the data conversion complexity is applied to the ETL conversion process.

6. A contract automation management and control method according to claim 5, characterized in that: The resource occupancy data includes a resource occupancy influence coefficient, and a specific method for obtaining the resource occupancy influence coefficient is as follows: Obtain the system resource usage for several time periods at different data extraction frequencies during the ETL process; Calculate the standard deviation and average value of system resource usage within the same time interval; Calculate the coefficient of variation of resource utilization based on the standard deviation and average value of system resource utilization; The resource occupancy rate variation coefficient is calculated based on a preset resource occupancy influence coefficient calculation formula to calculate the resource occupancy influence coefficient.

7. A contract automation management and control method according to claim 6, characterized in that: The delay data includes a synchronization timeliness influence coefficient, and a specific method for obtaining the synchronization timeliness influence coefficient is as follows: At different data extraction frequencies, the processing time required from initiating a data extraction request to completing data loading and generating conversion results is obtained as the response time; For the same data extraction frequency, perform several response time calculations and calculate the average response time; Obtain the data processing volume and total time of the system at different data extraction frequencies, and calculate the system data processing rate; Set the initial timeliness reduction effect model based on the average response time and system data processing rate; Set several response time points to perform system performance testing, and collect timeliness score data of data processing results at each response time point; According to the experience rating data, the undetermined parameters of the initial timeliness decreasing effect model are optimized based on the nonlinear least squares method to obtain a calculation model for the synchronous timeliness influence coefficient. The undetermined parameters include the maximum value of the timeliness score, the rate of decrease of the score with increasing response time, and the sensitivity coefficient of response time to the decreasing score.

8. A contract automation management and control method according to claim 7, characterized in that: The conversion quality data includes a data quality impact coefficient, and a specific method for obtaining the data quality impact coefficient is as follows; Through consistency verification in the data conversion process, the data conversion deviation generated is set as the reference deviation value; the data conversion deviation with a deviation value less than the reference deviation value is set as an acceptable deviation, and the data conversion deviation with a deviation value greater than the reference deviation value is set as a potential hidden danger deviation; Obtain all data conversion deviation values ​​generated during the data conversion process, and compare these deviation values ​​with the benchmark deviation values; establish a hidden danger deviation set for all potential hidden danger deviation values, and establish an acceptable deviation set for all acceptable deviation values; Data analysis is performed based on the hidden danger deviation set and the acceptable deviation set, and the data quality impact coefficient is calculated.

9. A contract automation management and control method according to claim 8, characterized in that: The output of the ETL evaluation model and the transformation array are matched and a three-dimensional effect display model is constructed, specifically: Construct a three-dimensional coordinate system to show the impact of different transformation arrays on the overall effect in the ETL process; The three dimensions of the three-dimensional coordinate represent the frequency of data extraction, the complexity of data conversion, and the ETL evaluation coefficient; According to the output results of the ETL evaluation model, the evaluation results of each transformation array are mapped into a three-dimensional space with its corresponding data extraction frequency and data conversion complexity; Each transformation array corresponds to a point, and the position of the point is determined by the output of the ETL evaluation model and the transformation array; A surface fitting algorithm is used to draw a surface composed of points in a three-dimensional coordinate system to obtain a three-dimensional effect display model.

10. A contract automation management and control method according to claim 9, characterized in that: Perform surface analysis on the 3D effect display model to obtain the selected transformation array, which is applied to contract automation management and control, specifically: Determine the peak value of the three-dimensional effect display model, and expand it based on the peak value according to the preset control range to obtain the selected area; According to business needs, the candidate areas in the surface are weighed. If the real-time requirement is high, the conversion array corresponding to the peak in the area close to the high-frequency extraction in the surface is selected as the selected conversion array, giving priority to satisfying the synchronization timeliness impact coefficient; If the data quality requirement is higher, the transformation array corresponding to the peak in the area with higher transformation complexity in the surface is selected as the selected transformation array to ensure the best data quality impact coefficient; If the system resources are limited, the conversion array corresponding to the peak value in the full surface area is selected as the selected conversion array.

Images