Data Asset Management and Control Platform Based on Multivariate Heterogeneous Fusion Technology

Abstract

The present application discloses a data asset management and control platform based on a multi - heterogeneous fusion technology, which relates to the technical field of data asset management and control. It solves the technical problem that when OLTP and OLAP share computer resources in the prior art, they may preempt computer resources from each other and affect the normal execution of tasks. The present application performs redundant calculation based on the total resource occupancy of the OLTP unit and the OLAP unit, and configures computer resources according to the calculation results. After merging the computer resources of the OLTP unit and the OLAP unit, the present application plans resource allocation rules according to the task processing time to avoid resource preemption between the two. When allocating computer resources based on the average value of the allocation ratio, if the difference between the average values of the allocation ratios in adjacent time periods is less than the difference threshold, the standard value of the average value of the allocation ratio in the adjacent time period is used instead. This solution ensures the smooth process of scheduling computer resources based on the resource scheduling sequence and avoids increasing system pressure due to frequent adjustment.

Description

Technical Field

[0001] This application belongs to the field of data asset management and control, and relates to heterogeneous fusion technology of multi-source data. Specifically, it is a data asset management and control platform based on multi-source heterogeneous fusion technology. Background Art

[0002] To solve the problem of information silos, the current common solution is to process the originally independent business data through ETL (Extract, Transform, Load), data governance, etc. and put it into the data warehouse, thus forming two sets of data for transactional online transaction processing OLTP and analytical online real-time analysis OLAP. This approach solves the problem of data silos to a certain extent, but there are also many drawbacks. For example, the timeliness is low. Generally, the data in the data warehouse needs to go through cleaning and transformation, etc., with a timeliness difference of T+N. In many cases, serious consequences will be caused due to timeliness issues.

[0003] The invention patent application with the publication number CN109635042A discloses an integrated OLTP and OLAP automotive finance big data system, which improves the utilization efficiency of computer resources by integrating the computer resources of the two projects of OLTP and OLAP. Although the existing technology solves the problem of resource waste when the OLTP and OLAP computing resources are independent by sharing computer resources, sharing computer resources will cause OLTP and OLAP to compete for computer resources, affecting the normal operation of the two projects of OLTP and OLAP.

[0004] This application provides a data asset management and control platform based on multi-source heterogeneous fusion technology to solve the above technical problems. Summary of the Invention

[0005] This application aims to solve at least one of the technical problems existing in the prior art. For this purpose, this application proposes a data asset management and control platform based on multi-source heterogeneous fusion technology, which is used to solve the technical problem that when OLTP and OLAP share computer resources in the prior art, they may compete for computer resources with each other and affect the normal execution of tasks.

[0006] To achieve the above object, the first aspect of this application provides a data asset management and control platform based on multi-source heterogeneous fusion technology, including a data processing module, as well as an intelligent terminal and a library lake connected thereto. The intelligent terminal is used to query and display data;

[0007] The data processing module includes an OLTP unit and an OLAP unit; the OLTP unit is used to process online transactions based on the data stored in the library lake and transmit the data generated during the processing to the library lake for storage; the OLAP unit is used to process analysis tasks based on the data stored in the library lake;

[0008] The data processing module is used to configure and allocate computer resources based on the historical task data of the OLTP unit and the OLAP unit to ensure that the OLTP unit and the OLAP unit can execute tasks normally.

[0009] In the existing data asset management process, one of the difficulties lies in how to reasonably allocate computer resources to ensure that both the OLTP unit and the OLAP unit can execute tasks normally while reducing the computer resource configuration cost. The existing technology configures the computer resources required by the OLTP unit and the OLAP unit uniformly without configuring computer resources separately for each unit, which can avoid waste of computer resources. However, how to allocate the computer resources after unified configuration is also a problem. Theoretically, the OLTP unit requires more computer resources during the day, and the OLAP unit requires more computer resources at night. However, it cannot be excluded that the OLTP unit needs computer resources at night and the OLAP unit needs computer resources during the day.

[0010] This application analyzes the historical task data, discovers the rules of task processing of the OLTP unit and the OLAP unit, and configures the computer resources of the entire data management platform according to the resource occupancy during task processing. Of course, if the data asset management platform is an already established platform, the computer resources can be updated according to the resource occupancy.

[0011] After configuring the computer resources, it is necessary to allocate the computer resources according to the resource amounts required by the OLTP unit and the OLAP unit in each time period, which can ensure that the computer resources required by the OLTP unit and the OLAP unit are independent and can be dynamically adjusted, and there will be no resource preemption phenomenon, ensuring the smooth execution of tasks of the OLTP unit and the OLAP unit.

[0012] Preferably, the configuring and allocating computer resources based on the historical task data of the OLTP unit and the OLAP unit includes:

[0013] Obtaining the historical task data of the OLTP unit and the OLAP unit; wherein, the historical task data includes task type, task time, and occupied resource amount;

[0014] Statistical analysis of the occupied resources in the historical task data to configure computer resources; and comprehensive analysis of the historical task data to allocate computer resources to the OLTP unit and the OLAP unit.

[0015] This application first needs to calculate the computer resources required by the OLTP unit and the OLAP unit, and then configure resources for the data asset platform according to the calculation results. In addition to the computer resources required by the OLTP unit and the OLAP unit, the computer resources of other modules in the data asset management platform also need to be configured.

[0016] By calculating the historical task data corresponding to the OLTP unit and the OLAP unit, the resource occupancy during task execution is extracted from the historical task data. Based on the resource requirements of both the OLTP unit and the OLAP unit, determine how much computer resources should be configured for the data asset management and control platform.

[0017] Preferably, the statistics of the resource occupancy in the historical task data include:

[0018] Statistically construct the resource occupancy curve corresponding to the OLTP unit, marked as the first resource curve; statistically construct the resource occupancy curve corresponding to the OLAP unit, marked as the second resource curve;

[0019] Obtain the sum of the maximum values in the first resource curve and the second resource curve, marked as the total resource occupancy; perform redundancy calculation based on the total resource occupancy, and configure computer resources according to the calculation results.

[0020] The computer resources occupied by both the OLTP unit and the OLAP unit are the main part. Then, calculate the sum of the maximum values in the two curves through the first resource curve and the second resource curve as the total resource occupancy. Appropriate redundancy design based on the total resource occupancy can obtain the computer resources required for the entire data asset management and control platform. Of course, time can also be used as an independent variable to merge the first resource curve and the second resource curve, and take the maximum value after merging as the total resource occupancy, and then complete the subsequent configuration of computer resources.

[0021] Taking the computer resources required by both the OLTP unit and the OLAP unit as the core, expand and obtain the computer resources required for the integrated data asset management and control platform. This configuration method can ensure that the computer resources can meet the resource requirements of the OLTP unit and the OLAP unit under high-load operating conditions, and there will be no shortage of resources.

[0022] Preferably, the redundancy calculation based on the total resource occupancy includes:

[0023] Extract the key performance indicators of the data asset management and control platform, and integrate the total resource occupancy with the key performance indicators into the redundancy basic data; among them, the key performance indicators include the recovery time objective and the recovery point objective;

[0024] Input the redundancy basic data into the redundancy evaluation model to obtain the redundancy coefficient; multiply the redundancy coefficient by the total resource occupancy to obtain the resource configuration amount; among them, the redundancy evaluation model is constructed based on an artificial intelligence model.

[0025] After obtaining the total amount of resource occupation in this application, corresponding redundancy design needs to be carried out, that is, the computer resources to be configured are more than the total amount of resource occupation. This redundancy design is mainly to avoid resource shortage caused by emergencies.

[0026] In this application, the required computer resources, that is, the resource configuration amount, are calculated by setting a redundancy coefficient. The redundancy coefficient is obtained through an artificial intelligence model. This redundancy coefficient is mainly related to the key performance indicators of the data asset management platform and is also related to the total amount of resource occupation corresponding to the OLTP unit and the OLAP unit. Through the artificial intelligence model, the acquisition efficiency of the redundancy coefficient can be guaranteed to be faster and the accuracy to be higher.

[0027] Preferably, allocating computer resources to the OLTP unit and the OLAP unit includes:

[0028] Dividing the occupied resources of the OLTP unit and the OLAP unit into several data pairs according to a set period; wherein, each data pair includes the resource occupation amounts of the OLTP unit and the OLAP unit within the set period.

[0029] Calculating the ratio of the resource occupation amounts of the OLTP unit and the OLAP unit in the data pair, which is marked as the allocation ratio; wherein, the set period includes one hour or one minute.

[0030] Obtaining the average value of the allocation ratios corresponding to each set period within the adjustment period, and allocating computer resources based on the average value of the allocation ratios; wherein, the adjustment period includes one day or one week.

[0031] Since the task types processed by the OLTP unit and the OLAP unit are different, the computer resources occupied by the two will change over time. If no targeted resource allocation and planning are carried out for the OLTP unit and the OLAP unit, there may be a situation where: if the OLAP suddenly receives an online analysis task with a higher priority during the day, it will then preempt more computer resources and affect the task execution efficiency of the OLTP.

[0032] In this application, the allocation ratio for each time period is determined by analyzing the resource occupation amounts of the OLTP unit and the OLAP unit in each time period. The computer resources are divided according to this allocation ratio and associated with the OLTP unit and the OLAP unit. Then, when the OLTP unit and the OLAP unit execute tasks subsequently, they can only utilize the computer resources associated with them, and there will be no situation of resource preemption.

[0033] Preferably, the allocating computer resources based on the average value of the allocation ratios includes:

[0034] Allocating the resource configuration amount to the time periods corresponding to the set period according to the average value of the allocation ratios to obtain a resource scheduling sequence.

[0035] Dynamically adjust computer resources according to the resource scheduling sequence; the resource allocation amount is the total amount of computer resources.

[0036] Preferably, the allocation of computer resources based on the average allocation ratio includes:

[0037] Calculate whether the difference between the average allocation ratios of adjacent time periods is less than the difference threshold; if so, replace it with the standard value of the average allocation ratio within this adjacent time period; if not, do not modify the average allocation ratio; where the standard value is any one of the average allocation ratios corresponding to this adjacent time period, or the average of the two.

[0038] Allocate the resource allocation amount to the time periods corresponding to the set cycle according to the average allocation ratio or the standard value to obtain a resource scheduling sequence; dynamically adjust computer resources according to the resource scheduling sequence; the resource allocation amount is the total amount of computer resources.

[0039] When allocating resources for each time period through the average allocation ratio, there may be a problem of frequent adjustment due to little change in resource allocation between adjacent time periods. This application performs smoothing processing by analyzing the average allocation ratios of adjacent time periods, improves the fluency when adjusting computer resources according to the resource scheduling sequence, and avoids the increase in cost caused by frequent adjustment.

[0040] Compared with the prior art, the beneficial effects of this application are:

[0041] 1. This application obtains the historical task data of the OLTP unit and the OLAP unit; counts the occupied resources in the historical task data to obtain the total resource occupancy; performs redundancy calculation based on the total resource occupancy, and configures computer resources according to the calculation results; after merging the computer resources of the OLTP unit and the OLAP unit, this application plans the resource allocation rules according to the task processing time, avoiding resource contention between the two.

[0042] 2. When this application allocates computer resources based on the average allocation ratio, if the difference between the average allocation ratios of adjacent time periods is less than the difference threshold, then replace it with the standard value of the average allocation ratio within this adjacent time period; this solution ensures the smooth process of scheduling computer resources based on the resource scheduling sequence and avoids frequent adjustment from increasing the system pressure. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0044] Figure 1 Schematic diagram of the system principle of the embodiment of the present application;

[0045] Figure 2 Schematic diagram of the process for allocating computer resources based on historical task data in the embodiment of the present application;

[0046] Figure 3 Schematic diagram of the calculation process for the total amount of resource occupancy in the embodiment of the present application;

[0047] Figure 4 Schematic diagram of the calculation process for the allocation ratio in the embodiment of the present application. Detailed implementation manners

[0048] The technical solutions of the present application will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0049] Please refer to Figure 1 - Figure 2 , the data asset management and control platform based on the multi-source heterogeneous fusion technology provided by the first aspect embodiment of the present application includes a data processing module, as well as an intelligent terminal and a library lake connected thereto. The intelligent terminal is used to query and display data;

[0050] The data processing module includes an OLTP unit and an OLAP unit; the OLTP unit is used to process online transactions based on the data stored in the library lake and transmit the data generated during the processing to the library lake for storage; the OLAP unit is used to process analysis tasks based on the data stored in the library lake;

[0051] The data processing module is used to configure and allocate computer resources based on the historical task data of the OLTP unit and the OLAP unit to ensure the normal execution of tasks by the OLTP unit and the OLAP unit.

[0052] The data processing module in this example mainly includes an OLTP unit and an OLAP unit, and is responsible for data collection and analysis of the data asset management and control platform, as well as the execution of various online transactions and online analysis tasks. The library lake is used to store various types of data related to data assets, and the stored data includes structured data and unstructured data. The intelligent terminal is mainly responsible for data display, task formulation, etc.

[0053] In this embodiment, the OLTP unit can directly extract data from the data warehouse lake for online transaction processing, and after the processing is completed, transmit the data generated during the processing to the data warehouse lake for storage. The OLAP unit can extract data from the data warehouse lake for tasks such as analysis and mining. Moreover, since the data generated during the task execution of the OLTP unit is stored in the data warehouse lake, the OLTP unit can directly extract useful data without having to reprocess it, which can improve the data processing efficiency. This cannot be achieved by setting up separate databases for the OLTP unit and the OLAP unit in the prior art. This method not only requires the OLAP unit to reprocess the data, but also may require data migration, which affects the efficiency.

[0054] The computer resources in this embodiment include hardware resources and software resources, such as processors, memory, storage devices, network resources, etc. When planning computer resources, for software resources, they can be configured according to the resource configuration amount, but the hardware resources cannot be lower than the requirements in the resource configuration amount. For example, if 5.1 1TB hard disks are required in the resource configuration amount, then 6 1TB hard disks, or integer numbers of hard disks of other capacities, need to be actually configured.

[0055] In a preferred embodiment, computer resources can be configured and allocated according to the historical task data of the OLTP unit and the OLAP unit, including:

[0056] Obtain the historical task data of the OLTP unit and the OLAP unit; count the occupied resources in the historical task data and configure computer resources; and comprehensively analyze the historical task data to allocate computer resources to the OLTP unit and the OLAP unit.

[0057] The historical task data in this embodiment includes task type, task time, and occupied resource amount, and is obtained by recording the task processing processes of the OLTP unit and the OLAP unit. For a newly built data asset management and control platform, the historical task data can be replaced by the historical task data of a similar platform. The task type is used to distinguish whether the task belongs to the OLTP unit or the OLAP unit, while the task time and occupied resource amount are used for subsequent computer resource allocation.

[0058] After obtaining the computer resources required by the OLTP unit or the OLAP unit through statistical analysis of the historical task data, the resource configuration amount can be determined from the following two aspects.

[0059] Please refer to Figure 3 , on the one hand, count the occupied resources in the historical task data and configure computer resources, including:

[0060] Statistically construct the resource occupancy curve corresponding to the OLTP unit and mark it as the first resource curve; statistically construct the resource occupancy curve corresponding to the OLAP unit and mark it as the second resource curve;

[0061] Obtain the sum of the maximum values in the first resource curve and the second resource curve and mark it as the total resource occupancy; perform redundancy calculation based on the total resource occupancy and configure computer resources according to the calculation results.

[0062] Taking the task time as the independent variable and the occupied resource amount as the dependent variable, the first resource curve corresponding to the OLTP unit and the second resource curve corresponding to the OLAP unit can be constructed based on historical task data. Both the first resource curve and the second resource curve are used to express the change of the occupied resource amount over time.

[0063] The maximum values of the occupied resource amounts in the first resource curve and the second resource curve can be extracted respectively, and the sum of the maximum values is marked as the total resource occupancy; perform redundancy design based on this total resource occupancy to obtain the computer resources that the data asset management and control platform needs to configure.

[0064] Of course, the first resource curve and the second resource curve can also be merged according to the task time to obtain a new resource curve. Mark the maximum value in the new resource curve as the total resource occupancy. Redundancy design can also be performed based on this total resource occupancy to obtain the computer resources that need to be configured.

[0065] It should be noted that the total resource occupancy in this embodiment is only calculated through the historical task data corresponding to the OLTP unit and the OLAP unit. However, in the actual data asset management and control platform, there are other modules or units that need to occupy a certain amount of computer resources to run in addition to the OLTP unit and the OLAP unit. The computer resources required by other modules or units can be calculated according to a fixed value, and the result after redundancy calculation is added to this fixed value to obtain the computer resources that need to be configured. Or the fixed value can be added to the total resource occupancy to obtain a new total resource occupancy, and redundancy design is performed for the new total resource occupancy.

[0066] After calculating the total resource occupancy, appropriate redundancy design is still required. In this embodiment, redundancy calculation is performed based on the total resource occupancy, including:

[0067] Extract the key performance indicators of the data asset management and control platform and integrate the total resource occupancy and the key performance indicators into redundant basic data; among them, the key performance indicators include the recovery time objective and the recovery point objective;

[0068] Input the redundant basic data into the redundancy evaluation model to obtain a redundancy coefficient; multiply the redundancy coefficient by the total resource occupancy to obtain the resource allocation amount; among them, the redundancy evaluation model is constructed based on an artificial intelligence model.

[0069] In this embodiment, redundancy design is carried out by obtaining a redundancy coefficient. Specifically, key performance indicators closely related to computer resources in the data asset management and control platform are obtained, such as the Recovery Time Objective (RTO) and the Recovery Point Objective (RPO). The Recovery Time Objective (RTO) is an important indicator to measure the disaster tolerance ability of an information system. It refers to the maximum time required for the system to resume normal operation after a failure or disaster. RTO reflects the organization's tolerance for service interruption and is a key component of Business Continuity Planning (BCP). The Recovery Point Objective (RPO) is an indicator to measure data recovery ability. It defines the maximum tolerable time for data loss when a system failure or disaster occurs. RPO focuses on how much data state can be restored to the time before the failure during the recovery process.

[0070] Integrate these key performance indicators with the total resource occupancy calculated previously and input them into the redundancy evaluation model to obtain the corresponding redundancy coefficient. Multiply the redundancy coefficient by the total resource occupancy to obtain the required resource configuration. It should be noted that the finally calculated total resource configuration may not be directly applicable to resource allocation and necessary rounding processing needs to be carried out in combination with the actual situation; for example, if it is calculated that 9G of RAM is required, but the actual specification is only 2G, then 5 pieces of 2G RAM are needed to meet the requirements.

[0071] The redundancy evaluation model in this embodiment is constructed based on an artificial intelligence model. The specific construction process is as follows:

[0072] Obtain standard training data; among them, the standard training data includes model input data and model output data. The model input data includes key performance indicators and the calculated total resource occupancy, and the model output data is the redundancy coefficient corresponding to the model input data;

[0073] Construct an artificial intelligence model; train the artificial intelligence model with the model input data and model output data to obtain a redundancy evaluation model.

[0074] The standard training data is reliable data extracted from a platform similar to the data asset management and control platform in this application. These data are from the key performance indicators of these similar platforms and the calculated total resource occupancy, and the redundancy coefficient is calculated based on the total amount of computer resources that can ensure the smooth operation of the platform according to the actual configuration. Moreover, the standard training data will be continuously updated, and the redundancy evaluation model will also be continuously updated.

[0075] Please refer to Figure 4, after calculating the total resource occupancy, it is necessary to allocate it. This embodiment allocates computer resources to the OLTP unit and the OLAP unit, including:

[0076] Dividing the occupied resources of the OLTP unit and the OLAP unit into several data pairs according to a set period; where each data pair includes the resource occupancy of the OLTP unit and the OLAP unit within the set period;

[0077] Calculating the ratio of the resource occupancy of the OLTP unit and the OLAP unit in the data pair, marked as the allocation ratio; where the set period includes one hour or one minute;

[0078] Obtaining the average value of the allocation ratios for the corresponding time periods of each set period within the adjustment period, and allocating computer resources based on the average value of the allocation ratios; where the adjustment period includes one day or one week.

[0079] In this embodiment, first, the occupied resources of the OLTP unit and the OLAP unit in the historical task data are divided into several data pairs according to the set period, and each data pair includes the resources occupied by the OLTP unit and the OLAP unit within the corresponding set period, so that the ratio of the resources occupied by the OLTP unit and the OLAP unit within each set period can be calculated, marked as the allocation ratio. Then, based on the adjustment period, calculate the average value of the allocation ratios for the corresponding time periods of each set period, and the computer resources (total resource occupancy) within each time period can be allocated according to the average value of the allocation ratios.

[0080] The adjustment period in this embodiment is greater than the set period. The set period includes one hour, one minute, or even one minute, and the adjustment period includes one week, one day, or one hour. Examples of allocating computer resources based on the adjustment period and the set period are as follows:

[0081] Dividing the occupied resources of the OLTP unit and the OLAP unit into several data pairs every hour, and the duration of each time period corresponding to each data pair is one hour, and one day can be divided into twenty-four hours.

[0082] Each data pair includes the resource occupancy of the OLTP unit within the corresponding time period and also includes the resource occupancy of the OLAP unit within the corresponding time period; taking the ratio of the resource occupancies of the two as the allocation ratio, that is, each time period corresponds to an allocation ratio.

[0083] Determine the adjustment period as one day, but the historical task data is data for many days, that is, each set period corresponds to multiple allocation ratios. For example, the time period from 8:00 to 9:00 every day will correspond to many allocation ratios, and calculate the average value of these allocation ratios as the average value of the allocation ratios for this time period.

[0084] Next, computer resources can be allocated according to the corresponding average allocation ratio within each time period. That is, an allocation rule can be determined every hour within a day, and computer resources can be allocated according to this allocation rule to ensure the efficient operation of the OLTP unit and the OLAP unit.

[0085] Theoretically, the smaller the set period, the more accurate the allocation ratio, and the more reasonable the subsequent computer resource allocation. However, the smaller the set period, the higher the frequency of computer resource allocation within the adjustment period, which will increase the resource allocation pressure. In another preferred embodiment, the time for resource allocation is determined by comparing the magnitudes of the average allocation ratios of adjacent time periods, including:

[0086] Calculating whether the difference between the average allocation ratios of adjacent time periods is less than the difference threshold; if yes, replacing it with the standard value of the average allocation ratio within this adjacent time period; if no, not modifying the average allocation ratio; where the standard value is any one of the average allocation ratios corresponding to this adjacent time period, or the average of the two;

[0087] Allocating the resource configuration amount to the time periods corresponding to the set period according to the average allocation ratio or the standard value to obtain a resource scheduling sequence; dynamically adjusting computer resources according to the resource scheduling sequence; the resource configuration amount is the total amount of computer resources. It should be noted that the above difference threshold is set according to experience.

[0088] When the difference between the average allocation ratios of adjacent time periods within the adjustment period is small, the average of the average allocation ratios can be used as the new average allocation ratio and associated with the corresponding time period; or any one of the two average allocation ratios can be used as the new average allocation ratio, and this new average allocation ratio is associated with this adjacent time period. The reason for this processing to be feasible lies in the redundant design of the resource configuration amount.

[0089] Sort the average allocation ratios of each set period after readjustment into a resource scheduling sequence within the adjustment period range, that is, the allocation of computer resources should be adjusted according to the corresponding average allocation ratio in each set period. This can reduce the adjustment frequency as much as possible within the adjustment period and reduce resource waste during the adjustment process.

[0090] Some of the data in the above formula are calculated by removing the dimension and taking their numerical values. The formula is obtained by software simulation of a large amount of collected data to get a formula closest to the real situation; the preset parameters and preset thresholds in the formula are set by those skilled in the art according to the actual situation or obtained through simulation of a large amount of data.

[0091] The above embodiments are only used to illustrate the technical method of the present application and not to limit it. Although the present application has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical method of the present application can be modified or equivalently replaced without departing from the spirit and scope of the technical method of the present application.

Claims

1. A data asset management and control platform based on multi-heterogeneous fusion technology, including a data processing module, and an intelligent terminal and a warehouse lake connected thereto, wherein the intelligent terminal is used to query and display data; characterized in that: The data processing module includes an OLTP unit and an OLAP unit; the OLTP unit is used to process online transactions based on the data stored in the warehouse lake, and transmit the data generated during the processing to the warehouse lake for storage; The OLAP unit is used for data processing and analysis tasks based on the storage in the warehouse lake; The data processing module is used to configure and allocate computer resources based on the historical task data of the OLTP unit and the OLAP unit to ensure that the OLTP unit and the OLAP unit perform tasks normally; The configuration and allocation of computer resources based on the historical task data of the OLTP unit and the OLAP unit includes: Obtain historical task data of OLTP units and OLAP units; Collect statistics on occupied resources in historical task data and configure computer resources; The counting of the occupied resources in the historical task data includes: The resource occupancy curve corresponding to the OLTP unit is constructed by statistics, which is marked as the first resource curve; the resource occupancy curve corresponding to the OLAP unit is constructed by statistics, which is marked as the second resource curve; Obtaining the sum of the maximum values ​​in the first resource curve and the second resource curve, marking it as the total resource occupation; performing redundancy calculation based on the total resource occupation, and configuring computer resources according to the calculation result; The redundancy calculation based on the total amount of resource usage includes: Extract the key performance indicators of the data asset management and control platform, and integrate the total resource usage and key performance indicators into redundant basic data; among them, the key performance indicators include recovery time objectives and recovery point objectives; Input the redundant basic data into the redundant evaluation model to obtain the redundant coefficient; multiply the redundant coefficient by the total resource occupancy to obtain the resource configuration amount; wherein, the redundant evaluation model is constructed based on the artificial intelligence model.

2. The data asset management and control platform based on multi-heterogeneous fusion technology according to claim 1 is characterized in that: The data processing module is used to configure and allocate computer resources based on the historical task data of the OLTP unit and the OLAP unit, including: Divide the occupied resources of the OLTP unit and the OLAP unit into a number of data pairs according to a set period; wherein each data pair includes the resource occupied amount of the OLTP unit and the OLAP unit in the set period; Calculate the ratio of the resource usage of the OLTP unit and the OLAP unit in the data pair, marked as an allocation ratio; wherein the setting period includes one hour or one minute; The average value of the allocation ratio in the time period corresponding to each set period is obtained within the adjustment period, and the computer resources are allocated based on the average value of the allocation ratio; wherein the adjustment period includes one day or one week.

3. The data asset management and control platform based on multi-heterogeneous fusion technology according to claim 2 is characterized in that: The method of allocating computer resources based on the allocation ratio mean value includes: Allocate the resource configuration amount to the time period corresponding to the set period according to the average allocation ratio to obtain a resource scheduling sequence; Dynamically adjust computer resources according to the resource scheduling sequence; the resource allocation amount is the total amount of computer resources.

4. The data asset management and control platform based on multi-heterogeneous fusion technology according to claim 2 is characterized in that: The method of allocating computer resources based on the allocation ratio mean value includes: Calculate whether the difference between the distribution ratio means of adjacent time periods is less than the difference threshold; if yes, replace the distribution ratio mean standard value in the adjacent time period; if no, do not modify the distribution ratio mean; wherein the standard value is any one of the corresponding distribution ratio means of the adjacent time period, or the mean of the two; Allocate resource configuration amounts to time periods corresponding to a set period according to a mean value or a standard value of allocation ratios to obtain a resource scheduling sequence; dynamically adjust computer resources according to the resource scheduling sequence; and the resource configuration amount is the total amount of computer resources.

Images