Method, device, medium and product for regulating a data governance system

Abstract

The application discloses a data governance system adjusting method, device, medium and product, relates to the technical field of big data, and in the application, each function node in the data governance system monitors the input rate of the received task data of the function node, and in the case that the input rate exceeds the rate threshold corresponding to the function node, triggers the sending of a corresponding back pressure signal to the related node to reduce the data sending rate of the related node, thereby reducing the load of the function node and avoiding the high load state of the function node. It can be understood that since each function node in the embodiment of the application can adaptively send a pressure signal to the related node, the adjusting granularity of the data governance system is improved, the flexibility of the adjustment is improved, and therefore even in the face of a sudden increase in data volume, the problem can be well solved, thereby ensuring the robustness of the data governance system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, and particularly relates to a data governance system adjustment method and device, a medium and a product. BACKGROUND

[0002] With the rapid development of computer technology, the production rate of data in the network is getting faster and faster, and accordingly, the timeliness requirement for data processing is getting higher and higher. Therefore, the existing scheme provides a back pressure mechanism of a streaming engine, which automatically adjusts the data governance operator rate to match the processing capacity of the downstream operator by monitoring the operator queue filling state of the processing node. However, in actual application, the existing back pressure mechanism of the streaming engine has poor flexibility, and cannot respond in time when facing a sudden increase in data volume, which can easily cause the entire data governance system to fail to cope with the sudden increase in data volume, that is, the robustness of the data governance system is low.

[0003] The above content is only used to assist in understanding the technical solutions of the present application and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0004] The main purpose of the present application is to provide a data governance system adjustment method, device, medium and product, which aims to solve the technical problem of poor flexibility of the existing back pressure mechanism of the streaming engine.

[0005] To achieve the above purpose, the present application provides a data governance system adjustment method, which is applied to any functional node in a data governance system, and the data governance system adjustment method comprises the following steps:

[0006] Monitoring the total amount of data processed by the functional node in a preset time window for a data governance task;

[0007] Determining the input rate of the functional node based on the total amount of data and the window length of the preset time window;

[0008] In the case that the input rate exceeds the rate threshold value corresponding to the functional node, a first back pressure signal is sent to the related node of the functional node, wherein the first back pressure signal is used to reduce the data sending rate of the related node.

[0009] Optionally, the data governance method further comprises:

[0010] After receiving the second back pressure signal sent by the related node, the pressure level of the related node is determined based on the second back pressure signal, and the rate reduction coefficient is determined based on the pressure level, wherein the greater the pressure level, the smaller the rate reduction coefficient.

[0011] According to a current data sending rate of sending data to the related node and the rate reduction coefficient, a target data sending rate of sending data to the related node is calculated, and data is sent to the related node based on the target data sending rate, wherein the target sending rate is less than the current data sending rate.

[0012] Optionally, the adjusting method of the data governance system further comprises:

[0013] The total number of tasks and the task processing interval of the functional node in processing data governance tasks in a preset time window are monitored, and the task processing rate of the functional node is determined based on the total number of tasks and the task processing interval;

[0014] The node stress degree of the functional node is determined based on the node computing resource usage rate of the functional node and the task processing rate;

[0015] In the case that the functional node is determined to be overloaded through the node stress degree, the data governance tasks of the functional node are transferred to other target nodes, wherein the other target nodes are other functional nodes that are not in an overloaded state.

[0016] Optionally, in the case that the functional node is a resource scheduling node, the adjusting method of the data governance system further comprises:

[0017] For any one data governance task in the data governance system, the computing resource allocation amount of the data governance task is determined based on the priority proportion of the data governance task and the current total available computing resource of the data governance system, wherein the priority proportion of the data governance task is the priority weight of the data governance task in the proportion of the total priority weight of each data governance task in the data governance system;

[0018] The computing resource of the functional node for executing the data governance task is allocated according to the computing resource allocation amount.

[0019] Optionally, the adjusting method of the data governance system further comprises:

[0020] The current total available computing resource in the data governance system is monitored;

[0021] In the case that the data governance system is determined to be in a resource shortage state through the current total available computing resource, the priority weight of a low-priority data governance task is reduced, or the execution of the low-priority data governance task is suspended.

[0022] Optionally, the step of reducing the priority weight of a low-priority data governance task or suspending the execution of the low-priority data governance task comprises:

[0023] differentiate the priority weights of the data governance tasks in the data governance system to obtain new priority weights of the data governance tasks, and return, based on the priority weights, the step of determining the amount of computing resources allocated to the data governance tasks based on the priority proportion of the data governance tasks and the current total available computing resources of the data governance system; or

[0024] adding the low-priority data governance task to the waiting queue, wherein the data governance task added to the waiting queue will be suspended and the computing resources corresponding to the task in the waiting queue will be suspended.

[0025] Optionally, the adjustment method of the data governance system further comprises:

[0026] monitoring utilization information of the total available computing resources of the data governance system in a preset time window and load information of each functional node in the data governance system in the preset time window;

[0027] inputting the utilization information and the load information into a pre-trained prediction model to predict a computing resource demand estimate of the data governance system in a future preset time period through the pre-trained prediction model;

[0028] using the computing resource demand estimate as an adjustment target to adjust the current total available computing resources of the data governance system.

[0029] In addition, to achieve the above-mentioned purposes, the present application also provides an electronic device, which comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the adjustment method of the data governance system as described above.

[0030] In addition, to achieve the above-mentioned purposes, the present application also provides a storage medium, which is a computer-readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the adjustment method of the data governance system as described above.

[0031] In addition, to achieve the above-mentioned purposes, the present application also provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the adjustment method of the data governance system as described above.

[0032] The one or more technical solutions proposed in the present application have at least the following technical effects:

[0033] In the embodiments of the present application, the function node in the data governance system will monitor the total amount of data processed by the function node in a preset time window for data governance tasks; determine the input rate of the function node based on the total amount of data and the window length of the preset time window; in the case where the input rate exceeds the rate threshold corresponding to the function node, a first back pressure signal is sent to the related node of the function node, wherein the first back pressure signal is used to reduce the data sending rate of the related node. That is, each function node in the data governance system will monitor the input rate of the task data received by itself, and in the case where the input rate exceeds the rate threshold corresponding to the function node, the corresponding back pressure signal is triggered to be sent to the related node to reduce the data sending rate of the related node, thereby reducing the load of itself and avoiding the state of high load. It can be understood that since each function node in the embodiments of the present application can adaptively send a pressure signal to the related node, the adjustment granularity of the data governance system is improved, the flexibility of adjustment is improved, and therefore even in the face of sudden increase in data volume, it can be well coped with without problems, thereby ensuring the robustness of the data governance system. BRIEF DESCRIPTION OF DRAWINGS

[0034] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the application.

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0036] Figure 1 Flowchart of the first embodiment of the adjustment method of the data governance system of the present application;

[0037] Figure 2 Flowchart of the second embodiment of the adjustment method of the data governance system of the present application;

[0038] Figure 3 Flowchart of the third embodiment of the adjustment method of the data governance system of the present application;

[0039] Figure 4 Flowchart of the fourth embodiment of the adjustment method of the data governance system of the present application;

[0040] Figure 5 Framework diagram of the adjustment method of the data governance system of the present application;

[0041] Figure 6 Overall flowchart of the adjustment method of the data governance system of the present application;

[0042] Figure 7 The device structure diagram of the hardware running environment involved in the adjustment method of the data governance system in the embodiments of the present application.

[0043] The purposes, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0044] It should be understood that the specific embodiments described herein are merely intended to explain the technical solutions of the present application, and are not intended to limit the present application.

[0045] In order to better understand the technical solutions of the present application, the specific embodiments will be described in detail below with reference to the drawings and the accompanying drawings.

[0046] With the rapid development of computer technology, the production rate of data in the network is getting faster and faster, and accordingly, the timeliness requirement of data processing is getting higher and higher. Therefore, the existing scheme provides a back pressure mechanism of the streaming engine, which automatically adjusts the data governance operator rate to match the processing capacity of the downstream operator by monitoring the operator queue filling state of the processing node. However, in actual application, due to the poor flexibility of the existing back pressure mechanism of the streaming engine, it cannot respond in time when facing a sudden increase in data volume, which can easily cause the entire data governance system to fail to cope with the sudden increase in data volume, that is, to make the data governance system have low robustness.

[0047] The main solution of the embodiments of the present application is: monitoring the total task quantity and the task processing interval of the function node in a preset time window, and determining the task processing rate of the function node through the total task quantity and the task processing interval; determining the node pressure degree of the function node based on the node computing resource usage rate of the function node and the task processing rate; in the case that the function node is determined to be in overload through the node pressure degree, a first back pressure signal is sent to the related node of the function node, wherein the first back pressure signal is used to reduce the data sending rate of the related node.

[0048] That is, each function node in the data governance system will detect the node pressure degree of itself and determine whether it is in overload, if it is determined to be in overload, the corresponding back pressure signal is sent to the related node, which is used to reduce the data sending rate of the related node, and then reduce the self node pressure, so as to avoid the self node being in overload. It can be understood that since each function node in the embodiments of the present application can adaptively send the pressure signal to the related node, the adjustment granularity of the data governance system is improved, the flexibility of the adjustment is improved, so even in the face of a sudden increase in data volume, it can be better coped with without problems, thereby ensuring the robustness of the data governance system.

[0049] It should be noted that the execution subject of the embodiment can be a computing service device with data processing, network communication and program running functions, such as a cloud platform, a computer, a mobile phone, etc., or an electronic device capable of realizing the above functions.

[0050] It should be noted that the embodiments of the present application mainly relate to various functional nodes in the data governance system, and the functional nodes can be data governance operators with different functions in the data governance system. For example, data filtering operators, data conversion operators, data aggregation operators, and data quality detection operators.

[0051] Regarding the data filtering operator. Data filtering is a common data governance step, and the purpose is to reduce the amount of irrelevant data entering the processing system. By deploying a listener at the data operator, real-time data capture and dynamic filtering of data that does not meet the conditions according to the pre-defined rules. In the real-time stream processing system of the CDN log data (Content Delivery Network Log) in the telecommunications scenario, it is ensured that only the most critical data is processed, thereby improving the efficiency and response speed of the entire system. Use the built-in filtering function of the stream processing tool to optimize the data before it reaches the processing pipeline. Further, the data filtering operator can include:

[0052] 1. Data value rule operator: For example, the status code field (such as HTTP (Hypertext Transfer Protocol) response status code) in the CDN log in the telecommunications scenario shows 404 or 500, which means that the webpage cannot be found or the server has an error, and such information is crucial for error analysis. The system sets a rule to only keep the log records of these status codes.

[0053] 2. Range rule operator: For example, only interested in request logs with response times exceeding a certain threshold, such as selecting records with response times exceeding 500 milliseconds to analyze and optimize potential performance bottlenecks.

[0054] 3. Pattern matching rule operator: Use regular expressions to filter specific patterns of URLs (Uniform Resource Locato) or query parameters, for example, filter access logs containing specific query strings or specific API paths to help focus on traffic and performance of specific services or pages.

[0055] 4. Data filtering operator: Use real-time stream engine APIs to apply rules before the data governance operator enters the system. Including `filter` and `map` operations, which are executed directly on the data governance operator, thereby filtering and converting the data governance operator.

[0056] In addition, custom Predicate functions can be written to apply logic to the complexity of each log record, ensuring that only records that meet certain criteria are further processed. These functions make decisions based on the detailed content of the log record, such as status codes, response times, URL patterns, and more, to determine whether to keep the record. The order of execution of the data filtering rules is optimized, with the rules that exclude the most data placed at the front to reduce the burden of subsequent processing. As data patterns change or system requirements change, regularly update and adjust the filtering rules to ensure that the data governance process always maintains the optimal configuration.

[0057] Regarding data transformation operators, data transformation ensures that the format or structure of each data item meets the needs of subsequent processing. This step not only involves basic format conversion, such as converting CSV (Comma-Separated Values) format to JSON (JavaScript Object Notation) format, but also includes more complex data structuring and data enrichment processes. Using a stream processing platform developed with a real-time streaming engine, these operations can be effectively performed through rich APIs and custom transformation logic. Further, data transformation operators can include:

[0058] 1. Format conversion operator: When processing data, the original data may exist in CSV format, which may not be optimal for some types of analysis. Using the `MapFunction` of the real-time streaming engine, a converter is implemented to parse CSV rows into JSON objects. This process involves reading each row of the CSV file, splitting by column, and then mapping to the corresponding key-value pairs in the JSON structure. For unstructured data, such as plain text data, regular expressions or text parsing techniques are used to extract key information and format it into structured data formats. For example, extract timestamps, IP (Internet Protocol) addresses, response codes, and other elements from logs, and organize these elements into structured JSON or XML (eXtensible Markup Language) formats.

[0059] 2. Data enrichment operator: To enhance the information content of data, log data is enriched by external API calls. For example, information about the user's geographic location is obtained by querying a service with the IP address, and then this data is added to each log record. In real-time streaming engines, `RichMapFunction` is used to access external systems or databases. This function allows to maintain state and perform complex business logic during the data governance operator processing. For example, matching the user ID in the log with the user information in the database in a telecom scenario and adding specific information about the user such as age, gender, etc. to the log record.

[0060] 3. Custom transformation operator: According to specific business needs and data characteristics, corresponding transformation logic is designed. This includes selectively converting data fields, merging data from multiple data operators, or applying different transformation rules based on data content.

[0061] Regarding the data aggregation operator, in real-time data governance, data aggregation is a key step to optimize the processing efficiency of data governance operators. Through aggregation operations, the amount of data that needs to be further processed is reduced, and summary information is provided for subsequent analysis. When processing large-scale data governance operators such as CDN log data in a telecom scenario, using time window-based aggregation strategies such as sliding windows or tumbling windows is very effective. Real-time streaming processing frameworks provide powerful window aggregation functions, allowing users to flexibly define window size and sliding interval to adapt to different business needs. Further data aggregation operators can include:

[0062] 1. Define time window operator: sliding window, in sliding window aggregation, the window slides at a fixed interval, and each sliding overlaps the previous window, this type of window is suitable for scenarios that need to continuously monitor data, for example, set a 5-minute window, slide every 1 minute, continuously calculate the data aggregation in the past 5 minutes; tumbling window (TumblingWindow), tumbling window is a special type of sliding window, where the window interval is equal to the window size, and the windows do not overlap, for example, calculate the total sum of data every 10 minutes, the data in each 10-minute segment is processed independently, suitable for scenarios that need to reset the counter periodically.

[0063] 2. Aggregation Function Operators: Statistical Analysis, Aggregation operations include calculating the mean, sum, max, and min of data, etc. In real-time streaming engines, this is accomplished by implementing the `AggregateFunction` interface, which requires defining three methods: `createAccumulator()` (create a new accumulator instance), `add()` (add a data item to the accumulator), and `getResult()` (get the aggregation result from the accumulator); Advanced Aggregation, In addition to basic statistical calculations, real-time streaming engines can implement more complex aggregation logic such as weighted average, moving average, or condition-based aggregation, which can be implemented in the `add()` method through custom logic.

[0064] 3. Window Data Governance Operators: Window Triggers, In real-time streaming engines, window triggers define when data aggregation calculations are performed, which can be based on time or data volume (such as triggering after receiving 1000 data points); Watermarks, To handle potential data delays or out-of-order issues, real-time streaming engines support watermark technology, which allows the system to understand the time progress in data governance operators, ensuring correct window processing even in out-of-order data.

[0065] Regarding data quality detection operators include:

[0066] 1. Data Integrity Verification Operator: Ensure that the received data is complete and intact, and there should be no missing data. For example, check if the required fields exist, and the data packet is complete.

[0067] 2. Data Accuracy Check Operator: Verify the accuracy and consistency of data, ensuring that data meets predefined formats and standards, including verifying data types (such as integers, floating-point numbers, strings, etc.), data formats (such as date formats), and whether it meets specific data patterns (such as email addresses, phone numbers, etc.).

[0068] 3. Data Consistency Check Operator: Ensure data consistency between different data sources, which involves cross-system data comparison to confirm that related data fields are the same in all systems.

[0069] 4. Outlier Detection Operator: Identify outliers or outliers in data, usually implemented through statistical analysis, such as calculating standard deviation and mean, and then identifying data points that deviate from the normal range.

[0070] 5. Data Repetition Check Operator: Detect duplicate records in data. This function is usually used to clean up data sets to avoid processing errors caused by data duplication.

[0071] 6. Time-sensitive verification operator: Ensures that data is up-to-date, especially when dealing with time-sensitive decision support systems. Verify the timestamp of data records to ensure that data is not outdated.

[0072] 7. Data normalization operator: Formats data to a standard format, ensuring that even if source data varies in format or units, it can be unified to a standard format for subsequent processing.

[0073] Through the above functions, the data quality detection operator can ensure that the data stream meets the quality requirements of the system before entering the next processing stage. Such operators are usually configurable, allowing detection rules and parameters to be adjusted according to different application requirements.

[0074] In addition, it should be noted that the embodiments of the present application also monitor each operator in real time, such as developing a DataListeners function in the real-time streaming engine in the present application, using a message middleware as a key component to listen to and capture data governance operators. These listeners are responsible for capturing data from various data generation sources in real time, and supporting subsequent data governance and analysis by accurately recording key information.

[0075] Data operator listeners are directly deployed at data production points, such as databases, application servers, or Internet of Things devices. Such deployment ensures that listeners can seamlessly access data governance operators and reduce latency and packet loss in data transmission.

[0076] Listeners connect to data operators through application program interface (API) hooks. For databases, use database triggers or log listeners; for application servers, capture HTTP requests through middleware; for Internet of Things devices, directly access the device's transport layer.

[0077] Listeners capture each data governance operator in real time. Important statistical information includes:

[0078] Data governance operator size: The number of bytes of each data governance operator, used to calculate traffic and data load.

[0079] Arrival time: Timestamp of data governance operator arrival at listener, accurate to milliseconds, critical to calculating data governance operator speed and monitoring data governance operator time distribution.

[0080] Source information: Identification of data governance operator source, such as IP address, port number, computing node, etc. This is crucial for tracing data governance operator sources and analysis.

[0081] Destination information: Related information of data governance operator destination, such as IP address, port number, computing node, etc., used to analyze data receiving end and flow direction.

[0082] The recording of these information adopts an efficient data structure and is stored in a time series database to support fast data insertion and query operations.

[0083] Based on the above introduction, the embodiment of the present application provides a regulation method of a data governance system, which refers to Figure 1 The flowchart of the first embodiment of the regulation method of the data governance system of the present application is shown in the figure.

[0084] In this embodiment, the regulation method of the data governance system applied to any one of the functional nodes in the data governance system comprises steps S10-S30:

[0085] Step S10, monitoring the total amount of data processed by the functional node in a preset time window for data governance tasks;

[0086] It should be noted that the application process of the above-mentioned part of the regulation method of the data governance system for sending the back pressure signal in different functional nodes is roughly similar, so in this embodiment, one of the functional nodes will be taken as an example for illustration.

[0087] For example, for any one of the functional nodes in the data governance system, the total amount of data processed by the functional node in a preset time window for data governance tasks can be monitored. For example, the monitoring can be completed by a listener in the data governance system, and the total amount of data in the preset time window is obtained by accumulating the data size of each data governance task received in the preset time window. In addition, in order to reduce the influence of instantaneous fluctuation on system decision, the exponential weighted moving average (EWMA) method can be used to smooth the input rate. In the smoothing process, the data governance task received at the time closest to the current time will be given a higher weight, so that the calculated total amount of data can more sensitively reflect the latest trend.

[0088] Step S20, determining the input rate of the functional node based on the total amount of data and the window length of the preset time window;

[0089] For example, after determining the total amount of data processed for data governance tasks, the input rate of the functional node can be calculated in combination with the window length of the preset time window (e.g. 1s). For example, the total amount of data is divided by the window length, and the input rate of the functional node is obtained. The calculation formula of the input rate is as follows:

[0090]

[0091] In the formula, the total amount of data in the window is the total amount of data processed for data governance tasks in the above-mentioned preset time window.

[0092] Step S30, in the case that the input rate exceeds the rate threshold corresponding to the functional node, a first back pressure signal is sent to the related node of the functional node, wherein the first back pressure signal is used to reduce the data sending rate of the related node.

[0093] For example, after calculating the input rate, the input rate is compared with the speed threshold corresponding to the functional node. In the case that the input rate exceeds the rate threshold corresponding to the functional node, it is confirmed that the data processing pressure of the functional node is too large, that is, in a high load state. In order to reduce the pressure and load of itself, the functional node can trigger the sending of a first back pressure signal to other related nodes. It is worth noting that the first back pressure signal is used to represent that the back pressure signal is sent by the pressure node, and the related node can include the upstream node of the functional node, or the downstream node of the functional node, which is not limited here. In addition, the first back pressure signal sent is used to reduce the data sending rate of the related node. Specifically, the way to reduce the data sending rate of the related node can be to adjust the data sending interval of the related node (increase the interval time), or reduce the number of data governance tasks sent by the related node each time, or temporarily stop the sending of data of the related node in an extreme case.

[0094] In the embodiment, the functional node in the data governance system will monitor the total amount of data processed by the functional node in a preset time window; determine the input rate of the functional node based on the total amount of data and the window length of the preset time window; in the case that the input rate exceeds the rate threshold corresponding to the functional node, a first back pressure signal is sent to the related node of the functional node, wherein the first back pressure signal is used to reduce the data sending rate of the related node. That is, each functional node in the data governance system will monitor the input rate of the task data received by itself, and in the case that the input rate exceeds the rate threshold corresponding to the functional node, trigger the sending of a corresponding back pressure signal to the related node to reduce the data sending rate of the related node, thereby reducing the load of itself and avoiding the high load state. It can be understood that since each functional node in the embodiment can adaptively send a pressure signal to the related node, the adjustment granularity of the data governance system is improved, the flexibility of adjustment is improved, so that even in the face of sudden increase of data amount, it can be better coped with without problem, thereby ensuring the robustness of the data governance system.

[0095] In a feasible implementation, the data governance method further comprises steps S41-S42:

[0096] Step S41, after receiving the second back pressure signal sent by the related node, determining the pressure level of the related node based on the second back pressure signal, and determining the rate reduction coefficient based on the pressure level, wherein the greater the pressure level, the smaller the rate reduction coefficient;

[0097] Step S42, calculating the target data sending rate for sending data to the related node according to the current data sending rate for sending data to the related node and the rate reduction coefficient, and sending data to the related node based on the target data sending rate, wherein the target sending rate is smaller than the current data sending rate.

[0098] It should be noted that the functional node will also receive the back pressure signal sent by the related node, i.e. the second back pressure signal, and the second back pressure signal represents the back pressure signal received by the functional node.

[0099] For example, the second back pressure signal can carry different pressure level identifiers, and the functional node can determine the pressure level according to the identifier. Then the pressure level is used to calculate the rate reduction coefficient. It should be noted that the greater the pressure level, the smaller the rate reduction coefficient. The functional node calculates the target data sending rate for sending data to the related node according to the current data sending rate for sending data to the related node and the rate reduction coefficient. For example, the calculation formula of the target data sending rate is as follows:

[0100] [New = Current · (1-Reduction)]

[0101] In the formula, Reduction represents the pressure level, 1-Reduction represents the rate reduction coefficient, Current represents the current data sending rate, and New represents the target sending rate. It can be seen from the above formula that the smaller the rate reduction coefficient, the smaller the target data sending rate.

[0102] Finally, after determining the target data sending rate, data is sent to the related node based on the target data sending rate, such as taking the target data sending rate as the new current data sending rate.

[0103] In a possible implementation, the adjusting method of the data governance system further includes step S51 ～ Step S53:

[0104] Step S51, monitoring the total number of data governance tasks processed by the functional node in a preset time window and the task processing interval, and determining the task processing rate of the functional node through the total number of tasks and the task processing interval;

[0105] Step S52, determining the node stress degree of the functional node based on the node computing resource utilization and the task processing rate of the functional node;

[0106] Step S53, in the case that the functional node is determined to be overloaded through the node stress degree, transferring the data governance tasks of the functional node to other target nodes, wherein the other target nodes are other functional nodes not in an overloaded state.

[0107] For example, the functional node also monitors the total number of data governance tasks processed by the functional node in a preset time window and the task processing interval, and calculates the task processing rate of the functional node through the detected task number and task processing interval.

[0108] The performance evaluation index task processing rate (Processed per Second, PPS) is calculated according to the amount of data processed in a certain period of time (i.e. the total number of tasks) and the time interval (i.e. the task processing interval), and the calculation formula is as follows:

[0109]

[0110] The calculated task processing rate is used to evaluate the subsequent node stress degree.

[0111] It should be noted that the node computing resource utilization mainly refers to the utilization rate of CPU, memory and other indicators of the functional node.

[0112] For example, the calculation formula of the node computing resource utilization is as follows:

[0113] [RU = w1 x CPU utilization rate + w2 x memory utilization rate + w3 x I / O utilization rate]

[0114] In the formula, RU (Resource Utilization) is the node computing resource utilization, W1, W2, W3, etc. are weight factors, which can be set by technical personnel according to actual conditions, and I / O refers to input / output operation in a computer system.

[0115] Finally, the node stress degree of the functional node is calculated using the node computing resource utilization and the task processing rate. The calculation formula is as follows:

[0116]

[0117] In the formula, PPS is the task processing rate, RU is the node computing resource usage rate, and SI (Stress Index) is the node stress degree.

[0118] It can be understood that the higher the SI, the greater the stress of the functional node, i.e., the processing capacity is close to or exceeds the design limit. In actual applications, a stress threshold can be set, and when the SI reaches or exceeds the stress threshold, the node is considered to be overloaded. The trigger condition for re-routing can be set, such as exceeding the stress threshold for N consecutive times in a measurement period, or the percentage of SI exceeding the stress threshold. Once the SI of a certain functional node exceeds the stress threshold, part of the data governance tasks on the functional node can be automatically transferred to other target nodes with lower load, i.e., other functional nodes that are not in an overloaded state. In addition, the SI indices of all nodes can be continuously monitored, and the stress threshold and weight factors W1, W2, and W3 can be adjusted regularly based on historical data and prediction models.

[0119] In a feasible implementation, the related nodes include upstream nodes and downstream nodes of the functional node, and before the step of monitoring the total number of tasks processed by the functional node in a preset time window and the task processing interval, the method includes steps S01-S02:

[0120] Step S01, receiving first initialization configuration completion information sent by the upstream node, and performing initialization verification on the upstream node based on the first initialization configuration completion information;

[0121] Step S02, after the initialization verification passes, performing initialization self-checking based on the first initialization configuration completion information, and after the initialization self-checking passes, sending second initialization configuration completion information to the downstream node.

[0122] It should be noted that at the initial stage of the data governance system, data flow initialization needs to be performed, and the tasks and dependency relationships of each node need to be configured. Each node is assigned a specific data processing task, such as data access, data quality inspection, data conversion, etc.

[0123] For example, in one scenario, there are node A and node B, and node A is the functional node described above, responsible for data access tasks. In this case, node A directly accesses the data source and there is no upstream node. Node A asynchronously receives data from an external data source and performs preliminary formatting and cleaning. After successful data access, node A generates a completion signal containing status information (such as the second initialization configuration completion information described above). This signal not only indicates the completion of data access to the next operator, but also contains detailed status information about the success or failure of data access. Node A sends the completion signal to the downstream node, i.e., node B. After receiving the completion signal and status information from node A, node B first performs a status check. The check process checks the status information passed by node A to confirm whether the data has been successfully accessed. If the verification result indicates that the data has not been successfully accessed, node B will terminate the current process and send an error report to the system monitoring center. If the data is successfully accessed, node B starts the data quality checking program. This program performs a series of quality checks on the accessed data, such as integrity checks and outlier detection.

[0124] In addition, it can be understood that, based on the above example, in another scenario, if node B is the functional node described above, then node A is the upstream node of node B. Accordingly, the completion information sent by node A to node B is the first initialization configuration completion information described above, and the other processes of the two scenarios are basically similar, so they will not be described here.

[0125] Reference Figure 2 For the first embodiment of the adjustment method of the data governance system based on the present application, the second embodiment of the adjustment method of the data governance system based on the present application is proposed. In this embodiment, the same or similar content as the above embodiment can be referred to in the above introduction, and will not be described hereinafter. In the case where the functional node is a resource scheduling node, the adjustment method of the data governance system further comprises steps S100-S200:

[0126] Step S100, for any one data governance task in the data governance system, based on the priority proportion of the data governance task and the current total available computing resources of the data governance system, determining the computing resource allocation amount of the data governance task, wherein the priority proportion of the data governance task is the priority weight of the data governance task in the total priority weight of each data governance task in the data governance system;

[0127] Step S200, according to the computing resource allocation amount, allocating computing resources to the functional node executing the data governance task.

[0128] It should be noted that in this embodiment, if the function node is a resource scheduling node, the function node will also implement the function of resource scheduling. For example, the resource scheduling node can monitor each data governance task in the data governance system and allocate computing resources to the data governance tasks according to the urgency of the data governance tasks. Since the process of allocating resources to each data governance task is roughly the same, in this embodiment, one of the data governance tasks will be taken as an example for description.

[0129] For example, for any one data governance task in the data governance system, the computing resource allocation of the data governance task is calculated according to the priority proportion of the data governance task and the current total available computing resources of the data governance system. The priority proportion of the data governance task refers to the proportion of the priority weight of the data governance task in the total priority weight of the data governance tasks in the data governance system. The calculation formula of the computing resource allocation of the data governance task is as follows:

[0130]

[0131] In the formula, P task represents the priority weight of the data governance task, R allocated represents the computing resource allocation, A available represents the current total available computing resources, ∑ all P task represents the total priority weight.

[0132] After the resource scheduling node determines the computing resource allocation of the data governance task, it can allocate computing resources to the function node that executes the data governance task.

[0133] In addition, the priority weight of the data governance task can be dynamically calculated based on the urgency, expected resource demand and business importance of the task. For example, tasks that are time-sensitive or have a greater impact on the system can be given higher weights. These weights are adjusted according to historical performance data and prediction models to adapt to the actual situation of system operation.

[0134] It can be understood that through the above method, resources can be dynamically allocated and real-time performance changes can be responded to, so as to realize efficient execution of tasks and stable operation of the system.

[0135] Referring to Figure 3 For the first and second embodiments of the adjustment method of the data governance system based on the present application, the third embodiment of the adjustment method of the data governance system based on the present application is proposed. In this embodiment, the same or similar contents as the above embodiments can be referred to the above introduction, and will not be described in detail. The adjustment method of the data governance system further comprises steps A100-A200:

[0136] Step A100, monitoring the current total available computing resources in the data governance system;

[0137] Step A200, in the case of determining that the data governance system is in a resource shortage state through the current total available computing resources, then reducing the priority weight of a low-priority data governance task, or suspending the execution of the low-priority data governance task.

[0138] It should be noted that in the present embodiment, the above-mentioned resource scheduling node will also monitor the current total available computing resources in the application data governance system to determine whether the data governance system is in a resource shortage state. For example, the occupation amount of the current total available computing resources can be monitored, and when the occupation amount is greater than a preset occupation amount threshold, it can be considered that the data governance system is in a resource shortage state. Correspondingly, in order to ensure that the data governance tasks with high priority can be processed in time, in the present embodiment, the priority weight of the low-priority data governance task is adaptively reduced, or the execution of the low-priority data governance task is suspended. After the data governance system exits the resource shortage state, the priority weight of the low-priority data governance task is restored, and the execution of the low-priority data governance task is restarted.

[0139] It can be understood that in the present embodiment, when the system resource is in shortage, the resource scheduling node can increase the resource allocation to the high-priority task, reduce the resource allocation of the low-priority task, or postpone the execution thereof. So as to ensure that the high-priority task can be supported by sufficient computing resources.

[0140] The resource scheduling node will adjust the resource allocation strategy according to the real-time performance monitoring data. This includes monitoring the CPU utilization, memory occupation, I / O throughput and other key indicators. Any adjustment of resource allocation aims to optimize task execution efficiency and respond to system load changes, reduce task waiting time and execution delay. When the system detects task execution errors or performance problems, it will automatically trigger the task rescheduling or restart mechanism. Recover the task from the latest checkpoint, clear the resources affected by the faulty task or reassign the task to other nodes. During the rescheduling process, the resource scheduling node will consider the current system resource state and the priority of all tasks to determine the optimal task execution order and resource allocation.

[0141] In a feasible implementation, the step of reducing the priority weight of the low-priority data governance task, or suspending the execution of the low-priority data governance task includes steps A210-A220:

[0142] Step A210, differentiating the priority weights of each data governance task in the data governance system to obtain a new priority weight of each data governance task, and returning to execute the step of determining the computing resource allocation amount of the data governance task based on the priority proportion of the data governance task and the current total available computing resources of the data governance system based on the priority weight of each data governance task; or

[0143] Step A220, adding the low-priority data governance task to the waiting queue, wherein the data governance task added to the waiting queue will be suspended and the computing resources corresponding to the task in the waiting queue will be suspended.

[0144] It should be noted that in the case of resource shortage state of the data governance system, various measures can be taken to ensure that the high-priority data governance task can be normally completed.

[0145] For example, the resource scheduling node can differentiate the priority weights of each data governance task in the data governance system, wherein the differentiation processing is to narrow the weight gap between high-priority tasks and low-priority tasks, that is, to make the priority weight of high-priority tasks higher and the priority weight of low-priority tasks lower. Thus, more computing resources are allocated to support high-priority tasks. After differentiation processing, the new priority weight of each data governance task can be obtained, and the step of determining the computing resource allocation amount of the data governance task based on the priority proportion of the data governance task and the current total available computing resources of the data governance system based on the priority weight of each data governance task can be returned.

[0146] Alternatively, in another processing manner, the low-priority data governance task is added to the waiting queue, and the data governance task added to the waiting queue is suspended and the computing resources for supporting the execution of the task in the waiting queue are suspended.

[0147] It should be noted that the selection of the above two methods can be based on the level of resource shortage state, such as in the low-level resource shortage state, the above step A210 scheme can be selected, and in the high-level resource shortage state, the above step A220 scheme can be selected. Through the above method, the flexibility of resource scheduling can be greatly improved.

[0148] Reference Figure 4 For the first embodiment, the second embodiment, and the third embodiment of the adjustment method of the data governance system based on the present application, the fourth embodiment of the adjustment method of the data governance system based on the present application is proposed. In this embodiment, the same or similar contents as the above embodiments can be referred to the above introduction, and will not be described in detail. The adjustment method of the data governance system further comprises step B100～ Step B300:

[0149] Step B100, monitoring utilization information of total available computing resources of the data governance system in a preset time window, and load information of each functional node in the data governance system in the preset time window;

[0150] Step B200, inputting the utilization information and the load information into a pre-trained prediction model, and predicting, by the pre-trained prediction model, a computing resource demand estimation of the data governance system in a future preset period;

[0151] Step B300, taking the computing resource demand estimation as an adjustment target, and adjusting current total available computing resources of the data governance system.

[0152] For example, the resource scheduling node can also monitor utilization information of total available computing resources of the data governance system in a preset time window, and load information of each functional node in the data governance system in the preset time window. The utilization information can include an average value of total available computing resources corresponding to each time node in the preset time window, or a change trend of total available computing resources corresponding to each time node in the preset time window, etc. The load information of each functional node in the data governance system can include indicators such as a monitoring operator queue length, a data processing rate, and a resource consumption of each functional node. The utilization information and the load information are input into a pre-trained prediction model. The pre-trained prediction model can be a neural network model that is pre-trained, and a specific training process is not described herein. The computing resource demand estimation of the data governance system in a future preset period can be predicted by the pre-trained prediction model. The resource scheduling node takes the computing resource demand estimation as an adjustment target, and adjusts current total available computing resources of the data governance system, so that the current total available computing resources of the data governance system approaches the computing resource demand estimation.

[0153] For example, a calculation formula of an adjustment amount of the current total available computing resources is as follows:

[0154] [ΔR=γ×(D predicted -D current )]

[0155] In the formula, D predicted is the computing resource demand estimation, D current is the current total available computing resources, γ is an adjustment sensitivity parameter, which can be set by a technician according to a demand, and ΔR is the adjustment amount of the current total available computing resources. The computing resource demand estimation D predicted can be obtained by prediction using a regression analysis model, and a prediction formula is as follows:

[0156] [D predicted= β0+ β1x past data input rate + β2x past CPU usage rate + ∈

[0157] where β0, β1, β2 are model parameters, which are trained by historical training samples, and ∈ is a hyperparameter error term.

[0158] In addition, in the real-time streaming engine, the expansion and contraction strategy of the current total available computing resources is defined according to the business requirements and the requirements of cost control. For example, when the CPU usage rate continuously exceeds 80%, the system automatically increases the computing resources to cope with the increased load; on the contrary, when the CPU usage rate is less than 20%, the computing resources are reduced to avoid unnecessary cost overhead.

[0159] The setting of the trigger condition is the core part of the resource elasticity scaling strategy, which is based on the back pressure adjustment and the prediction result. In addition to the resource usage rate, the trigger condition can also include the task delay, error rate and other key performance indicators.

[0160] When the trigger condition is met, the resource management scheduler will automatically perform the predefined resource expansion or reduction operation. This process is usually implemented through API interface integration. This automatic resource adjustment mechanism allows the system to seamlessly expand or reduce to cope with changing business needs.

[0161] In addition, referring to Figure 5 is the overall framework diagram of the embodiment of the present application. As shown in Figure 5 the operator shown in the figure, and the function of the operator can refer to the above content, and here, it will not be repeated. In addition, the back pressure adjustment logic in the figure is the process of triggering the sending of the back pressure signal by the functional node in the embodiment of the present application. The resource management scheduler is the resource scheduling node described above, and the elastic resource management is the process of scheduling resources according to the task priority by the resource scheduling node. The feedback back data access is used to feedback the success message after data access.

[0162] Referring to Figure 6 is the overall flow diagram of the embodiment of the present application. Figure 6 including:

[0163] Step one: Define data governance operators. Data governance operators include data filtering, data conversion, data aggregation, and data quality detection operators to ensure data format uniformity, information enhancement, statistical analysis, and quality checks. Data filtering operator: filter out data that does not meet the conditions through predefined rules, reduce irrelevant data volume, and improve processing efficiency. Data conversion operator: convert data format to the structure required for subsequent processing, such as converting CSV to JSON and enriching data. Data aggregation operator: use time window strategy to aggregate data and provide statistical analysis results. Data quality detection operator: check data integrity, accuracy, consistency, outliers, and repeatability to ensure data quality.

[0164] Step two: Dynamic data operator monitoring. Deploy listeners at data production points to capture key data (size, arrival time, source information, destination information) of data governance operators in real time, and use time series databases to support fast data insertion and query. Listener deployment: deploy listeners on databases, application servers, or IoT devices to capture data governance operators. Real-time capture: record the size, arrival time, source information, and destination information of data governance operators and store them in a time series database.

[0165] Step three: Operator data flow control. Initialize data flow, configure the tasks and dependencies of each operator, perform state verification and error handling, dynamically adjust input data flow, and implement backpressure mechanism. Data flow initialization: configure the tasks and dependencies of each operator to ensure the orderliness of data flow. State verification: check data access state to ensure data quality. Backpressure mechanism: dynamically adjust input data flow to avoid system overload.

[0166] Step four: Real-time rate calculation. Use a sliding time window to calculate the average input rate of each data operator in real time, and use an exponential weighted moving average method to reduce the impact of fluctuations. Sliding time window: calculate the average input rate of each data operator in real time. Smoothing: use an exponential weighted moving average method to reduce fluctuations in rate calculation.

[0167] Step five: Load evaluation algorithm. Dynamically adjust the load threshold based on resource usage and historical data, and use a regression analysis model to predict resource demand. Dynamic threshold adjustment: adjust the load threshold based on resource usage and historical data. Regression analysis model: predict resource demand to optimize system performance.

[0168] Step six: Real-time pressure evaluation algorithm. Monitor resource usage in real-time, calculate Load Index (LI) and Stress Index (SI), and automatically re-route to adjust the distribution of data governance operators. Resource usage monitoring: Monitor resource usage of each node in real-time. Calculate load and stress index: Calculate Load Index and Stress Index based on resource usage. Automatic re-routing: Adjust the distribution of data governance operators according to the stress index.

[0169] Step seven: Back pressure adjustment logic. Trigger back pressure mechanism based on real-time input rate and resource utilization, multi-level back pressure adjustment, and hierarchical transmission of back pressure signals to optimize system-level data governance operator management. Back pressure adjustment trigger: Trigger back pressure mechanism based on real-time input rate and resource utilization. Multi-level back pressure adjustment: Hierarchical transmission of back pressure signals to optimize system-level data governance operator management.

[0170] Step eight: Resource management scheduler. Dynamically allocate resources, automatically adjust computing resources based on task priority and resource demand, and monitor and adjust in real-time based on system performance data. Dynamic resource allocation: Automatically adjust computing resources based on task priority and resource demand. Real-time monitoring and adjustment: Optimize resource allocation based on system performance data.

[0171] Step nine: Elastic resource management. Predict future resource demand, dynamically adjust resource allocation, and perform resource expansion or reduction based on trigger conditions to ensure efficient system operation. Prediction and real-time data analysis: Predict future resource demand and dynamically adjust resource allocation. Automatic resource adjustment: Perform resource expansion or reduction based on trigger conditions.

[0172] The present application provides an electronic device, comprising: at least one processor; and a memory communicatively connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the adjustment method of the data governance system in the above-mentioned embodiment one.

[0173] Reference will now be made to the following description Figure 7 , which shows a structural schematic diagram suitable for implementing the electronic device of the embodiments of the present application. The electronic device in the embodiments of the present application can include, but is not limited to, mobile terminals such as computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and the like, as well as fixed terminals such as computers. Figure 7The electronic device shown is merely an example and should not impose any limitation on the functions and usage scope of the embodiments of the present application.

[0174] As shown in Figure 7 The electronic device can include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. Various programs and data required for the operation of the electronic device are also stored in the RAM 1004. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although the electronic device having various systems is shown in the figure, it should be understood that all the systems shown are not required to be implemented or possessed. More or less systems can be alternatively implemented or possessed.

[0175] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program codes for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of the embodiments disclosed in the present application are performed.

[0176] The electronic device provided in the present application adopts the adjustment method of the data governance system in the above-mentioned embodiments, and can solve the technical problem of poor flexibility of the back pressure mechanism of the existing stream engine. Compared with the prior art, the electronic device provided in the present application has the same beneficial effects as the adjustment method of the data governance system provided in the above-mentioned embodiments, and other technical features in the electronic device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0177] It should be understood that portions of the application disclosed can be implemented in hardware, software, firmware, or combinations thereof. In the description of the embodiments above, specific features, structures, materials or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0178] The above description is merely illustrative of the application and is not intended to limit the scope of the application. Any changes and modifications that can be made to the application in light of the teachings described herein are to be encompassed by the application. Therefore, the scope of the application should be determined by the scope of the claims.

[0179] The application provides a computer readable storage medium having stored thereon computer readable program instructions (i.e., computer programs) for performing the adjusting method of the data governance system in the above-described embodiments.

[0180] The computer readable storage medium provided by the application may, for example, be a U disk, but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more conductive wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer readable storage medium can be transmitted in any suitable medium, including but not limited to electrical wire, optical cable, RF (Radio Frequency), etc., or any suitable combination of the above.

[0181] The above computer readable storage medium can be included in an electronic device; or can exist separately and not be assembled into an electronic device.

[0182] The above computer readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to:

[0183] monitoring a total task quantity and a task processing interval of the functional node in processing tasks in a preset time window, and determining a task processing rate of the functional node according to the total task quantity and the task processing interval;

[0184] determining a node pressure degree of the functional node according to the node computing resource usage rate of the functional node and the task processing rate;

[0185] in a case where it is determined through the node pressure degree that the functional node is overloaded, sending a first back pressure signal to a related node of the functional node, wherein the first back pressure signal is used to reduce a data sending rate of the related node.

[0186] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0187] The computer program instructions can also be loaded onto a computer or other programmable information processing apparatus to cause a series of operations to be performed on the computer or other programmable information processing apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable information processing apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0188] The modules described in the embodiments of the present application can be implemented in the form of software or in the form of hardware. In some cases, the names of the modules do not constitute a limitation on the modules themselves.

[0189] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e., a computer program) for executing the adjustment method of the data governance system, and can solve the technical problem of poor flexibility of the back pressure mechanism of the existing stream engine. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the adjustment method of the data governance system provided by the above embodiments, and will not be described here.

[0190] The present application also provides a computer program product comprising a computer program, which, when executed by a processor, implements the steps of the adjustment method of the data governance system as described above.

[0191] The computer program product provided by the present application can solve the technical problem of adjusting the data governance system. Compared with the prior art, the computer program product provided by the present application has the same beneficial effects as the adjustment method of the data governance system provided by the above embodiments, and will not be described here.

[0192] The above only describes some embodiments of the present application, and does not limit the patent scope of the present application. Any equivalent structural transformation, direct / indirect application in other related technical fields based on the technical concept of the present application, and the contents of the present application and the accompanying drawings are included in the patent protection scope of the present application.

Claims

1. A method for regulating a data governance system, characterized in that, The application is applied to any one function node in a data governance system, and the adjustment method of the data governance system comprises the following steps: Monitoring the total amount of data processed by the function node in a preset time window; Determining the input rate of the function node based on the total amount of data and the window length of the preset time window; In the case where the input rate exceeds the corresponding rate threshold of the function node, a first back pressure signal is sent to the related node of the function node, wherein the first back pressure signal is used to reduce the data sending rate of the related node; In the case where the function node is a resource scheduling node, the adjustment method of the data governance system further comprises: For any one data governance task in the data governance system, based on the priority proportion of the data governance task and the current total available computing resources of the data governance system, the computing resource allocation amount of the data governance task is determined, wherein the priority proportion of the data governance task is the priority weight of the data governance task, and the proportion in the total priority weight of each data governance task in the data governance system; According to the computing resource allocation amount, computing resources are allocated to the function node for executing the data governance task; The adjustment method of the data governance system further comprises: Monitoring the current total available computing resources in the data governance system; In the case where the current total available computing resources determine that the data governance system is in a resource shortage state, the priority weight of a low-priority data governance task is reduced, or the execution of the low-priority data governance task is suspended; The step of reducing the priority weight of a low-priority data governance task or suspending the execution of the low-priority data governance task comprises: Differentiating the priority weight of each data governance task in the data governance system to obtain the new priority weight of each data governance task, and returning to the step of determining the computing resource allocation amount of the data governance task based on the priority proportion of the data governance task and the current total available computing resources of the data governance system; or Joining the low-priority data governance task to a waiting queue, wherein the data governance task joined to the waiting queue will be suspended, and the corresponding computing resources for the tasks in the waiting queue are suspended.

2. The method of claim 1, wherein, The data governance method further comprises: After receiving the second back pressure signal sent by the related node, the pressure level of the related node is determined based on the second back pressure signal, and the rate reduction coefficient is determined based on the pressure level, wherein the greater the pressure level, the smaller the rate reduction coefficient; According to the current data sending rate of sending data to the related node and the rate reduction coefficient, the target data sending rate of sending data to the related node is calculated, and data is sent to the related node based on the target data sending rate, wherein the target sending rate is less than the current data sending rate.

3. The method of claim 1, wherein the data governance system is adjusted by, The adjustment method of the data governance system further comprises: monitoring a total task quantity of data governance tasks processed by the functional node in a preset time window and a task processing interval, and determining a task processing rate of the functional node based on the total task quantity and the task processing interval; determining a node stress degree of the functional node based on a node computing resource usage rate of the functional node and the task processing rate; in a case where it is determined through the node stress degree that the functional node is overloaded, transferring data governance tasks of the functional node to other target nodes, wherein the other target nodes are other functional nodes that are not in an overloaded state.

4. The method of claim 1, wherein, The adjustment method of the data governance system further comprises: monitoring utilization information of total available computing resources of the data governance system in a preset time window, and load information of each functional node in the data governance system in a preset time window; inputting the utilization information and the load information into a pre-trained prediction model, and predicting a computing resource demand estimation of the data governance system in a future preset time period through the pre-trained prediction model; using the computing resource demand estimation as an adjustment target to adjust current total available computing resources of the data governance system.

5. An electronic device, comprising: The device comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of adjusting the data governance system according to any one of claims 1 to 4.

6. A storage medium, characterized by The storage medium is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of adjusting the data governance system according to any one of claims 1 to 4.

7. A computer program product, characterised in that, The computer program product comprises a computer program, and the computer program is executed by a processor to implement the steps of adjusting the data governance system according to any one of claims 1 to 4.

Images