A data processing method and system for visualizing modeling

Abstract

The application relates to the technical field of visual modeling, in particular to a data processing method and system for visual modeling. The method comprises the following steps: dividing a target task into several processing stages, extracting resource demand vectors, occupation rate dispersion and unit data benchmark processing time length of each stage; when the task is close to the next processing stage, amplifying the resource demand vector by the occupation rate dispersion to obtain a residual demand vector, and calculating the ratio of real-time resource idle rate and the residual demand vector to obtain a resource residual ratio to determine resource shortage; when the resource is in shortage, a candidate node with the maximum positive net income is selected to perform task migration. Through the technical scheme, the accurate scheduling of computing resources can be realized, and the efficiency of visual modeling is improved.

Description

Technical Field

[0001] This application relates to the field of visualization modeling technology, and in particular to a data processing method and system for visualization modeling. Background Technology

[0002] Visualization modeling refers to the process of dynamically presenting data through graphical means, enabling users to intuitively observe data changes. The visualization modeling process includes multiple steps such as data acquisition, data processing, visualization chart generation, and visualization effect rendering. Each step requires the use of corresponding computing resources (including CPU, GPU, memory, and storage I / O) to achieve visualization modeling. Therefore, the scheduling of various computing resources during the visualization modeling process directly affects the efficiency of visualization modeling.

[0003] Currently, Chinese patent application CN115834588A discloses a method, system, and related equipment for improving load balancing of visualized data streams. This method receives data requests from clients, determines target data requests based on the requested data volume, identifies multiple candidate service nodes from among the service nodes for each target data request, selects the target service node with the lowest load index from among the candidate service nodes, sends the target data request to the target service node, and receives the returned data. This method achieves request-level load balancing scheduling by comparing the load indexes of each node.

[0004] However, the scheduling granularity of the above methods is based on a single data request. Scheduling decisions are made solely based on the node's current overall load index, failing to differentiate the varying resource requirements of different processing stages, or consider the adaptation overhead caused by sudden changes in resource requirements when switching from the current stage to the next. When a multi-stage data processing task switches stages during execution, the overall load index alone cannot determine whether the target node's resource structure matches the specific requirements of the next stage. This may lead to scheduling the task to a node with low overall load but incompatible resource types, resulting in increased overhead such as cache invalidation and page replacement, and ultimately failing to effectively improve processing efficiency. Summary of the Invention

[0005] To address the technical issues of scheduling decisions failing to differentiate the varying resource requirements of different processing stages in multi-stage tasks, and the inability to effectively improve processing efficiency due to sudden changes in resource requirements during stage switching, this application provides a data processing method and system for visualization modeling.

[0006] In a first aspect, this application provides a data processing method for visualization modeling, the method comprising: S101, performing ordered sample clustering on the historical resource occupancy time series of the target task, dividing it into several processing stages, and extracting the resource demand vector, occupancy rate dispersion, and unit data baseline processing time of each processing stage; S102, in response to the current processing stage reaching a preset progress threshold, collecting the real-time resource idle rate of the current node, amplifying the resource demand vector of the next processing stage with the occupancy rate dispersion of the next processing stage to obtain a demand vector with surplus, and calculating the ratio of the real-time resource idle rate to the demand vector with surplus in each dimension to obtain the resource surplus ratio, wherein there exists a ratio less than 1. When the resource margin ratio is low, it is determined that resources are tight; S103, when resources are tight, the bottleneck margin ratio is determined according to the minimum value of the resource margin ratio, and the standby processing time is obtained by calculating the unit data baseline processing time of the next processing stage based on the bottleneck margin ratio. The standby processing time is corrected by the difference in resource demand between two adjacent processing stages to obtain the corrected standby time; S104, candidate nodes with resource margin ratios of not less than 1 in each dimension are screened, the target processing time and transmission time are calculated for each candidate node, and the target processing time and transmission time are successively subtracted from the corrected standby time to obtain the switching net benefit. The candidate node with the largest and positive switching net benefit is selected as the target node to perform task migration.

[0007] The process divides the processing stages using ordered sample clustering and extracts resource demand vectors, occupancy rate dispersion, and unit data baseline processing time as benchmark data. Before stage switching, resource sufficiency is determined dimension-by-dimensionally using the demand vector with margin. For resource shortages, the retention time is adjusted using a two-factor calculation based on the bottleneck margin ratio and resource demand difference. Candidate nodes are screened, and the migration criterion is the largest and positive net switching benefit. This decision chain links resource gap assessment, retention cost estimation, and migration benefit comparison layer by layer, enabling migration decisions to simultaneously reflect the efficiency impact of both resource quantity gaps and resource type adaptation.

[0008] Preferably, after the task migration, the method further includes: taking the target node as the new current node, referring to the current processing stage of the task as the current processing stage, and repeatedly executing the actions of collecting the real-time resource idle rate, determining resource shortage, calculating the corrected stay time, and calculating the switching net benefit when the task approaches each subsequent processing stage, until the task enters the last processing stage.

[0009] Preferably, the process of dividing the process into several processing stages includes: gradually increasing the number of processing stages starting from a preset initial number; calculating the decrease of the sum of squared deviations within each group of all processing stages relative to the previous number after each increase; and determining the final number of processing stages when the decrease is lower than the preset proportion of the previous decrease for the first time.

[0010] The decreasing rate of the sum of squared deviations within a group is used as the stopping criterion for the number of processing stages. The number of stages is determined by the characteristics of the data structure to avoid too few stages masking changes in demand or too many stages causing statistical distortion, thus ensuring the statistical reliability of the baseline data for each stage.

[0011] Preferably, the process of obtaining the unit data baseline processing time includes: for each processing stage, dividing the historical execution records of the processing stage into ample bins, moderate bins, and tense bins according to the resource idle rate of the node at the time of execution, and calculating the average unit data processing time of the processing stage in the ample bin as the unit data baseline processing time of the processing stage; the unit data processing time is the stage duration in the historical execution record divided by the amount of input data processed in the stage.

[0012] Using the unit data processing time of a sufficiently large bin as the benchmark processing time, the interference of abnormally high values ​​under resource-scarce conditions on the benchmark is eliminated, so that the estimation of subsequent standby costs and target processing time reflects the actual processing capacity of nodes under resource-scarce conditions.

[0013] Preferably, the process of obtaining the demand vector with margin includes: using the sum of 1 and the occupancy rate dispersion of the next processing stage as a safety margin coefficient, and multiplying each component of the resource demand vector of the next processing stage by the safety margin coefficient to obtain the demand vector with margin.

[0014] Using the sum of 1 and the dispersion of occupancy rate as a safety margin coefficient to amplify the resource demand vector, the resource sufficiency judgment automatically adapts to the differences in the fluctuation range of demand at each stage. A wider margin is reserved for stages with large fluctuations, and the margin is close to zero for stages with small fluctuations. This reduces the risk of a sharp drop in processing efficiency caused by instantaneous demand peaks without over-consuming resource margins.

[0015] Preferably, the calculation process of the resource demand difference between two adjacent processing stages includes: calculating the cosine similarity between the resource demand vector of the previous processing stage and the resource demand vector of the next processing stage, and using the difference between 1 and the cosine similarity as the resource demand difference between the two adjacent processing stages.

[0016] Preferably, the calculation process of the corrected stay-at-home time includes: using the sum of the resource demand differences between 1 and the two adjacent processing stages as the difference correction factor, and multiplying the stay-at-home processing time by the difference correction factor to obtain the corrected stay-at-home time.

[0017] The retention time is amplified by using the sum of 1 and the difference in resource demand as a correction factor. Implicit overheads such as cache invalidation caused by differences in resource demand direction between adjacent stages are factored into the retention cost. When the demand direction is similar, the correction factor is close to 1. When the difference is large, the retention cost is significantly amplified, making the switching decision more inclined to migrate to the node with matching resource type.

[0018] Preferably, the process of determining the bottleneck margin ratio includes: comparing the minimum value of the resource margin ratio with a preset margin lower limit, and taking the larger of the minimum value and the preset margin lower limit as the bottleneck margin ratio.

[0019] The minimum value of the resource reserve ratio is compared with the preset reserve lower limit, and the larger value is taken as the bottleneck reserve ratio. This avoids the reserve ratio from approaching zero when resources are extremely scarce, which would cause the standby processing time to be excessively amplified to an unrealistic value. This keeps the corrected standby time within a reasonable range and maintains the stability of the switchover net benefit calculation.

[0020] Preferably, the process of migrating the execution task includes: the current node dividing the intermediate data produced by the current processing stage of the task and used as the input for the next processing stage into several data blocks according to a preset block size, attaching a checksum, and transmitting them to the target node; the target node checks each block; and requests retransmission for data blocks that fail the checksum. During the transmission, the ratio of the real-time resource idle rate of the target node to the demand vector with surplus is calculated at preset intervals to obtain the resource surplus ratio of the target node. If there is a target node resource surplus ratio less than 1, the transmission is stopped and the task is rolled back to the current node to continue execution.

[0021] In a second aspect, this application also provides a data processing system for visualization modeling, including a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, a data processing method for visualization modeling according to the first aspect of this application is implemented.

[0022] The technical solution of this application has the following beneficial technical effects:

[0023] By performing ordered sample clustering on the historical resource occupancy time series, the task is divided into several processing stages with relatively uniform resource demand. When the task is about to switch stages, a demand vector with spare capacity is constructed based on the occupancy rate dispersion to determine the resource sufficiency dimension by dimension. After determining that the resources are tight, the dwell time is adjusted by a two-factor conversion of the bottleneck spare capacity ratio and the difference in resource demand. The adjusted dwell time is quantitatively compared with the target processing time and transmission time of the candidate nodes to decide whether to perform task migration. This makes the scheduling decision change from qualitative judgment based on experience thresholds to quantitative optimization based on time cost. Before the stage switch, the task is migrated to the node with higher resource matching degree for execution, reducing the decrease in processing efficiency caused by the difference in resource demand type between adjacent processing stages, and realizing the precise scheduling of computing resources in the process of visualization modeling. Attached Figure Description

[0024] Figure 1 This is a flowchart of a data processing method for visualization modeling according to an embodiment of this application.

[0025] Figure 2 This is a schematic diagram of the ordered sample clustering and partitioning process according to an embodiment of this application.

[0026] Figure 3 This is a schematic diagram illustrating the net benefit decision-making process for candidate node screening and switching according to an embodiment of this application.

[0027] Figure 4 This is a structural block diagram of a data processing system for visualization modeling according to an embodiment of this application. Detailed Implementation

[0028] According to a first aspect of this application, a data processing method for visualization modeling is provided. This embodiment is applied to the resource scheduling scenario of visualization modeling tasks in a distributed computing cluster. A complete visualization modeling data processing task consists of multiple sequentially dependent processing stages. The requirements of each stage for processor, memory, and disk read / write bandwidth differ significantly. When the task switches stages, the processing efficiency decreases due to the mismatch between the node resource supply and the requirements of the new stage. Figure 1 This is a flowchart of a data processing method for visual modeling according to an embodiment of this application. Figure 1 As shown, the data processing method for visualization modeling includes steps S101 to S104, which are described in detail below.

[0029] S101 performs ordered sample clustering on the historical resource occupancy time series of the target task, divides it into several processing stages, and extracts the resource demand vector, occupancy rate dispersion and unit data benchmark processing time of each processing stage.

[0030] In one embodiment, historical execution data is collected for a target data processing task, which is a visualization modeling task. Historical execution records of this task are extracted from the cluster resource monitoring system, and the resource usage status at each sampling moment is recorded at a fixed sampling period to form a resource usage time-series sequence. The sampling period is 1 second, determined based on the minimum collection granularity of the cluster resource monitoring system. A three-dimensional resource usage vector is recorded at each sampling moment, with three components: processor utilization, memory utilization, and disk read / write bandwidth utilization. Each component ranges from 0 to 1, is in ratio form, has uniform dimensions, and requires no additional normalization.

[0031] After the resource occupancy time series is collected, ordered sample clustering is performed to divide it into processing stages. Ordered sample clustering is a clustering method that preserves the temporal order of samples. It finds several split points on the time axis such that samples between adjacent split points are as similar as possible, and samples between different split intervals are as different as possible. The splitting criterion is to minimize the sum of squared deviations within groups of all stages. The sum of squared deviations within groups is defined as the sum of the squares of the Euclidean distances between the resource occupancy vectors of all sampling points in each stage and the mean vector of that stage. Suppose there are N sampling points in the time series, and the goal is to divide it into K processing stages. The optimal split point positions are solved through dynamic programming to obtain the time boundaries of the K processing stages.

[0032] like Figure 2 As shown in the diagram, this is a schematic diagram of the ordered sample clustering and partitioning process according to an embodiment of this application. The horizontal axis represents the sampling time, and the vertical axis represents the resource utilization rate. The solid line represents the processor utilization rate, the long dashed line represents the memory utilization rate, and the dotted line represents the disk read / write bandwidth utilization rate. Segmentation points 1, 2, and 3 divide the resource utilization time sequence into four processing stages. In the interval before segmentation point 1, the processor utilization rate remains around 0.8 while the memory and disk utilization rates are low, exhibiting computationally intensive characteristics. In the interval between segmentation point 1 and segmentation point 2, the memory utilization rate jumps to around 0.8 while the processor utilization rate falls back, exhibiting memory-intensive characteristics. In the interval between segmentation point 2 and segmentation point 3, the disk read / write bandwidth utilization rate increases significantly, exhibiting storage-intensive characteristics. In the interval after segmentation point 3, the utilization rates of all three resources are at a moderate level, exhibiting a mixed characteristic.

[0033] The method for determining the number of processing stages K includes: starting from a preset initial number of 2, the number of processing stages is incremented sequentially. After each increment, the decrease in the sum of squared deviations within each group of all processing stages relative to the previous number is calculated. When the decrease in the current increment is lower than the preset percentage of the previous decrease for the first time, the number before the current increment is determined as the final number of processing stages. The preset percentage is 20%, which is determined through historical task calibration. The specific process is as follows: select more than 10 groups of historical tasks, set a preset percentage value for each group of tasks in increments of 1% within the range of 15% to 25%, and execute the above-mentioned incremental stopping criterion. Record the number of final stages output by each group of tasks at each preset percentage value. For each preset percentage value, count the percentage of all historical tasks whose final stage number is exactly the same as the number of final stages output by the preceding and following percentage values ​​adjacent to that percentage value. When the percentage corresponding to a certain preset percentage value first reaches 80%, that preset percentage value is determined as the final preset percentage. If multiple preset percentage values ​​meet the 80% condition, the one with the smallest value is selected to retain a more sensitive stopping response. In this embodiment, the above conditions are met when the preset ratio is 20% after 10 sets of historical task calibration.

[0034] After dividing the process into K processing stages, the following baseline data is extracted for each processing stage: Calculate the resource demand vector for each processing stage. Take the arithmetic mean of the three-dimensional resource occupancy vectors at all sampling times within a stage to obtain the resource demand vector for that processing stage. Its three components represent the average demand intensity of processor, memory, and disk read / write bandwidth for that stage.

[0035] Calculate the occupancy dispersion for each processing stage. For the three-dimensional resource occupancy vector at all sampling times within a stage, calculate the standard deviation of each of the three components, and then take the arithmetic mean of the three standard deviations as the occupancy dispersion for that processing stage. The occupancy dispersion reflects the degree of fluctuation in resource demand within a stage; the larger the value, the more unstable the resource demand in that stage.

[0036] Extract the baseline processing time per unit of data for each processing stage. For each processing stage, divide the historical execution records of that stage into three bins based on the resource idle rate of the node at the time of execution: Ample, Moderate, and Tense. The resource idle rate is the arithmetic mean of the idle rates of the node across the three dimensions. Ample bins correspond to records with a resource idle rate greater than 0.6, moderate bins to records with a resource idle rate between 0.3 and 0.6, and tense bins to records with a resource idle rate less than 0.3. The bin boundaries are determined based on the ternary digits of the cluster's daily load distribution. Calculate the mean processing time per unit of data for each processing stage within each bin. The unit processing time equals the stage duration divided by the amount of input data processed in that stage. Take the mean of the ample bins as the baseline processing time per unit of data for that processing stage. This value represents the average time required to process one megabyte of input data under sufficient resource conditions, expressed in seconds per megabyte. If the number of historical samples in a bin is less than 5, the bin result is marked as unreliable, and the mean of all samples is used instead.

[0037] Taking processing phase 2 as an example, this phase has 20 historical execution records, of which 8 records with a resource idle rate greater than 0.6 are assigned to the ample bin. The phase durations of the 8 records are 45 seconds, 42 seconds, 48 ​​seconds, 44 seconds, 46 seconds, 43 seconds, 47 seconds, and 41 seconds, respectively, and the corresponding input data volumes are 120 megabytes, 110 megabytes, 130 megabytes, 115 megabytes, 125 megabytes, 112 megabytes, 128 megabytes, and 108 megabytes, respectively. The unit data processing time for each record is: 45 / 120=0.375, 42 / 110=0.382, 48 / 130=0.369, 44 / 115=0.383, 46 / 125=0.368, 43 / 112=0.384, 47 / 128=0.367, 41 / 108=0.380, all in seconds per megabyte. The average time for ample bins is 0.376 seconds per megabyte, meaning the baseline unit data processing time for processing stage 2 is 0.376 seconds per megabyte.

[0038] The resource demand difference between two adjacent processing stages is calculated using the standard cosine similarity formula. The difference between 1 and this cosine similarity is taken as the resource demand difference between the two processing stages. The resource demand difference ranges from 0 to 1; a larger value indicates a more significant difference in the types of resource demands between adjacent stages.

[0039] Furthermore, the intermediate data volume of each processing stage is recorded; the output data volume produced by each processing stage is counted, and this output data volume is the input data volume of the next processing stage, as well as the data volume that needs to be transferred when a task migration occurs later.

[0040] Discretizing the continuous resource occupancy time series into several processing stages with relatively uniform resource demand allows subsequent scheduling decisions to formulate strategies based on the differentiated demand characteristics of each stage, rather than applying a single resource evaluation standard to the entire task. Extracting the occupancy rate dispersion provides a quantitative basis for subsequent differentiated safety margin settings, and calculating the resource demand difference provides data support for cost correction during stage switching.

[0041] S102, in response to the current processing stage reaching a preset progress threshold, collect the real-time resource idle rate of the current node, amplify the resource demand vector of the next processing stage by the occupancy rate dispersion of the next processing stage to obtain a demand vector with surplus, calculate the ratio of the real-time resource idle rate to the demand vector with surplus in each dimension to obtain the resource surplus ratio, and determine that there is a resource surplus ratio less than 1 when there is a resource surplus ratio less than 1.

[0042] In one embodiment, the current node is the node executing the target task. During the execution of the task on the current node, the real-time resource status vector of the current computing node is continuously collected at 1-second intervals. The vector is three-dimensional, and the three components are the current node's processor idle rate, memory idle rate, and disk read / write bandwidth idle rate, all of which range from 0 to 1.

[0043] The execution progress of the current processing stage is measured by the ratio of the amount of data processed to the total input data volume of the current processing stage. A preset progress threshold of 0.8 is set. When this ratio reaches 0.8, a forward-looking assessment of resource adequacy for the next processing stage is triggered. The preset progress threshold is determined based on the following: within the execution time window of the remaining 20% ​​of the data volume in the current stage, a series of decision-making actions must be completed, including real-time resource acquisition, calculation of demand vectors with remaining capacity, resource scarcity assessment, estimation of standby costs, and candidate node selection. Furthermore, sufficient start-up time must be reserved for data transmission when determining migration. This threshold can be adjusted within the range of 0.75 to 0.85 based on the cluster network bandwidth and typical intermediate data volume. For clusters with lower bandwidth or larger intermediate data volumes, the threshold should be appropriately lowered to reserve a longer transmission window.

[0044] In response to the current processing stage reaching a preset progress threshold, the resource demand vector and occupancy dispersion for the next processing stage are extracted from the baseline data output by the ordered sample clustering above. The three components of the resource demand vector represent the average occupancy requirements of the processor, memory, and disk read / write bandwidth during the execution of the next processing stage. They are already in ratio form, consistent with the dimensions of the node idle rate, and can be directly compared.

[0045] The safety margin coefficient is calculated by multiplying the sum of 1 and the occupancy rate dispersion of the next processing stage. Each component of the resource demand vector for the next processing stage is then multiplied by this safety margin coefficient to obtain a demand vector with a safety margin. The physical meaning of the safety margin coefficient is: an additional resource margin equal to the fluctuation range is reserved on top of the average demand; the greater the occupancy rate dispersion, the more is reserved. The occupancy rate dispersion ranges from 0 to 1 (positive numbers), and the corresponding safety margin coefficient ranges from 1 to 2. If the occupancy rate dispersion of a certain processing stage is 0.15, then the safety margin coefficient is 1.15, and each component of the demand vector with a safety margin is amplified by 15% compared to the original resource demand vector.

[0046] The resource surplus ratio is obtained by calculating the ratio of real-time resource idle rate to the demand vector with surplus for each dimension. For the three dimensions of processor, memory, and disk read / write bandwidth, the real-time idle rate of the current node in that dimension is divided by the corresponding component of the demand vector with surplus, resulting in three resource surplus ratios. A resource surplus ratio greater than or equal to 1 indicates that the available resources in that dimension can cover the demand with surplus in the next processing stage, while a ratio less than 1 indicates that the resource supply in that dimension is insufficient. If a component in the demand vector with surplus is zero, meaning that the next processing stage will not use the resources of that dimension at all, the resource surplus ratio of that dimension is directly set to positive infinity, and the stress judgment for that dimension is skipped.

[0047] When the resource availability ratios of all three dimensions are greater than or equal to 1, the current node has sufficient resources, the task continues to execute on the current node, the current round of evaluation ends, and the node awaits re-collection and judgment in the next sampling period. When the resource availability ratio of at least one dimension is less than 1, the current node is determined to be in a resource-scarce state, the minimum value of the resource availability ratios of the three dimensions is recorded, and then the calculation process for correcting the dwell time begins.

[0048] The dimensional ratio judgment unifies the three heterogeneous resources onto a dimensionless sufficiency scale, so that a single bottleneck dimension can trigger a scheduling response, avoiding the comprehensive average of multi-dimensional resources from masking local shortcomings.

[0049] S103, when resources are scarce, the bottleneck margin ratio is determined based on the minimum value of the resource margin ratio, and the standby processing time is obtained by calculating the unit data baseline processing time of the next processing stage based on the bottleneck margin ratio. The standby processing time is then corrected based on the difference in resource demand between two adjacent processing stages to obtain the corrected standby time.

[0050] In one embodiment, after a node is determined to be in a resource-scarce state, the bottleneck margin ratio is first determined. The minimum value among the resource margin ratios of the three dimensions mentioned above is compared with a preset margin lower limit, and the larger of the two is taken as the bottleneck margin ratio. The preset margin lower limit is set to 0.1. When the available resources for a certain dimension are less than 10% of the demand, the node has usually triggered the overload protection mechanism at the operating system level, and the actual processing capacity will not continue to decrease linearly. Using 0.1 as the lower limit truncation avoids extreme amplification values ​​in subsequent calculations. This lower limit can be adjusted within the range of 0.05 to 0.15 based on the resource idle rate statistics when the node overload protection is triggered in the cluster operation and maintenance logs.

[0051] Once the bottleneck margin ratio is determined, it is used to calculate the dwell time for the next processing stage at the current node. The unit data baseline processing time for the next processing stage is extracted from the baseline data output by the ordered sample clustering above. Simultaneously, the amount of intermediate data actually produced in the current stage is obtained as the input data for the next processing stage. The dwell time equals the unit data baseline processing time multiplied by the amount of intermediate data measured, then divided by the bottleneck margin ratio. In essence, multiplying the unit data baseline processing time by the data volume yields the ideal total processing time under resource-sufficient conditions, and dividing by the bottleneck margin ratio converts the resource gap into a proportional decrease in processing speed. For example, a bottleneck margin ratio of 0.5 means that the available resources for the most critical dimension are only half of the demand, the processing speed is correspondingly reduced to half the ideal speed, and the processing time doubles.

[0052] The dwell time only reflects the efficiency reduction caused by insufficient resources, and does not take into account the additional adaptation overhead caused by the difference in resource demand types between adjacent stages. When the current processing stage is compute-intensive and the next processing stage is memory-intensive, the processor cache of the current node contains hot data of compute-intensive tasks, and the memory page allocation strategy is also optimized for compute-intensive loads. After entering the next processing stage, the cache hit rate drops sharply and memory page replacement occurs frequently. These implicit overheads make the dwell time lower than the estimated dwell time when using only the bottleneck margin.

[0053] The cosine similarity between the resource demand vector of the current processing stage and the resource demand vector of the next processing stage is calculated using the standard cosine similarity formula. The difference between 1 and the cosine similarity is taken as the resource demand difference between the two adjacent processing stages. The value of the resource demand difference ranges from 0 to 1. The closer the value is to 1, the greater the difference in the direction of resource demand between the two stages.

[0054] The difference between resource demand in stage 1 and the sum of the differences between the two adjacent processing stages is used as the difference correction factor. The corrected wait time is obtained by multiplying the wait time by the difference correction factor. The difference correction factor ranges from 1 to 2. When the difference is close to 0, the resource demand types of the two stages are similar, the correction factor is close to 1, and the wait time remains almost unchanged. When the difference is 0.8, the correction factor is 1.8, and the wait time is amplified by 80%, reflecting the additional time overhead caused by significant changes in resource type, such as cache invalidation and page replacement.

[0055] To illustrate the above process using a specific scenario: Assume the baseline processing time per megabyte for the next processing stage is 0.02 seconds. The measured intermediate data output in the current stage is 500 megabytes, and the bottleneck margin ratio, after truncation, is 0.4. Therefore, the dwell time is 0.02 × 500 ÷ 0.4 = 25 seconds. The three components of the resource demand vector for the current processing stage are 0.7, 0.2, and 0.1, respectively. The three components of the resource demand vector for the next processing stage are 0.2, 0.7, and 0.1, respectively. The cosine similarity between the two is calculated using the standard formula to be 0.537. The resource demand difference is 1 - 0.537 = 0.463. The difference correction factor is 1.463, and the corrected dwell time is 25 × 1.463 = 36.6 seconds. This corrected dwell time will be used as the cost of remaining at the current node in the subsequent calculation of the net benefit of switching.

[0056] The revised hold time comprehensively reflects two layers of efficiency loss: the first layer is the proportional decrease in processing speed due to insufficient resources, quantified by the bottleneck-to-spare ratio; the second layer is the adaptation overhead caused by resource type switching, quantified by differences in resource demand. The estimation of hold costs considers both the sufficiency and suitability of resources, providing a more realistic benchmark for subsequent switching decisions.

[0057] S104, filter candidate nodes whose resource margin ratio in each dimension is not less than 1, calculate the target processing time and transmission time for each candidate node, subtract the target processing time and transmission time from the corrected stay time to obtain the switching net benefit, and select the candidate node with the largest and positive switching net benefit as the target node to perform task migration.

[0058] In one embodiment, after the revised stay-at-home duration is determined, the real-time resource status of all computing nodes in the cluster, excluding the current node, is scanned. For each node, the real-time resource idle rate of the node in each of the three dimensions is divided by the corresponding component of the demand vector with remaining capacity in the next processing stage, using the same calculation method as the resource reserve ratio. The selection criteria are that the resource reserve ratios in all three dimensions are not less than 1 and the node is in normal operating condition. Nodes that meet the criteria are included in the candidate node set. If there are no candidate nodes that meet the criteria in the cluster, the current switchover evaluation is abandoned, the task continues to be executed on the current node, and the evaluation is waited for the next sampling period.

[0059] like Figure 3 The diagram illustrates the candidate node screening and switching net benefit decision-making according to an embodiment of this application. The cluster node pool contains five computing nodes, from node A to node E. Node A is the current node, and nodes B to E are nodes to be evaluated. Each node's three-dimensional resource idle rate is marked below it. A screening judgment is performed on nodes B to E, based on the criterion that the resource availability ratio in each dimension is not less than 1. Nodes that do not meet the condition are excluded, and those that do are included in the candidate node set. The target processing time and transmission time are calculated sequentially for each candidate node in the set. The target processing time equals the unit data baseline processing time multiplied by the intermediate data volume, and the transmission time equals the intermediate data volume divided by the network bandwidth. The switching net benefit equals the corrected stay time minus the target processing time minus the transmission time. It is determined whether the maximum switching net benefit is greater than 0. If it is, the candidate node with the largest net benefit is selected as the target node for task migration. The example table in the lower right corner of the diagram shows that node C's switching net benefit of 0.12 is the maximum value. Otherwise, the task remains on the current node, waiting for re-evaluation in the next cycle.

[0060] For each candidate node in the candidate node set, calculate the target processing time and transmission time respectively.

[0061] The calculation process for the target processing time includes: the resource margin ratio of the candidate nodes in all three dimensions is not less than 1, the resource supply can cover the demand of the next processing stage with margin, the processing efficiency is close to the ideal level under the condition of sufficient resources, and there is no need to reduce the margin ratio; the target processing time is equal to the unit data baseline processing time of the next processing stage multiplied by the actual measured intermediate data volume.

[0062] The calculation process for transmission time includes dividing the measured intermediate data volume by the network bandwidth between the current node and the candidate node. Network bandwidth is an inherent configuration parameter of the cluster and can be directly obtained from the cluster management system; the bandwidth between nodes in the same rack is usually higher than that between nodes across racks. For each candidate node, the actual bandwidth value between it and the current node is taken to calculate the transmission time separately.

[0063] For each candidate node, calculate the net switching benefit, which is equal to the corrected stay time minus the target processing time minus the transmission time. Select the node with the largest net switching benefit from all candidate nodes. If the largest net switching benefit is positive, decide to execute the task migration, and the corresponding candidate node becomes the target node. If the largest net switching benefit is less than or equal to zero, it indicates that the transmission cost of switching exceeds the difference between the stay time cost and the target processing cost, and the task continues to be executed on the current node, waiting for re-evaluation in the next sampling period.

[0064] Once the target node is identified, the current node continues to process the remaining data of the current processing stage, ensuring the complete output of intermediate data for the current processing stage. This intermediate data is the sole input for the next processing stage. Migrating midway through the current stage requires the simultaneous transmission of the original input and some intermediate results, increasing the risk of data inconsistency. Therefore, transmission is only initiated after the current stage has been completely completed.

[0065] After the current processing stage is completed, the intermediate data is transmitted from the current node to the target node. During transmission, the intermediate data is divided into several data blocks according to a preset block size, and a checksum is appended before each block is sent. The preset block size is 1 megabyte, determined based on the cluster network's maximum transmission unit and retransmission efficiency. Upon receiving the data, the target node uses standard cyclic redundancy check (CRC) to check each data block one by one. If the check passes, reception is confirmed; if the check fails, the target node requests retransmission of the block from the current node until the block passes the check, after which the target node continues receiving subsequent data blocks.

[0066] During transmission, the resource status of the target node is checked at preset intervals of 2 seconds to ensure that at least one target node status check is performed during transmission. Each check calculates the ratio of the target node's real-time resource idle rate to the demand vector with remaining capacity. If a target node resource remaining capacity ratio is less than 1, it indicates that other tasks are preempting resources, causing the target node to no longer meet the resource conditions for the next processing stage. Transmission is immediately stopped, incomplete data already received on the target node is discarded, and the task rolls back to the current node to continue executing the next processing stage. After rollback, the current node operates in a standby mode under resource pressure, waiting for subsequent sampling periods to reassess the switching conditions.

[0067] Once the transmission is complete and all verifications pass, the target node loads the intermediate data and begins executing the task from the starting point of the next processing stage. The task execution node is updated to the target node, and real-time resource monitoring in subsequent processing stages switches to the target node.

[0068] After migration, the target node is designated as the new current node, and the current processing stage of the task is referred to as the current processing stage. As the task approaches each subsequent processing stage—that is, when the current processing stage reaches a preset progress threshold—the actions of collecting real-time resource idle rate, determining resource scarcity, calculating and correcting dwell time, and calculating the net benefit of switching are repeatedly executed sequentially, forming a closed-loop scheduling mechanism throughout the entire task lifecycle. Iterative evaluation terminates when the task enters the final processing stage, and the final stage processing is completed directly at the current node, outputting the final result.

[0069] To mitigate the additional system overhead caused by frequent migrations, a cooling-off constraint is introduced into the iterative scheduling described above. After each task migration, the processing stage number at the time of the migration is recorded. When the next processing stage reaches a preset progress threshold and triggers the calculation of the switching net benefit, it is first determined whether the difference between the current processing stage number and the processing stage number at the time of the last migration is equal to 1. If it is equal to 1, it indicates that the current processing stage is adjacent to the last migration and is within the cooling-off period. In this case, the transmission time is multiplied by a preset penalty coefficient and used to replace the original transmission time in the calculation of the switching net benefit. That is, the switching net benefit equals the corrected stay time minus the target processing time minus the product of the transmission time and the preset penalty coefficient. If the difference in number is greater than 1, it indicates that at least one processing stage has been fully executed since the last migration, the cooling-off period ends, the transmission time is counted according to the actual value, and no preset penalty coefficient is applied.

[0070] The default penalty coefficient is set to 2. In addition to the network time consumed by data transmission itself, continuous migration requires the target node to rebuild its task execution context, reallocate memory pages and processor time slices, and the cluster scheduler to synchronize the node task mapping table. These implicit overheads are typically on the same order of magnitude as the time required for a complete data transmission in actual testing. A default penalty coefficient of 2 effectively offsets these implicit overheads into the transmission cost, ensuring that the net benefit of switching during the cooldown period is only positive if the actual benefit exceeds twice the transmission cost, thus effectively preventing repeated migrations within short intervals. This coefficient can be adjusted within the range of 1.5 to 2.5 based on the ratio of task context reconstruction time to transmission time measured in actual cluster testing.

[0071] Migration is only performed when the net benefit of switching is positive. That is, when the correction time exceeds the sum of the target processing time and the transmission time, the time savings brought by the migration will outweigh the time consumption of the migration itself, ensuring that each migration shortens the total processing time of the task. Block verification transmission and target node status monitoring jointly maintain the atomicity of migration. The task is either completely migrated to the target node to continue execution or rolled back to the current node. There is no intermediate state of incomplete data or repeated execution of stages. Cooling constraints increase the cost threshold of continuous migration, achieving a balance between scheduling flexibility and task execution continuity, and realizing precise scheduling of computing resources in the process of visualization modeling.

[0072] According to a second aspect of this application, this application also provides a data processing system for visualization modeling. Figure 4 This is a structural block diagram of a data processing system for visualization modeling according to an embodiment of this application. Figure 4 As shown, the system 50 includes a processor and a memory. The memory stores computer program instructions, which, when executed by the processor, implement the data processing method for visualization modeling according to the first aspect of this application. The system also includes other components well-known to those skilled in the art, such as a communication bus and a communication interface. Their configuration and functions are known in the art and will not be described further here.

[0073] It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of this application, and these all fall within the scope of protection of this application.

Claims

1. A data processing method for visual modeling, characterized in that, The method includes: S101, Perform ordered sample clustering on the historical resource occupancy time series of the target task, divide it into several processing stages, and extract the resource demand vector, occupancy rate dispersion, and unit data baseline processing time for each processing stage; S102, In response to the current processing stage reaching a preset progress threshold, collect the real-time resource idle rate of the current node, amplify the resource demand vector of the next processing stage with the occupancy rate dispersion of the next processing stage to obtain a demand vector with surplus, and calculate the ratio of the real-time resource idle rate to the demand vector with surplus in each dimension to obtain the resource surplus ratio. If there is a resource surplus ratio less than 1, it is determined that resources are tight; S103 When resources are scarce, the bottleneck margin ratio is determined based on the minimum value of the resource margin ratios. The standby processing time is obtained by calculating the unit data baseline processing time of the next processing stage based on the bottleneck margin ratio. The standby processing time is corrected by the difference in resource demand between two adjacent processing stages to obtain the corrected standby time. S104, candidate nodes with resource margin ratios of not less than 1 in each dimension are selected. The target processing time and transmission time are calculated for each candidate node. The net switching benefit is obtained by subtracting the target processing time and transmission time from the corrected standby time. The candidate node with the largest and positive net switching benefit is selected as the target node to perform task migration.

2. The data processing method for visualization modeling according to claim 1, characterized in that, After the task migration, the method further includes: The target node is taken as the new current node, and the current processing stage of the task is called the current processing stage. When the task approaches each subsequent processing stage, the actions of collecting the real-time resource idle rate, determining resource shortage, calculating the corrected stay time, and calculating the switching net benefit are repeated in sequence until the task enters the last processing stage.

3. The data processing method for visualization modeling according to claim 1, characterized in that, The process, which is divided into several processing stages, includes: The number of processing stages is increased sequentially from the preset initial number. After each increase, the decrease of the sum of squared deviations within the group of all processing stages relative to the previous number is calculated. When the decrease is lower than the preset proportion of the previous decrease for the first time, the number before the current increase is determined as the final number of processing stages.

4. The data processing method for visualization modeling according to claim 1, characterized in that, The process of obtaining the unit data benchmark processing time includes: For each processing stage, the historical execution records of the processing stage are divided into ample bins, moderate bins, and tense bins according to the resource idle rate of the node at the time of execution. The average unit data processing time of the processing stage in the ample bin is calculated as the unit data baseline processing time of the processing stage. The unit data processing time is the duration of the stage in the historical execution record divided by the amount of input data processed in the stage.

5. The data processing method for visualization modeling according to claim 1, characterized in that, The process of obtaining the demand vector with margin includes: Using the sum of 1 and the occupancy rate dispersion of the next processing stage as a safety margin coefficient, the resource demand vector of the next processing stage is multiplied by the safety margin coefficient to obtain the demand vector with margin.

6. The data processing method for visual modeling according to claim 1, characterized in that, The calculation process for the difference in resource requirements between two adjacent processing stages includes: Calculate the cosine similarity between the resource demand vector of the previous processing stage and the resource demand vector of the next processing stage in the two adjacent processing stages, and use the difference between 1 and the cosine similarity as the resource demand difference between the two adjacent processing stages.

7. The data processing method for visualization modeling according to claim 1, characterized in that, The calculation process for the corrected stay duration includes: The difference between resource requirements of 1 and the two adjacent processing stages is used as the difference correction factor. The stay-at-home processing time is multiplied by the difference correction factor to obtain the corrected stay-at-home time.

8. The data processing method for visual modeling according to claim 1, characterized in that, The process of determining the bottleneck margin ratio includes: comparing the minimum value of the resource margin ratio with a preset margin lower limit, and taking the larger of the minimum value and the preset margin lower limit as the bottleneck margin ratio.

9. A data processing method for visualization modeling according to claim 1, characterized in that, The process of migrating the execution task includes: The current node divides the intermediate data produced in the current processing stage of the task into several data blocks according to a preset block size, adds a checksum, and then transmits them to the target node. The target node checks each block, and requests retransmission of data blocks that fail the check. During transmission, the ratio of the real-time resource idle rate of the target node to the demand vector with surplus is calculated at preset intervals to obtain the resource surplus ratio of the target node. If there is a target node resource surplus ratio less than 1, the transmission is stopped and the task is rolled back to the current node to continue execution.

10. A data processing system for visual modeling, characterized in that, It includes a processor and a memory, the memory storing computer program instructions that, when executed by the processor, implement a data processing method for visualization modeling according to any one of claims 1 to 9.

Images