Control method, device and storage medium for data fusion compression

Abstract

This application discloses a control method, device, and storage medium for data fusion compression, relating to the field of data processing technology. The method includes: responding to a data compression command, determining candidate compression paths that match the inherent attribute characteristics of the data to be compressed and the current system's computing power status parameters; using the inherent attribute characteristics and computing power status parameters as independent variables of a decision function, determining the benefit score corresponding to each candidate compression path, and selecting the candidate compression path with the highest benefit score as the target execution path; during the compression process, executing resource scheduling actions based on the resource scheduling rules corresponding to the target execution path, and performing compression actions based on the target execution path to obtain a compressed file. This application solves the technical problem of low data compression efficiency by using dynamic path matching, decision function selection of the optimal compression path, and adaptive rule iteration, improving compression efficiency and computing power utilization, and achieving self-optimization of decision rules.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a control method, device and storage medium for data fusion compression. Background Technology

[0002] In scenarios involving massive data storage and transmission, the hardware adaptability and resource scheduling efficiency of file compression directly affect storage utilization and data transmission speed. Related technologies often employ fixed modes such as pure CPU compression or pure GPU compression, relying on the computing power of the corresponding hardware to complete data compression. This approach, which processes data according to a pre-defined fixed processing path, is prone to resource scheduling conflicts, leading to low data compression efficiency.

[0003] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main objective of this application is to provide a control method, device, and storage medium for data fusion compression, aiming to solve the technical problem of low data compression efficiency.

[0005] To achieve the above objectives, this application proposes a control method for data fusion compression, the method comprising:

[0006] In response to a data compression command, a candidate compression path is determined that matches the inherent attribute characteristics of the data to be compressed and the computing power status parameters of the current system.

[0007] The inherent attribute features and the computing power state parameters are used as independent variables of the decision function to determine the revenue score corresponding to each candidate compression path, wherein the decision function corresponds one-to-one with the candidate compression path;

[0008] The candidate compression path with the highest profit score is selected as the target execution path;

[0009] During the compression process, resource scheduling actions are performed based on the resource scheduling rules corresponding to the target execution path, and compression actions are performed based on the target execution path to obtain a compressed file.

[0010] In one embodiment, in response to the data compression command, an initial path that meets the attribute requirements and load requirements is determined based on the inherent attribute characteristics and the computing power status parameters;

[0011] Analyze the block continuity characteristics of the data to be compressed to determine the block adaptation parameters of the data to be compressed;

[0012] The initial path that matches the segmentation adaptation parameters is selected as the candidate compression path.

[0013] In one embodiment, the block continuity features of each segment in the data to be compressed are analyzed to obtain a block-level feature sequence;

[0014] Calculate the local information entropy of all block-level features in the block-level feature sequence, and arrange the local information entropy according to the order of the block-level features to obtain a local entropy value sequence;

[0015] Based on the local entropy value sequence, the entropy value difference between adjacent blocks is compared, the total proportion of consecutive blocks that meet the requirements is counted, and the block adaptation parameters of the data to be compressed are calculated.

[0016] In one embodiment, the inherent attribute features and the computing power state parameters are normalized according to the input feature rules to generate feature input parameters;

[0017] For each of the candidate compression paths, configure the corresponding revenue weight parameters according to the weight library;

[0018] The feature input parameters are input into the decision function, and the data adaptation benefit, computing load benefit, and transmission cost benefit of each candidate compression path are calculated by combining the benefit sub-item parameters of each candidate compression path.

[0019] The revenue score corresponding to the candidate compression path is obtained by weighting and summing the data adaptation revenue, computing load revenue, and transmission cost revenue of the candidate compression path according to the revenue weight.

[0020] In one embodiment, each candidate compression path is bound and paired with its corresponding revenue score to obtain a candidate revenue list;

[0021] Based on the refined decision markers in the candidate revenue list, the transmission cost of the corresponding candidate path is adjusted to obtain the revised candidate revenue list.

[0022] The revised candidate payout list is sorted from high to low according to the payout score to obtain an ordered candidate sequence;

[0023] The highest-scoring candidate path in the ordered candidate sequence is selected, and the candidate path corresponding to the highest-scoring candidate path is determined as the target execution path for this compression task.

[0024] In one embodiment, during the compression process, the resource scheduling rules corresponding to the target execution path are matched, and the appropriate compression algorithm is matched based on the redundancy of the data to be compressed.

[0025] The data to be compressed is preprocessed according to the resource scheduling rules. If it is a heterogeneous path, the data to be compressed is asynchronously transmitted to the heterogeneous video memory.

[0026] The computing unit corresponding to the target execution path is invoked to load the compression algorithm matching the data to be compressed, and the preprocessed data to be compressed is compressed to obtain segmented compressed data.

[0027] The segmented compressed data is standardized and a file checksum header is added to obtain the compressed file.

[0028] In one embodiment, the compression speed and compression ratio of this compression process are correlated with the integrity verification result of the compressed file to obtain the performance data of this compression process;

[0029] The performance data, feature parameters, and path selection structure of this compression process are stored in the historical task library, and the most recent valid historical task data are filtered out.

[0030] Based on the effective historical task data, adaptive optimization of decision parameters is performed, and the updated decision parameters are synchronously updated to the decision rule base.

[0031] In one embodiment, the performance comparison of the two types of paths and the failure rate of heterogeneous paths are statistically analyzed from the effective historical task data, and the boundary constraints are adjusted according to the threshold optimization rules to obtain the target decision threshold.

[0032] Based on the effective historical task data and weight optimization rules, the target weight parameters are iteratively calculated, and the triggering rules for refined decision markers are updated.

[0033] The target decision threshold, target weight parameters, and updated refined decision marker trigger rules are updated in the rule base to optimize the next compression task.

[0034] In addition, to achieve the above objectives, this application also proposes a data fusion compression device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the control method for data fusion compression as described above.

[0035] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the data fusion compression control method described above.

[0036] This application provides a control method for data fusion compression, which includes matching candidate compression paths by combining the inherent attribute characteristics of the data to be compressed with the computing power status parameters of the current system when responding to a data compression command, using the two types of characteristics as independent variables of a decision function that corresponds one-to-one with the candidate paths, calculating the benefit score corresponding to each candidate path to quantify the adaptability of the path to the current task, selecting the candidate path with the highest benefit score as the target execution path, and finally executing resource scheduling and compression actions according to the resource scheduling rules corresponding to the target execution path. This technical solution solves the technical problems of rigid path selection, fixed parameters that cannot adapt to dynamic tasks and computing power status, computing power resource conflicts, and insufficient decision-making accuracy in heterogeneous compression scenarios, and improves the path decision accuracy, computing power resource utilization, and overall compression execution efficiency of compression tasks under heterogeneous computing power.

[0037] In summary, this application solves the technical problem of low data compression efficiency by using dynamic path matching, decision function selection of the optimal compression path, and adaptive rule iteration, thereby improving compression efficiency and computing power utilization, and realizing self-optimization of decision rules. Attached Figure Description

[0038] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0039] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart illustrating the first embodiment of the control method for data fusion compression in this application;

[0041] Figure 2 This is a data compression framework diagram for this application;

[0042] Figure 3 This is a flowchart illustrating the second embodiment of the control method for data fusion and compression in this application;

[0043] Figure 4 This is a flowchart illustrating the third embodiment of the data fusion and compression control method of this application;

[0044] Figure 5 This is a flowchart illustrating the sixth embodiment of the control method for data fusion and compression in this application;

[0045] Figure 6 This is a schematic diagram of the data fusion and compression device used in this application.

[0046] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0047] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0048] In related technologies, data compression is performed by providing a fixed mode of pure CPU compression or pure GPU compression, relying on the computing power of the corresponding hardware. This method processes data according to a preset fixed processing path, which can easily cause resource scheduling conflicts, resulting in low data compression efficiency.

[0049] This application provides a solution: First, in response to a data compression command, candidate compression paths matching the inherent attribute characteristics of the data to be compressed and the computing power status parameters of the current system are determined. Then, the inherent attribute characteristics and the computing power status parameters are used as independent variables of a decision function to determine the revenue score corresponding to each candidate compression path. The decision function corresponds one-to-one with each candidate compression path. The candidate compression path with the highest revenue score is then selected as the target execution path. Finally, during the compression process, resource scheduling actions are performed based on the resource scheduling rules corresponding to the target execution path, and compression actions are performed based on the target execution path to obtain a compressed file.

[0050] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or data fusion compression device capable of performing the above functions. The following description uses a data fusion compression device as an example to illustrate this embodiment and the subsequent embodiments.

[0051] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0052] This application provides a data fusion compression control method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the control method for data fusion compression in this application.

[0053] In this embodiment, the data fusion compression control method includes steps S10 to S40:

[0054] Step S10: In response to the data compression command, determine the candidate compression path that matches the inherent attribute characteristics of the data to be compressed and the computing power status parameters of the current system.

[0055] Data compression instructions are the core control instructions that trigger compression tasks. The inherent attributes of the data to be compressed are intrinsic indicators describing its size and redundancy. Examples include total file size, data entropy, content divisibility, data redundancy, data block continuity, and data format type. The current system's computing power status parameters are operational indicators describing the real-time load and transmission status of general and heterogeneous computing power within the system. Examples include the real-time load rate of the CPU, the real-time load rate of the GPU, the bandwidth utilization of the high-speed serial computer expansion bus link, GPU memory utilization, CPU memory utilization, the number of available CPU cores, and the number of available GPU streaming multiprocessors.

[0056] In this embodiment, the aforementioned data compression instructions can be triggered in six ways. First, a scheduled task trigger: the system automatically generates and issues data compression instructions at a specified time point according to a preset periodic scheduling rule, used to perform routine historical data archiving and compression tasks. Second, a data volume threshold trigger: when the cumulative storage scale of the data to be compressed reaches a preset capacity threshold, the system automatically generates and issues data compression instructions to control the scale of storage resource usage. Third, user-initiated trigger: the user initiates a compression request through the system's interactive interface or inputs a compression execution command through a command-line tool; the system receives the request and generates and issues data compression instructions. Fourth, data write completion trigger: when a single complete file or batch dataset is completely written to the system's storage medium, the system automatically generates and issues data compression instructions to perform immediate compression processing on the newly written raw data. Fifth, system resource idle trigger: when the system detects that the real-time load of both the central processing unit and the graphics processing unit is lower than a preset idle threshold, it automatically generates and issues background data compression instructions to utilize the system's idle computing power to perform non-urgent compression tasks. Sixthly, the data transmission is triggered before data transmission. When the system receives a request for cross-node data transmission or off-site backup, it automatically generates and issues a data compression command, compresses the data to be transmitted before starting the transmission process.

[0057] After receiving the data compression instruction, the data fusion compression device begins to collect the inherent attribute characteristics of the data to be compressed, as well as the current computing power status parameters of the system.

[0058] For example, the inherent attribute characteristics of the data to be compressed and the current system's computing power status parameters are collected in two ways. The first is a pre-scanning full-volume synchronous collection. Before the compression task officially starts, the complete metadata of the file to be compressed is read to obtain the total file size and data format type. Then, a global content scan is performed on the file, calculating the data entropy value block by block, analyzing the continuity and distribution characteristics of the content, and statistically obtaining the inherent attribute characteristics of the data to be compressed. At the same time, the operating system kernel interface and the graphics processor driver interface are called synchronously to collect all computing power status parameters at once, including the real-time load rate of the central processing unit, the number of available cores, the memory usage rate, the real-time load rate of the graphics processor, the video memory usage rate, the number of available stream multiprocessors, and the bandwidth utilization rate of the high-speed serial computer expansion bus link. After the collection is completed, the parameters are uniformly output to the decision module. This method has a simple and stable collection logic, is not prone to data loss or statistical errors, and can provide a comprehensive and reliable basis for candidate path selection.

[0059] The second method is streaming incremental asynchronous acquisition. During data preprocessing and fragmented transmission, incremental attribute acquisition is performed while data is being read. The entropy and local continuity of the current data block are calculated block by block, and the global data redundancy and content divisibility are accumulated. After the data reading is completed, the inherent attribute characteristics of the data to be compressed are summarized. At the same time, the system computing power status is sampled asynchronously at fixed time intervals. Multiple sets of CPU, GPU load and link bandwidth data are continuously collected, and the average value within the sampling period is taken as the current system's stable computing power status parameter. At the same time, sudden load peaks during the sampling process are marked to provide risk reference for subsequent path decisions. This method is executed synchronously with the data preprocessing and fragmented transmission process, without requiring additional preprocessing time, and can make full use of pipeline idle periods to complete the acquisition.

[0060] Once the inherent attributes and system computing power status are obtained, an initial path that matches them can be selected as a candidate compression path based on the following methods.

[0061] In one alternative approach, all preset initial compression paths are first retrieved, and the adaptation threshold for each initial path is extracted. Then, the inherent attribute characteristics of the collected data to be compressed and the current system computing power status parameters are compared item by item with the adaptation threshold of each initial path. Initial paths whose inherent attribute characteristics and computing power status parameters all meet the adaptation threshold requirements are selected, while those that do not meet the adaptation conditions are eliminated. The final retained initial paths are the candidate compression paths. This method relies on preset paths and fixed adaptation thresholds for filtering, and its logic is intuitive, computationally inefficient, and fast.

[0062] In another alternative approach, the inherent attributes of the data to be compressed are first decomposed from multiple dimensions to extract core features such as total file size, data entropy, content divisibility, and data block continuity. Simultaneously, the current system's computing power status parameters are analyzed hierarchically to extract core operational metrics such as the number of available CPU cores, CPU real-time load rate, the number of available GPU stream processors, GPU real-time load rate, and PCIe link bandwidth utilization. Then, based on the inherent attributes of the data, suitable fragmentation granularity strategies, compression algorithm combinations, and data preprocessing rules are generated. Based on the system's computing power status parameters, suitable resource allocation ratios, the number of parallel processing instances, and transmission scheduling strategies are generated. These strategies and rules are then legally combined to generate multiple initial compression paths. Subsequently, a feasibility check is performed on all generated initial compression paths. The check includes whether the required computing resources for the path are within the current system's available resources, whether the algorithm combination of the path supports the format type of the data to be compressed, and whether the transmission scheduling strategy of the path complies with the current link bandwidth limitations. All infeasible initial compression paths are eliminated, and the remaining feasible initial compression paths are the candidate compression paths. This method can flexibly generate personalized compression paths that fit the current task, with richer strategy combinations and higher adaptation accuracy. It can fully tap the system's computing power potential and data compression characteristics, and improve the overall compression performance.

[0063] In an exemplary scheme for determining candidate compression paths, a pre-set set of compression paths is first loaded. This set includes two basic paths: CPU multi-threaded compression paths and GPU parallel compression paths. Each path is pre-configured with a corresponding adaptation threshold matrix and resource requirement list. Then, a preliminary screening based on basic thresholds is performed. The total file size of the data to be compressed is compared with a pre-set file size threshold range, the real-time GPU load rate is compared with a pre-set GPU load threshold, and the real-time CPU load rate is compared with a pre-set CPU load threshold. Initial paths that simultaneously meet the data attribute threshold requirements and computing power status threshold requirements are selected. Next, for GPU paths that pass the preliminary screening, the GPU memory utilization rate is checked to see if it is lower than a pre-set memory safety threshold, the high-speed serial computer expansion bus link bandwidth utilization rate is checked to see if it is lower than a pre-set bandwidth safety threshold, and the number of available GPU stream processors is checked to see if it meets the minimum parallel processing requirements. For CPU paths that pass the preliminary screening, the CPU memory utilization rate is checked to see if it is lower than a pre-set memory safety threshold, and the number of available CPU cores is checked to see if it meets the minimum multi-threaded scheduling requirements. Next, a format compatibility check is performed to verify whether the compression algorithm supported by each path is compatible with the format type of the data to be compressed. Paths that do not support the target data format are eliminated. Then, conflict path elimination is performed. If there are multiple paths of the same type that only differ in resource allocation ratio, the one with the resource allocation ratio that best matches the current computing power status is retained, and the remaining redundant paths are eliminated. Finally, all initial paths that have passed all the above checks are summarized to generate the final candidate compression paths, which are used for subsequent revenue calculation and selection of the optimal execution path.

[0064] It should be noted that in some special cases, if there is only one initial path that matches the inherent attributes of the data to be compressed and the system's computing power status, then the process of selecting the target execution path from the candidate compression paths is skipped, and it is directly used as the target execution path.

[0065] Step S20: Using the inherent attribute features and the computing power state parameters as independent variables of the decision function, determine the revenue score corresponding to each candidate compression path, wherein the decision function corresponds one-to-one with the candidate compression path.

[0066] The decision function is a pre-defined computational model used to quantify the overall suitability of candidate compression paths for this task. The independent variables are the basic input parameters fed into the decision function to calculate the benefit score. The benefit score is a quantitative indicator used to measure the overall benefit of candidate compression paths performing this compression task. For example, independent variables include total file size, data entropy value, content divisibility, CPU real-time load rate, GPU real-time load rate, and PCIe link bandwidth utilization.

[0067] In this embodiment, the inherent attribute features and computing power status parameters are used as independent variables of the decision function and input into the decision function corresponding to the candidate compression path to start the revenue calculation process of each candidate compression path.

[0068] For example, the calculation of the revenue score for each candidate compression path includes two methods. The first method is a serial, single-path, dimension-by-dimensional calculation. After the revenue score calculation process officially starts, the type information of each candidate compression path is extracted one by one according to their priority. Then, the inherent attribute characteristics and computing power status parameters are input into the decision function corresponding to the path, and the data adaptation revenue, computing power load revenue, and transmission cost revenue of the candidate compression path are calculated sequentially. After the sub-item revenue calculation of a single path is completed, a weighted sum is performed to obtain the final revenue score of the path. After the calculation of one path is completed, the calculation of the next candidate path begins. After the calculation of all paths is completed, the results are uniformly output to the decision module. This method has a simple and stable calculation logic, and the calculation process of each path is independent and controllable, making it less prone to calculation deviations and providing a reliable basis for determining the optimal path selection.

[0069] The second approach involves dimensional splitting and full-path batch parallel computation. First, the inherent attribute features and computing power status parameters of the input are split dimensionally, extracting feature subsets for data adaptation, computing power load, and transmission cost dimensions. Simultaneously, all candidate paths are categorized and aligned according to these dimensions, and parallel computation across all dimensions is initiated concurrently. Within each dimension, the corresponding sub-item revenue for all candidate paths is calculated simultaneously. After the sub-item revenue calculations for all dimensions are completed, a weighted summation is performed on all candidate paths to obtain the final revenue score for each path in batches. Abnormal paths during the calculation process are also marked to provide risk references for subsequent path selection. This method executes synchronously with the multi-dimensional parallel processing flow, eliminating the need for additional serial computation time. It fully utilizes idle system computing power to complete batch computations, significantly improving the overall efficiency of revenue score calculation.

[0070] Before calculating the profit score, the decision function for each candidate compression path can be determined based on the following method.

[0071] In one alternative approach, a pre-set set of fixed decision functions is invoked. This set pre-stores standardized decision functions adapted to different path types and data formats. Based on the type of the candidate compression path and the format of the data to be compressed, the corresponding pre-set decision function is directly matched and invoked, and then used directly for subsequent revenue calculation. This method is logically simple and stable, with unified and controllable function rules. It requires no additional adaptation calculations, can quickly determine the decision function, and is suitable for conventional compression scenarios with high requirements for decision response speed.

[0072] In another alternative approach, the core adaptation parameters of candidate compression paths are first extracted, along with the inherent attribute features of the data to be compressed and the current system's computing power status parameters. Then, decision rules optimized based on historical tasks are retrieved from the rule base. Based on the inherent attribute features of the data to be compressed and the current system's computing power status parameters, a decision function adapted to the current task is dynamically generated. The feasibility of the generated decision function is verified, confirming that the function's input and output range meets the current system's computational requirements. After verification, this function is used as the decision function for calculating the current revenue score. This method can flexibly generate customized decision functions that fit the current task and computing power status, without relying on fixed preset templates. It offers higher adaptation accuracy, fully leverages the characteristics of system computing power and data compression, improves the accuracy of revenue score calculation, and is suitable for complex compression scenarios such as large tasks and high loads.

[0073] In an exemplary scheme for determining the revenue score, the decision function set corresponding to all candidate paths is first loaded. This set includes two types of basic functions: CPU path decision functions and GPU path decision functions. Each function is pre-configured with corresponding weight parameters and sub-item calculation rules. Then, input parameter normalization is performed, standardizing the collected inherent attribute features and computing power status parameters to eliminate differences in the dimensions of different parameters. Next, for each candidate compression path, the normalized parameters are input into the corresponding decision function. First, the data adaptation sub-item revenue of the candidate compression path is calculated, then the computing power load sub-item revenue is calculated, and finally the transmission cost sub-item revenue is calculated. After completing the sub-item revenue calculation, a weighted sum is performed based on the weight parameters to obtain the preliminary revenue score for the path. Then, a revenue score validity check is performed to verify whether the revenue score of each path is within a reasonable range. Abnormal revenue score results are eliminated, and redundant results are excluded. If there are cases where the difference in revenue scores between paths of the same type is less than a preset threshold, the one with the highest accuracy is retained, and the remaining redundant results are eliminated. Finally, all the verified profit scores are aggregated to generate the final profit score set for each candidate path, which is used for the selection of the optimal execution path in the subsequent process.

[0074] Step S30: Select the candidate compression path with the highest benefit score as the target execution path.

[0075] In this embodiment, the target execution path can be selected in two ways. One is to sequentially sort and filter in sequence. First, all candidate compression paths are sorted from high to low according to their benefit scores. Then, the real-time availability of each path is verified in descending order. The path that passes the verification and is ranked first is taken as the target execution path. There is no need to verify the remaining candidate paths, and the target path is selected quickly. This method is simple and stable in logic, and the verification process is carried out step by step. It can quickly lock the optimal path and is suitable for conventional compression scenarios.

[0076] Secondly, the parallel batch screening of the entire path first performs parallel verification of the benefit score and real-time availability of all candidate compression paths, completes the validity marking of all paths in batches, and then directly selects the path with the highest benefit score among all valid paths as the target execution path. This method can make full use of parallel computing power to complete batch verification, avoid the time consumption of serial verification, and comprehensively evaluate the status of all paths to avoid missed detections, adapting to complex compression scenarios with large tasks and high loads.

[0077] Step S40: In the compression process, a resource scheduling action is performed based on the resource scheduling rules corresponding to the target execution path, and a compression action is performed based on the target execution path to obtain a compressed file.

[0078] Resource scheduling rules are pre-configured execution rules for the target execution path, used to schedule computing and storage resources. Resource scheduling actions are execution actions that allocate and initialize computing and storage resources according to the scheduling rules. Compression actions are core business actions that perform data compression processing based on the characteristics of the target execution path. For example, resource scheduling rules include CPU core allocation rules, GPU streaming multiprocessor allocation rules, and PCIe transmission scheduling rules. Resource scheduling actions include computing core binding, memory and GPU memory allocation, and algorithm instance initialization; compression actions include CPU multi-threaded compression, GPU parallel compression, and fragmented data transmission.

[0079] In this embodiment, resource scheduling rules can be invoked in two ways. One is to directly invoke a preset fixed resource scheduling rule, which retrieves the system's pre-stored set of resource scheduling rules, and directly matches and retrieves the corresponding resource scheduling rule based on the type of the target execution path, and directly uses it for the current resource scheduling action. This method has simple and stable invocation logic, unified and controllable rules, fast invocation speed, and is suitable for common compression scenarios.

[0080] Secondly, the dynamic adaptation rule generation and invocation method first extracts the core parameters of the target execution path, and at the same time extracts the inherent attribute features of the data to be compressed and the current computing power status parameters of the system. Then, it calls the decision rules optimized based on historical tasks in the rule base to dynamically generate resource scheduling rules adapted to the current task. After completing the feasibility verification of the rules, the resource scheduling rules are called for the current resource scheduling action. This method can flexibly generate customized rules that fit the current task, with higher adaptation accuracy, can fully tap the potential of the system's computing power, improve the accuracy of resource scheduling, and adapt to complex compression scenarios with large tasks and high loads.

[0081] Once the runtime resources are configured through resource scheduling, the compression action is performed on the data to be compressed according to the target execution path to obtain a compressed file.

[0082] In an exemplary scheme for performing resource scheduling and compression, the resource scheduling rule set corresponding to the target execution path is first loaded. This set includes two basic rule categories: CPU path scheduling rules and GPU path scheduling rules. Each rule is pre-configured with corresponding resource allocation parameters and execution flow. Then, resource allocation is performed. For the GPU path, this involves allocating GPU streaming multiprocessors, video memory, and PCIe transmission channels. For the CPU path, this involves allocating CPU cores and memory. Next, algorithm instance initialization is performed, loading the compression algorithm adapted to the target execution path and initializing parallel processing instances. Data preprocessing is then performed, dividing and regularizing the data to be compressed according to the fragmentation rules of the target execution path. Fragment transmission is then performed: for the GPU path, fragmented data is asynchronously transmitted to GPU video memory; for the CPU path, fragmented data is loaded into CPU memory. Compression processing is then performed: for the GPU path, multi-core parallel compression is initiated; for the CPU path, multi-threaded parallel compression is initiated. After all fragment compression is complete, compressed file standardization is performed, concatenating the fragment compression results, adding a checksum header, and generating the final compressed file.

[0083] For example, refer to Figure 2 , Figure 2 This is a data compression framework diagram for this application. In the scenario of data fusion compression, for the compression task of a 20GB structured log input file, after responding to the data compression command, the process first passes through the feature acquisition layer, which simultaneously completes file feature extraction (size 20GB, type structured text, entropy value 3.2, divisibility 92%) and system status acquisition (CPU and GPU loads are 28% and 35% respectively, PCIe link bandwidth 32GB / s). Then, it enters the core decision engine layer, where the rule engine performs multi-dimensional rule verification, selecting two candidate compression paths: the CPU path and the GPU path. The decision function constructed by the decision model then calculates the CPU path's benefit score (68 points) and the GPU path's benefit score (92 points). After benefit score calibration and sorting, the GPU path with the highest benefit score is determined as the target execution path. Subsequently, it enters the compression execution layer, matching the resource scheduling rules corresponding to the target path and adapting the compression algorithm to complete data preprocessing, heterogeneous path memory transfer, and parallel compression, generating segmented compressed data. After standardization and adding a file verification header, the compressed file is output. Finally, the feedback optimization layer records the feature data, decision results, total process time (18.12s), and compression ratio (12.3:1) of this task. After accumulating 100 valid tasks, the decision threshold and weight are iteratively optimized and updated synchronously to the decision engine layer.

[0084] By employing a fully closed-loop architecture encompassing feature acquisition, decision engine, compression execution, and feedback optimization, the core issues of rigid path selection and inability to dynamically adapt to data features and computing power status in data fusion and compression scenarios are resolved, thereby improving compression execution efficiency and the utilization rate of heterogeneous computing resources.

[0085] Second Embodiment

[0086] This embodiment provides an exemplary scheme for determining candidate compression paths. In this example, an initial screening is performed based on inherent attribute characteristics and computing power status parameters, followed by a secondary screening based on the block continuity characteristics of the data to be compressed, thereby finally selecting a candidate compression path suitable for this compression task. (Refer to...) Figure 3 , Figure 3 This is a flowchart illustrating the second embodiment of the data fusion and compression control method of this application. Step S10 includes steps A11 to A13:

[0087] Step A11: In response to the data compression command, determine an initial path that meets the attribute requirements and load requirements based on the inherent attribute characteristics and the computing power status parameters.

[0088] A12: Analyze the block continuity characteristics of the data to be compressed and determine the block adaptation parameters of the data to be compressed.

[0089] A13: Select the initial path that matches the segmentation adaptation parameter as the candidate compression path.

[0090] In this example, when initially screening the initial compression paths in the database based on inherent attribute characteristics and computing power status parameters, the method provided in the first embodiment can be referred to. Alternatively, based on the inherent attribute characteristics of the data to be compressed and the current system's computing power status parameters, a suitable initial compression path can be dynamically generated. After verifying the feasibility of all generated paths, an initial path that meets the attribute requirements and load requirements is obtained, thus completing the initial screening.

[0091] After initial path selection, the block continuity feature analysis process for the data to be compressed is initiated. This involves reading the local entropy sequence of the data, performing global feature distribution analysis and clustering, dividing the data into continuous feature intervals, calculating the total size of continuous blocks that meet the continuity requirements, and determining the proportion of this total size in the total block size of the data to be compressed. Based on this proportion, the block fit parameters of the data to be compressed are calculated using preset rules. Subsequently, the block fit parameters are compared item by item with the block fit thresholds of all initial paths. Initial paths matching the block fit parameters are selected, while those that do not meet the block fit requirements are removed. Finally, the selected initial paths are used as candidate compression paths for this compression task. This two-stage hierarchical selection improves the fit accuracy of candidate paths and avoids a decrease in compression efficiency due to block mismatch.

[0092] For example, in a data fusion compression scenario, using a 12GB binary model weight file from an AI training platform, after responding to a data compression command, real-time monitoring and data preprocessing are first performed through a feature acquisition layer: the file size of 12GB is read from the file system, identified as a scientific computing file by the magic number in the file header, and the average entropy value of the file is calculated to be 0.6 using a 1KB sliding window. CPU utilization of 22% and GPU utilization of 26% are collected through the system interface and NVIDIA NVML, respectively, and the real-time transmission bandwidth of 36GB / s is monitored by PCIe performance counting. Subsequently, based on the collected inherent attribute features and computing power status parameters, two initial paths, CPU and GPU, that meet the attribute load requirements are verified and determined, while simultaneously triggering the refined decision-making of the rule engine. The file is then divided into 64KB blocks of a fixed size, the local entropy value of each block is calculated, and the proportion of continuous blocks is statistically analyzed by comparing the entropy difference between adjacent blocks, resulting in a block adaptation parameter K value of 0.94. Finally, feature rule matching is performed based on this K value, and two candidate compression paths, CPU and GPU, that meet the adaptation requirements are selected.

[0093] By implementing a progressive process of full feature collection, initial path verification, block quantification, and rule-based path selection, the core problems of rigid path selection and lack of block adaptation quantification basis in data fusion and compression scenarios are solved, thereby improving the compressibility of compression paths and the efficiency of parallel compression execution.

[0094] Third Embodiment

[0095] This embodiment provides an exemplary scheme for determining block adaptability parameters. In this example, the block continuity characteristics of each segment in the data to be compressed are first analyzed, and a preliminary sorting is performed based on the block continuity characteristics. Then, entropy calculation and continuity verification are performed based on the sorted feature sequence, thereby finally accurately calculating the block adaptability parameters suitable for this compression task. (Refer to...) Figure 4 , Figure 4This is a flowchart illustrating the third embodiment of the data fusion and compression control method of this application. Step A12 includes steps B11-B13:

[0096] Step B11: Analyze the block continuity features of each segment in the data to be compressed to obtain a block-level feature sequence.

[0097] Step B12: Calculate the local information entropy of all block-level features in the block-level feature sequence, and arrange the local information entropy according to the order of the block-level features to obtain a local entropy value sequence.

[0098] Step B13: Based on the local entropy value sequence, compare the entropy value differences of adjacent blocks, count the total proportion of consecutive blocks that meet the requirements, and calculate the block compatibility adaptation parameters of the data to be compressed.

[0099] Segmented content refers to multiple data units with independent data boundaries obtained by splitting the data to be compressed according to preset splitting rules. It is the smallest processing unit for block continuity feature parsing. Block continuity features are a set of core attributes representing the content correlation, information distribution consistency, and feature similarity between adjacent segments. They are the core basis for characterizing the adaptability of parallel block processing of data. For example, the entropy fluctuation of adjacent data blocks, the difference in content redundancy, the similarity of feature vectors, the continuity of format, and the adaptability for parallel splitting. Local information entropy is a core quantitative indicator used to quantify the uniformity of information distribution, the degree of content disorder, and the level of redundancy within a single segment. For example, the character distribution entropy value of text segments, the byte distribution entropy value of binary segments, the field distribution entropy value of structured data segments, the content distribution entropy value of unstructured data segments, and the pixel distribution entropy value of image data segments.

[0100] In this example, when parsing the block continuity features of the data to be compressed to obtain the block-level feature sequence, the global content scanning method provided in the second embodiment can be used. Alternatively, a streaming incremental reading method can be used to read the content of each segment of the data to be compressed while parsing the block continuity features of each segment, accumulating the block-level feature sequence to complete the acquisition of block-level features.

[0101] After collecting the block-level feature sequence, the local information entropy calculation process is initiated. For each block-level feature in the block-level feature sequence, the local information entropy of each block is calculated sequentially. Then, according to the original order of the block-level features, all local information entropies are arranged to obtain a local entropy value sequence. Next, based on the local entropy value sequence, the entropy value differences of adjacent blocks are compared and the number of all adjacent blocks that meet the preset continuity requirements is counted. The total size of the consecutive blocks that meet the requirements is calculated as a percentage of the total size of all blocks in the data to be compressed. Based on this percentage, the block-specific adaptation parameters of the data to be compressed are calculated according to preset rules. In this way, through hierarchical feature analysis and statistics, the calculation accuracy of the block-specific adaptation parameters is improved, and path matching deviations caused by insufficient feature analysis are avoided.

[0102] For example, there are two methods for calculating the block compatibility adaptation parameters of the data to be compressed using the local entropy value sequence. The first method involves verifying each data block according to the storage time sequence. Based on the original data storage time sequence bound to the local entropy value sequence, starting from the beginning of the sequence, two adjacent entropy values ​​are selected sequentially for difference comparison. After each set of adjacent entropy value comparisons is completed, it is simultaneously determined whether the corresponding adjacent blocks meet the preset continuity requirements. If they meet the requirements, the corresponding data block is added to the set of consecutive blocks that meet the requirements, and the total data size of the consecutive blocks is continuously accumulated. If they do not meet the requirements, the accumulation process of the current consecutive block is terminated, and the determination and accumulation of the next consecutive block begins. After all adjacent block pairs have been compared and determined, the total data size of all consecutive blocks that meet the requirements is summarized and statistically analyzed, and the proportion of this total data size in the total block size of the data to be compressed is calculated. Based on this total proportion, the block compatibility adaptation parameters of the data to be compressed are calculated according to preset rules. This method employs a time-locked, pairwise serial comparison and breakpoint-based dynamic accumulation calculation logic. By matching the original data time sequence with successive comparisons and dynamic accumulation of continuous blocks, it avoids the problem of mismatch across time-series blocks and restores the actual continuous distribution of the data content to be compressed.

[0103] The second method is continuous interval parallel verification. It reads the local entropy value sequence and analyzes the global feature distribution. Based on the overall distribution characteristics of the entropy values ​​in the sequence, it performs clustering processing, dividing entropy values ​​that are within the same preset fluctuation range and are temporally continuous into the same continuous feature interval. It also marks the non-continuous boundary nodes between intervals, generating multiple continuous feature intervals that are computationally independent of each other. Then, parallel verification processing is simultaneously initiated for all the divided continuous feature intervals. Within each continuous feature interval, the entropy value differences of all adjacent blocks within the interval are compared one by one to verify whether all adjacent blocks within the interval fully meet the preset continuity requirements. The total size of data blocks within intervals that meet the full interval continuity requirement is then calculated. After all the parallel verification and statistics for all continuous feature intervals are completed, the total data size of all compliant continuous blocks is summarized, and the proportion of this total size in the total size of all blocks in the data to be compressed is calculated. Based on this proportion, the block adaptation parameters of the data to be compressed are calculated according to preset rules. This method employs a computational logic of global clustering interval partitioning and parallel verification of the entire interval. By pre-sorting the global entropy value distribution, it locks in the potential continuous content intervals of the data to be compressed in advance, reducing the number of invalid adjacent block comparisons.

[0104] For example, in a data fusion compression scenario, the data fusion compression of an 8GB binary perception model weight file for an autonomous driving platform first divides the data to be compressed into 131,072 time-series corresponding segments according to a fixed 64KB block rule. Core block features of each segment are extracted sequentially, and the block continuity features of adjacent segments are analyzed. These are then organized chronologically to obtain a complete block-level feature sequence. Subsequently, this block-level feature sequence is traversed, and the local information entropy of each segment is calculated using a 1KB sliding window. The entropy value is bound to the corresponding time sequence identifier, and after verification and arrangement, a local entropy value sequence matching the time sequence of the block-level feature sequence is generated. Finally, the absolute value of the entropy difference between adjacent blocks is less than 0.5 as the continuity criterion. The entropy differences of 131,071 groups of adjacent blocks in the sequence are compared, and the total size of the continuously compliant blocks is 7.52GB. Their proportion in the total file size is calculated to be 0.94, which is used as the block-level adaptation parameter for the data to be compressed.

[0105] By employing a progressive process of segmented feature analysis, local entropy quantification, and entropy difference comparison between adjacent blocks to evaluate divisibility, the problem of lack of quantitative basis for parallel compression adaptation capability and mismatch between path selection and data features in data fusion compression scenarios is solved, thereby improving the accuracy of compression path selection and the efficiency of parallel compression execution.

[0106] Fourth embodiment

[0107] This embodiment provides an exemplary scheme for determining the revenue score of candidate compression paths. In this example, standardized feature inputs are first generated by normalizing the inherent attributes and computing power status. Then, revenue weight configuration, multi-dimensional revenue sub-item calculation, and weighted summation are performed sequentially to finally accurately calculate the revenue score of each candidate path suitable for this compression task. Step S20 includes steps C11 to C14:

[0108] Step C11: Normalize the inherent attribute features and the computing power state parameters according to the input feature rules to generate feature input parameters.

[0109] Step C12: For each of the candidate compression paths, configure the corresponding revenue weight parameters according to the weight library.

[0110] Step C13: Input the feature input parameters into the decision function, and combine them with the revenue sub-item parameters of each candidate compression path to calculate the data adaptation revenue, computing load revenue, and transmission cost revenue of the candidate compression path.

[0111] Step C14: The data adaptation benefit, computing power load benefit, and transmission cost benefit of the candidate compression path are weighted and summed according to the benefit weight to obtain the benefit score corresponding to the candidate compression path.

[0112] Input feature rules are a pre-defined, unified processing rule system used to standardize the processing logic of multi-source heterogeneous features and eliminate differences in the dimensions and value ranges of features of different dimensions. The weight library is a set of standardized rules that centrally stores the revenue ratio relationships and pre-defined hierarchical parameters corresponding to various compression paths, providing a unified basis for parameter configuration. Revenue weight parameters are calibration parameters that characterize the influence of a single revenue dimension in the comprehensive evaluation, used to balance the strength of different evaluation dimensions in path selection. Revenue sub-item parameters are a pre-defined set of standardized parameters used to quantify the specific values ​​of each revenue dimension, corresponding to the three major revenue dimensions of data adaptation, computing power load, and transmission cost, and are the core basis for revenue calculation. Computing power load revenue is a revenue value that quantifies the rationality of the real-time load of the computing power unit corresponding to the candidate compression path and the resource utilization rate, reflecting the computing power utilization efficiency of the path. Revenue weights are pre-calibrated matching parameters used to define the proportion of influence of the three types of single-dimensional revenue in the comprehensive evaluation.

[0113] In this example, when performing normalization processing on the inherent attribute features and computing power status parameters according to the input feature rules, the pre-scanning full-scale synchronous acquisition method provided in the first embodiment can be used to complete the feature pre-acquisition before performing normalization processing. Alternatively, a streaming incremental reading method can be used to collect features while performing incremental normalization processing, accumulating the feature input parameters to complete the generation of feature input parameters.

[0114] After generating the feature input parameters, the configuration process for the revenue weight parameters is initiated. There are two configuration methods for revenue weight parameters. The first is a serial, line-by-line matching configuration. This method retrieves the identity identifiers of all candidate compression paths one by one, and sequentially traverses the parameter entries archived within the weight library according to a fixed search order, matching the basic revenue weight parameters corresponding to the path type, computing power attribute, and block processing capability. After completing the parameter binding for a single path, the search and configuration for the next path is initiated, thus completing a one-to-one revenue weight parameter attachment for all candidate compression paths. This method relies on the stable matching logic of serial, line-by-line retrieval, ensuring precise and controllable parameter binding, avoiding cross-path parameter confusion issues, and guaranteeing the uniqueness and compliance of the configuration parameters for each path.

[0115] The second method is batch configuration through categorization and grouping. First, based on the computing power architecture and block execution characteristics of the candidate compression paths, the entire domain is categorized and grouped, aggregating paths with similar operational attributes into unified groups. Then, a complete set of revenue weight parameters specific to each group is retrieved from the weight library, and batch parameters are synchronously mounted and configured on a group-by-group basis. After completing the batch configuration for each group, specific adjustment parameters are added and fine-tuned for individual differentiated paths, ultimately achieving the configuration of revenue weight parameters for all candidate compression paths. This method relies on categorization and grouping to achieve batch parallel configuration, reducing redundant processes caused by repeated searches and improving parameter configuration efficiency in multi-path scenarios.

[0116] After configuring the revenue weight parameters, the calculation process for the multi-dimensional revenue sub-items is initiated. There are two calculation methods for these sub-items: the first is a single-path, dimension-by-dimensional sequential calculation. Following the order of candidate compression paths, the multi-dimensional revenue calculation for each single path is initiated one by one. For a single candidate path, first, the feature dimensions related to data adaptation are extracted from the feature input parameters, and combined with the corresponding revenue sub-item parameters, the data adaptation revenue is quantified. Next, the feature dimensions related to computing power status are extracted from the feature input parameters, and combined with the corresponding revenue sub-item parameters, the computing power load revenue is calculated. Finally, the feature dimensions related to the transmission link are extracted from the feature input parameters, and combined with the corresponding revenue sub-item parameters, the transmission cost revenue is calculated. After all three revenue sub-items for a single path are calculated, the multi-dimensional calculation for the next candidate path is initiated. After all candidate paths have been calculated, the values ​​for the three revenue sub-items for each candidate path are summarized. This method employs a single-path, dimension-by-dimensional serial calculation logic. The calculation of the three major revenue sub-items of each path is completed independently without interference. This method can accurately control the calculation precision of each revenue dimension of a single path, avoid calculation interference between different paths and different revenue dimensions, and facilitate subsequent verification and adjustment of the calculation results of a single revenue dimension.

[0117] The second approach involves multi-dimensional parallel full-path batch computation. First, the feature input parameters are dimensionally split, extracting feature subsets corresponding to the three benefit dimensions: data adaptation, computational load, and transmission cost. Simultaneously, the benefit sub-item parameters of all candidate compression paths are read and categorized according to the three benefit dimensions, forming three independent benefit calculation datasets. Then, parallel computation is initiated synchronously for each of the three benefit dimensions. Within the computation process of each benefit dimension, the benefit calculation for that dimension is performed simultaneously on all candidate compression paths; that is, the data adaptation benefit, computational load benefit, and transmission cost benefit of all candidate paths are calculated simultaneously. After all parallel computations of the three dimensions are completed, the values ​​of the three benefit sub-items corresponding to each candidate path are associated and bound, summarizing to obtain the complete benefit sub-item data for each candidate path. This method employs multi-dimensional parallel, full-path batch computation logic, breaking the serial limitation of single-path, dimension-by-dimensional computation. Through dimensional splitting and parallel computation, it fully utilizes idle system computing resources, compresses the overall computation time in multi-path, multi-dimensional scenarios, and improves the efficiency of benefit sub-item calculation.

[0118] After calculating the revenue items for each dimension, a weighted summation calculation process is initiated. There are two methods for weighted summation: the first is a single-path, fully closed-loop, sequential calculation. For a single candidate compression path, the pre-configured revenue weight parameters for that path are retrieved, and the three revenue items—data adaptation revenue, computing power load revenue, and transmission cost revenue—are precisely matched and bound to their respective revenue weights. Then, according to the weighted impact percentage, the three revenue items are proportionally integrated and summarized. After completing the weighted summation of the revenue across all dimensions for that path, the calculation results are simultaneously validated. Once the results are confirmed to meet preset specifications, the final revenue score for that candidate compression path is generated. This identical process is then repeated sequentially for all remaining candidate compression paths, performing weighted calculations, validation, and revenue score generation. This method employs a single-path, fully closed-loop, sequential calculation logic, constructing an independent calculation and verification closed loop for each candidate compression path. The entire process of a single path is completely independent and does not interfere with each other. Result verification can be performed immediately after the calculation of a single path is completed, timely interception of abnormal calculation results, and prevention of invalid data from entering subsequent path selection stages, thereby improving the accuracy and reliability of the revenue calculation results.

[0119] The second method is dimensional aggregation and full-path batch parallel calculation. First, the three sub-items of the revenue and their corresponding weights of all candidate compression paths are aggregated and aligned across the entire domain according to the revenue dimension. The data adaptation revenue and corresponding weights of all paths are aggregated into the first calculation set, the computing power load revenue and corresponding weights of all paths are aggregated into the second calculation set, and the transmission cost revenue and corresponding weights of all paths are aggregated into the third calculation set. There is no computational dependency between the three calculation sets. Then, parallel weighted calculation is started simultaneously for the three calculation sets. Within each calculation set, the weight ratio calculation of the corresponding sub-items of the revenue of all candidate paths is completed simultaneously, and the weighted values ​​of each sub-item of all paths are obtained. After the parallel calculation of the three dimensions is completed, the weighted values ​​of the three dimensions of each candidate compression path are summarized and integrated, and the validity verification of the results of all paths is completed simultaneously. Finally, the final revenue score corresponding to each candidate compression path is generated at one time. This method adopts a dimension-based, full-path, batch-parallel accounting logic, breaking the timing limitations of single-path serial accounting. By splitting the data according to the revenue dimension, it achieves multi-path, multi-dimensional, dependency-free parallel accounting.

[0120] For example, in the scenario of fusion and compression of offline log data of user behavior on an Internet big data platform, for an 8GB uncompressed structured user behavior log file to be compressed, after responding to the data compression command, feature normalization processing is first performed. Preset input feature rules are read, and the inherent attribute features of the collected data to be compressed (file size S=8GB, file entropy H=0.42, divisibility K=0.93) and system computing power status parameters (GPU real-time load G=0.22, CPU real-time load C=0.28, PCIe real-time bandwidth B=38GB / s) are standardized and normalized to eliminate the differences in the units and value ranges of features of different dimensions, map them to a unified benchmark range, and generate feature input parameters that conform to the input specifications of the decision function. Subsequently, for the two candidate compression paths—CPU general computing power and GPU heterogeneous computing power—selected through pre-defined rules, a pre-calibrated weight library was retrieved to configure exclusive benefit weight parameters for each path. Specifically, the GPU path was configured with GPU adaptation weight α=0.45, entropy weight β=0.35, and bandwidth weight γ=0.2, while the CPU path was configured with CPU adaptation weight α=0.4, entropy weight β=0.35, and partitioning weight γ=0.25. Next, based on the standardized feature input parameters and the benefit sub-parameters of the two candidate paths, the three core sub-benefits—data adaptation benefit, computing power load benefit, and transmission cost benefit—were calculated for each path. Finally, a core decision function F was constructed, using file entropy H, partitioning K, GPU load G, and PCIe bandwidth B as core inputs. The three sub-benefits of the two paths were weighted and summed according to their corresponding benefit weights to complete the comprehensive benefit calculation for each single path. The final calculated benefit score was 6.114 for the GPU path and 2.7395 for the CPU path. Meanwhile, based on the characteristic that the file entropy value H=0.42<0.5, it is determined that this is a compression ratio priority scenario. The two paths are matched with the corresponding compression algorithms, and the file to be compressed is confirmed to be the uncompressed original file. Therefore, the compression-free rule is not triggered.

[0121] By employing a closed-loop accounting logic that combines feature standardization and normalization, path-specific weight configuration, multi-dimensional itemized benefit quantification, and weighted summation of decision functions, the problem of mismatch between algorithm and data features is solved, thereby improving the accuracy of compressed path selection, compression execution efficiency, and compression ratio.

[0122] Fifth embodiment

[0123] This embodiment provides an exemplary scheme for selecting a target execution path. In this example, a candidate revenue list is first generated by binding candidate paths with revenue points. Then, transmission cost correction is performed on the marked paths. Subsequently, the revenue of the candidate paths is sorted, thereby selecting the optimal target execution path suitable for this compression task. Step S30 includes steps D11 to D14:

[0124] Step D11: Bind and pair each candidate compression path with its corresponding revenue score to obtain a candidate revenue list.

[0125] Step D12: According to the refined decision markers in the candidate revenue list, adjust the transmission cost of the corresponding candidate path to obtain the revised candidate revenue list.

[0126] Step D13: Sort the revised candidate payout list from high to low according to the payout score to obtain an ordered candidate sequence.

[0127] Step D14: Select the highest profit score of the ordered candidate sequence and determine the candidate path corresponding to the highest profit score as the target execution path of this compression task.

[0128] The candidate revenue list is a structured dataset formed by integrating all candidate compression paths and their corresponding revenue scores, which have been bound one-to-one according to preset specifications. Refined decision markers are feature tags pre-embedded in the candidate revenue list and bound one-to-one with specific candidate compression paths, used to identify the path that requires specific transmission cost calibration correction. For example, specific correction markers are bound to GPU compression paths transmitting across PCIe links and backup compression paths transmitting across nodes remotely. The corrected candidate revenue list is a standardized structured dataset formed by accurately correcting the transmission costs of candidate paths with refined decision markers, synchronously updating their corresponding revenue scores, and then reorganizing and verifying them. The ordered candidate sequence is a structured dataset formed after sorting, clearly arranged according to the merits of revenue scores, with a clear correspondence between paths and revenue scores.

[0129] In this example, when binding each candidate compression path to its corresponding revenue point to obtain a candidate revenue list, the pre-binding of paths can be completed using the pre-scanning full-scale synchronous processing method provided in the first embodiment, followed by list integration. Alternatively, a parallel processing method can be used to perform incremental binding of paths and revenue points while calculating revenue points, accumulating the candidate revenue list to complete its generation.

[0130] After generating the candidate benefit list, the transmission cost correction process is initiated. For candidate paths in the list that carry detailed decision markers, a refined recalculation of the end-to-end transmission cost and benefit score calibration are performed. After correction, the sorting process is initiated. First, the corrected candidate benefit list is fully read. By iteratively traversing and comparing, the candidate path with the highest benefit score among the currently unselected paths is selected and moved into an ordered sequence container, generating an ordered candidate sequence arranged from highest to lowest benefit score. Finally, the first path of the ordered candidate sequence is extracted, and a full-process validity closed-loop verification is performed. Once confirmed to be correct, it is determined as the target execution path for this compression task. In this way, through layered pairing, correction, sorting, and filtering, the selection accuracy of the target execution path is improved, avoiding path selection errors caused by transmission cost calculation deviations.

[0131] For example, there are two methods for correcting transmission costs. The first is a line-by-line scanning single-path serial correction. Following a preset order in the candidate revenue list, the attribute fields of all candidate compression paths in the list are scanned line by line to identify target candidate paths carrying refined decision markers. For each identified target candidate path, the full transmission link feature data and initial transmission cost calculation parameters corresponding to that path are retrieved, and a refined recalculation of the full-link transmission cost is performed. Based on the recalculation results, the transmission cost is calibrated and corrected, and the corresponding sub-revenue and comprehensive revenue scores are updated simultaneously according to the corrected transmission cost. After completing the correction operation for a single path, the validity of the correction results is verified. Once it is confirmed that the corrected revenue score meets the preset specifications and has no calculation deviation, the corrected path data is updated to the candidate revenue list. Then, following the same process, the correction, verification, and list update of all remaining candidate paths with refined decision markers are completed sequentially. After all marked paths have been processed, a regularization verification is performed on the full list data to generate the final corrected candidate revenue list. This method employs a serial processing logic of line-by-line scanning and single-path independent closed loop, which can accurately adapt to the differentiated transmission characteristics of different paths. It avoids cross-interference and accounting deviation caused by batch correction from the source, thereby improving the accuracy of transmission cost correction and the reliability of correction results.

[0132] The second method is a clustering-group parallel correction. First, the candidate benefit list is read and parsed across all groups to identify all target candidate paths carrying refined decision markers. Then, based on the computing power type, transmission link attributes, and data processing characteristics of the target candidate paths, clustering is performed, grouping target candidate paths with similar transmission characteristics into the same processing group, generating multiple independent correction groups with no processing dependencies. Parallel correction processing is then initiated simultaneously for all independent correction groups. Within each correction group, the full set of transmission link features for the corresponding group's paths is retrieved, and a unified, refined recalculation of the transmission cost of all paths within the group is performed. Based on the recalculation results, the transmission cost calibration and correction of all paths within the group is completed, and the individual benefit and comprehensive benefit scores of the corresponding paths are updated synchronously. After the parallel processing of all correction groups is completed, a global benchmark alignment check is performed on the fully corrected path data to eliminate correction benchmark deviations between different groups. After confirming that all correction results meet the preset specifications, the corrected path data is synchronously updated to the candidate benefit list, completing the full normalization check of the list and generating the final corrected candidate benefit list. This method employs a batch correction logic that uses global clustering and grouping with parallel processing of all groups. By clustering and grouping based on transmission characteristics, it breaks the timing limitations of single-path serial correction, achieves multi-path dependency-free parallel correction processing, makes full use of the system's idle processing resources, reduces the overall processing time in scenarios with a large number of marked paths, and improves the overall processing efficiency of transmission cost correction.

[0133] For example, in the scenario of data fusion compression, a core decision function F is first constructed. The input feature parameters are file entropy H=0.43, divisibility K=0.95, GPU load G=0.24, PCIe bandwidth B=36GB / s, file size S=12GB, and CPU load C=0.31. Using experimentally calibrated weight parameters, the revenue share of the GPU heterogeneous computing power candidate path is calculated as 0.45×12×0.95×(1-0.24)+0.35×(1-0.43)-0.2×(12 / 36)=8.72, and the revenue share of the CPU general computing power candidate path is 0.4×12×(1-0.31)+0.35×(1-0.43)+0.25×0.95=3.86. Then, the two candidate compression paths are bound and paired with their corresponding revenue shares one by one to generate candidate compression paths. The candidate benefit list is selected, where the GPU path carries a refined decision marker for the dynamic loss of PCIe transmission. Then, according to the refined decision marker in the candidate benefit list, the transmission cost of the GPU path is corrected, and its benefit score is updated to 8.59. The CPU path has no corresponding marker and does not need to be corrected. A corrected candidate benefit list is generated. Then, the corrected candidate benefit list is sorted from high to low according to the benefit score, resulting in an ordered candidate sequence with the GPU path first and the CPU path second. Finally, the highest benefit score of 8.59 in the ordered candidate sequence is selected, and its corresponding GPU heterogeneous computing power candidate path is determined as the target execution path for this compression task. Simultaneously, it is verified that the file to be compressed is the uncompressed original file and does not trigger the compression-free rule. At the same time, based on the file entropy value H=0.43<0.5, it is determined to be a compression ratio priority scenario, and a compression algorithm is matched for the target GPU path.

[0134] By binding and pairing paths with revenue points, refining markers to correct transmission costs, sorting revenue points in descending order, and locking the optimal path with the highest revenue point, the decision logic, combined with multi-dimensional feature decision functions and scenario-based algorithm matching rules, solves the problem of mismatch between algorithms and data features in data fusion and compression scenarios, thereby improving the accuracy of compression path decision-making, compression execution efficiency, and computing resource utilization efficiency.

[0135] Sixth Embodiment

[0136] This embodiment provides an exemplary scheme for heterogeneous compression execution and resource scheduling. In this example, the resource scheduling rules corresponding to the target path are first matched with the appropriate compression algorithm. Then, the preprocessing of the data to be compressed and the heterogeneous data transmission are completed. Subsequently, the corresponding computing unit is called to perform compression processing, thereby finally generating a standardized compressed file. (Refer to...) Figure 5 , Figure 5 This is a flowchart illustrating the sixth embodiment of the data fusion and compression control method of this application. Step S30 includes steps E11 to E14:

[0137] Step E11: In the compression process, match the resource scheduling rules corresponding to the target execution path, and match the appropriate compression algorithm based on the redundancy of the data to be compressed.

[0138] Step E12: Preprocess the data to be compressed according to the resource scheduling rules. If it is a heterogeneous path, asynchronously transmit the data to be compressed to the heterogeneous video memory.

[0139] Step E13: Call the computing unit corresponding to the target execution path, load the compression algorithm matching the data to be compressed, and compress the preprocessed data to obtain segmented compressed data.

[0140] Step E14: Standardize the segmented compressed data and add a file checksum header to obtain the compressed file.

[0141] Resource scheduling rules are bound one-to-one with the target execution path, and are standardized control rules used to regulate the allocation of computing power, data transmission, and task scheduling for this compression task. Examples include the core allocation rules for CPU multi-threaded paths and the PCIe asynchronous transmission rules for GPU parallel paths. Compression algorithms are instances of algorithms matched to the redundancy of the data to be compressed, used to perform data compression. Examples include the LZ4 algorithm for highly redundant text data and the ZSTD algorithm for low-redundancy image data. Heterogeneous video memory is a high-speed storage medium dedicated to heterogeneous computing units (such as graphics processing units), used to store the original data to be compressed and intermediate compression results. Segmented compressed data are segmented compressed result units with independent boundaries generated after compression processing. File checksum headers are standardized header data used to identify the compressed file format and record checksum information, ensuring the integrity and resolvability of the compressed file.

[0142] In this example, when matching the resource scheduling rules and adapting the compression algorithm corresponding to the target execution path, the matching of rules and algorithms can be completed by directly calling the preset fixed rules provided in the previous embodiments. Alternatively, the matching of rules and algorithms can be completed by dynamically customizing rule generation and generating adapted scheduling rules and compression algorithms based on the current task and data status.

[0143] After matching the rules and algorithms, the data preprocessing process is initiated. According to the resource scheduling rules, preprocessing operations such as fragmentation and alignment are performed on the data to be compressed. If the current compression path is heterogeneous, the preprocessed data is asynchronously transferred to heterogeneous video memory. After data preparation, the computing unit corresponding to the target path is called to load the matching compression algorithm and perform compression processing on the preprocessed data, resulting in segmented compressed data. Finally, the segmented compressed data is standardized and normalized, and a file checksum header is added to generate the final compressed file. This layered scheduling, preprocessing, compression, and standardization process improves the efficiency of compression execution and the reliability of the compressed file.

[0144] For example, there are two implementation methods for compression execution and resource scheduling. The first is serial progressive scheduling compression. After the scheduling and compression process officially starts, the allocation of computing resources, storage resources, and initialization of compression algorithm instances are completed sequentially according to the resource scheduling rules of the target execution path. After all resource scheduling actions are completed, the serial execution of data preprocessing, fragmented transmission, and compression processing is started. After all actions are completed, the final compressed file is generated. This method has a simple and stable execution logic. Each action proceeds sequentially, making it less prone to resource conflicts or execution deviations. It can provide a stable operating environment for compression tasks and is suitable for common compression scenarios.

[0145] The second method is pipelined parallel scheduling compression. First, based on the resource scheduling rules of the target execution path, parallel scheduling is initiated simultaneously for computing resource allocation, storage resource allocation, and compression algorithm instance initialization. While scheduling actions are being executed, data preprocessing and fragmented transmission preparations are also initiated concurrently. After scheduling is complete, pipelined compression processing begins directly, with compression of preceding fragments, transmission of intermediate fragments, and preprocessing of subsequent fragments executed synchronously and in parallel. Once all actions are completed, the final compressed file is generated. This method executes synchronously with the pipelined parallel processing flow, eliminating the need for additional serial scheduling time. It fully utilizes idle system computing power to complete parallel processing, significantly improving the overall execution efficiency of the compression task.

[0146] For example, in a scenario of vehicle-road cooperative roadside perception data fusion and compression, for a 10GB uncompressed roadside perception raw data packet (redundancy 58%, entropy H=0.42), the resource scheduling rules corresponding to the heterogeneous target execution path of the determined graphics processing unit (GPU) are first matched, and the nvcomp GDeflate compression algorithm is matched and adapted based on the data redundancy. Then, the data is preprocessed into 64KB segments according to the rules. After being identified as a heterogeneous path, it is asynchronously transmitted to the GPU heterogeneous memory through a high-speed serial computer expansion bus (PCIe, Peripheral Component Interconnect Express) link. Subsequently, the GPU computing unit corresponding to the target path is called to load the adaptation algorithm to complete the compression, resulting in 1920 segmented compressed data. Finally, the segmented data is standardized and spliced, and a file header containing full file verification information is added to generate a 3.8GB final compressed file.

[0147] By employing a closed-loop execution logic involving path matching, preprocessing transmission, parallel compression, and standardized encapsulation, the core problems of mismatch between computing power and algorithms, heterogeneous transmission blockage, and low compression efficiency in data fusion compression scenarios are solved, thereby improving compression execution efficiency and computing resource utilization.

[0148] Seventh Embodiment

[0149] This embodiment provides an exemplary scheme for adaptive optimization of decision parameters. In this example, the compression speed, compression ratio, and file integrity verification results of this compression are first correlated to obtain process performance data. Then, the full data of this task is stored in the historical task database, and the most recent valid historical task data is filtered out. Finally, the adaptive optimization of decision parameters is completed based on the valid historical data, and the decision rule base is updated synchronously. After step S40, steps F11 to F13 are also included:

[0150] Step F11: Associate the compression speed and compression ratio of this compression process with the integrity verification result of the compressed file to obtain the performance data of this compression process.

[0151] Compression speed is a core metric for quantifying the efficiency of a compression task, reflecting the size of the original data processed per unit time. Compression ratio is a core metric for quantifying the data compression effect of a compression task, reflecting the degree of reduction in file size relative to the original file. Performance data is a standardized dataset that comprehensively reflects the performance and efficiency of a compression task by integrating the file integrity verification results, compression speed, and compression ratio. For example, if the compression speed is 100MB / s, the compression ratio is 0.3, and the integrity verification result is passed, the performance data would be a standardized dataset containing all three metrics.

[0152] In this embodiment, the performance data generation process can be triggered in three ways. First, it can be triggered automatically after the compressed file is generated. Once the compressed file is generated, the integrity verification and performance statistics process is automatically initiated to achieve real-time performance data collection for a single task. Second, it can be triggered before task archiving. Before archiving the data of this task to the historical task library, the performance data generation process is triggered to ensure the integrity of the archived data. Third, it can be triggered before parameter optimization. Before initiating adaptive optimization of decision parameters, the performance data generation process for this task is triggered to provide accurate input data for optimization.

[0153] Once the compression task is completed, the execution sequence, file size, and verification baseline information of this compression will be collected.

[0154] For example, the compression performance data is generated in two ways. The first is a serial closed-loop verification and statistics method. This method extracts the verification benchmark information, original file attribute information, and corresponding compression algorithm identifier from the file verification header. Based on the matching decompression and restoration rules, it performs integrity verification of the compressed file, simultaneously verifying the normal decompression capability of the compressed file and the consistency between the decompressed and restored data and the original data to be compressed. After confirming that the compressed file is complete, lossless, and without data deviation, it retrieves the execution time sequence record of this compression task, calculates the total execution time of the compression task and the total size of the original processed data, and obtains the compression speed of this compression process. Then, it calculates the compression ratio of this compression process by calculating the total size of the original files to be compressed and the total size of the final compressed file. Finally, it integrates the integrity verification results, compression speed, and compression ratio to generate standardized performance data for this compression process. This method employs a serial closed-loop execution logic with integrity verification as the pre-processing step and performance statistics as the post-processing step. It uses the availability verification of compressed files as a rigid prerequisite for performance statistics, and only performs the calculation and correlation of performance indicators after confirming that the compressed file is complete and usable. This avoids the problem of performance data from invalid or damaged compressed files entering the subsequent optimization process, and ensures the authenticity, validity and reference value of the performance data.

[0155] The second method is parallel synchronous statistical verification. During the compression task execution, the execution sequence and data scale of the task are statistically analyzed simultaneously, and the compression speed and compression ratio are calculated in real time. Simultaneously, after the compressed file is generated, integrity verification is initiated synchronously. After verification is completed, the three metrics are correlated and integrated to obtain performance data. This method is performed synchronously with the compression execution process, eliminating the need for additional time spent on serial statistical analysis and improving the efficiency of performance data generation.

[0156] Step F12: Store the performance data, feature parameters, and path selection structure of the current compression process into the historical task library, and filter out the most recent valid historical task data.

[0157] The feature parameters are collected throughout the entire compression task, covering input and processing parameters of core dimensions such as the inherent attributes of the data to be compressed, the system's computing power status, and the performance of the transmission link. The path selection structure is formed after multi-dimensional benefit calculation, sorting, and filtering, and includes a complete decision-making system architecture encompassing target execution path attributes, benefit weight configuration, and decision logic. The historical task library is a pre-built structured data storage set used to centrally store all the associated data of completed compression tasks. Valid historical task data is archived data of historical compression tasks that has undergone dual verification of compliance and integrity, confirming data integrity and normal task execution, and can be used for subsequent decision optimization.

[0158] In this embodiment, the aforementioned task data archiving and historical data filtering process can be triggered in three ways. First, it is automatically triggered after performance data generation. Once the performance data for the current task is generated, the data archiving and filtering process is automatically initiated to achieve real-time data accumulation for a single task. Second, it is triggered periodically. The system periodically initiates the organization of the historical task library and the filtering of valid data according to a preset cycle to ensure the validity of historical data. Third, it is triggered before parameter optimization. Before initiating decision parameter optimization, the archiving of the current task and the filtering of valid historical data are triggered to provide input data for optimization.

[0159] Once the performance data for this compression is obtained, the task data association, binding, and archiving storage will begin.

[0160] For example, there are two methods for archiving task data and filtering valid historical data. The first method is sequential closed-loop archiving followed by filtering. First, the performance data, feature parameters, and path selection structure of the current compression process are linked and bound one-to-one across all dimensions, establishing an inseparable, exclusive mapping relationship between the three. Simultaneously, the integrity and compliance checks of the linked data are performed. After confirming that the data is complete, without any missing or abnormal data, and that the mapping relationship is accurate, the complete task data after being linked and bound is written into the historical task library according to the pre-defined standardized storage specifications, completing the archiving and storage of the current task data. Then, based on the standardized time-series index of the historical task library, all archived task data in the library is traversed, and a full data validity check is performed, filtering out invalid historical data with missing data or abnormal task execution, resulting in a complete and valid historical task dataset. Finally, the valid historical task dataset is sorted in descending order of task completion time from most recent to oldest, and the latest data at the top of the sorted list is extracted according to the pre-defined filtering rules, resulting in the final, most recent valid historical task data. This method employs a fully sequential closed-loop execution logic that initiates filtering only after archiving is completed. It constructs an independent closed loop of association, verification, and archiving for data in a single task, ensuring the accuracy and integrity of the mapping of archived data and avoiding the problems of archived data disorder and loss of association relationships from the source.

[0161] The second method is incremental synchronous archiving and batch filtering. First, incremental validity checks are performed on the current task data. Once the data is confirmed to be valid, it is directly written to the historical task database. Then, based on the time-series index within the database, the most recent preset number of task data is extracted, and batch validity checks are performed synchronously to obtain valid historical task data. This method uses incremental validation logic, eliminating the need to traverse the entire database, thus improving filtering efficiency and making it suitable for fast filtering scenarios with large databases.

[0162] Step F13: Based on the effective historical task data, adaptive optimization of decision parameters is performed, and the updated decision parameters are synchronously updated to the decision rule base.

[0163] In this embodiment, the above-mentioned adaptive optimization process for decision parameters can be triggered in four ways. First, it is automatically triggered after the effective historical data filtering is completed. Once the most recent effective historical task data has been filtered, the parameter optimization process is automatically initiated to achieve real-time parameter iteration in small steps. Second, it is triggered by the cumulative number of tasks. When the cumulative number of newly added effective task data in the historical task library reaches a preset threshold (e.g., 100 groups), the batch parameter optimization process is automatically triggered to achieve precise parameter iteration supported by batch data. Third, it is triggered by a timed period. The system automatically triggers the parameter optimization process at specified time points according to preset periodic scheduling rules to achieve periodic rule base updates. Fourth, it is triggered by performance degradation. When the system detects that the average execution performance of recent compression tasks is lower than a preset performance threshold, the parameter optimization process is automatically triggered to correct performance degradation issues.

[0164] Once valid historical task data is obtained, adaptive optimization of decision parameters and rule base updates are initiated.

[0165] For example, there are two ways to implement adaptive optimization of decision parameters. The first is a step-by-step sequential optimization. This involves reading the filtered valid historical task data, breaking down the input feature parameters, decision parameter configurations, path selection results, and performance data corresponding to each historical task, and classifying and aggregating them according to the functional dimensions of the decision parameters. Then, adaptive optimization is initiated dimension by dimension in a fixed order: feature normalization rule parameters, benefit ratio weight parameters, path selection threshold parameters, and algorithm adaptation judgment parameters. For a single-dimensional decision parameter, the correlation between different configurations of that parameter in historical tasks and the final execution performance is compared to identify the deviation range between the current parameter configuration and the historical best execution effect. Based on the optimal value range fed back from valid historical data, adaptive adjustment and optimization of the parameter in that dimension are completed. After the single-dimensional parameter optimization is completed, backtracking verification of the optimization effect is performed simultaneously based on historical task data. After confirming that the optimized parameters can effectively improve the decision accuracy of the corresponding dimension, the parameter optimization of the next dimension is initiated. After all decision parameters in all dimensions have been optimized and verified, the updated decision parameters are integrated and generated. The updated decision parameters are then compared and verified against the existing rule system in the decision rule base. Once it is confirmed that there are no rule conflicts and the adaptability meets the requirements, the updated parameters are synchronously updated in the decision rule base. This method employs a progressive, sequential optimization and single-dimensional closed-loop verification execution logic. Through independent optimization and backtracking verification in a single dimension, the optimization direction of each decision parameter is located, avoiding coupling interference between multi-dimensional parameters. Each optimized parameter adjustment is supported by clear historical data, ensuring strong traceability and interpretability, and resulting in extremely high accuracy and stability in parameter optimization.

[0166] The second approach is full-dimensional parallel batch optimization. It reads the filtered, valid historical task data, breaks down the decision parameters by functional dimension, and simultaneously initiates parallel optimization across all dimensions. Each dimension independently adjusts and validates its parameters. Once all dimensions are optimized, the updated decision parameters are integrated and synchronously updated to the decision rule base. This method employs full-dimensional parallel optimization logic, fully utilizing the system's idle computing power, significantly improving parameter optimization efficiency, and is suitable for batch optimization scenarios with large amounts of historical data.

[0167] In an exemplary scheme for adaptive optimization of decision parameters, the final generated compressed file is first subjected to a full integrity verification. Simultaneously, the actual compression speed and compression ratio throughout the entire task are statistically analyzed, and the three core indicators are correlated and integrated to generate complete performance data for the compression process. This performance data, along with the task's corresponding file entropy value, file size, and other characteristic parameters, as well as the entire path selection decision structure, are then uniformly bound, archived, and stored in a historical task database. Next, 100 sets of valid historical task data that are the most recent in time and executed normally are selected from the database. Following this, data review is performed based on the operating logic of two main compression paths. For the graphics processor path, asynchronous data transfer from CPU memory to video memory is achieved through a unified computing device architecture interface, coupled with multi-threaded block-based parallel compression, and the compression results are sent back. For the CPU path, files are split according to the number of cores, subjected to multi-threaded parallel compression, and the results are then aggregated and merged. Next, relying on the feedback optimization layer mechanism, the feature parameters, decision results, and total time of multiple rounds of tasks were accumulated and nearly one hundred sets of historical data were collected to carry out parameter adaptive optimization. On the one hand, the average compression speed ratio of the graphics processor path and the central processing unit path was compared, and the file size threshold was adjusted according to the ratio range and strictly kept within a fixed range. At the same time, the failure rate of the graphics processor path was counted, and the load threshold was fine-tuned according to the failure ratio while following the upper and lower boundary constraints. On the other hand, a loss objective function with the reciprocal of compression speed and the reciprocal of compression ratio as the core was built, and the weights of the two indicators were balanced with fixed performance hyperparameters. After collecting the average loss of one hundred tasks, the three core weight parameters in the decision function were iteratively updated through gradient descent algorithm with a set learning rate. Finally, all the optimized and updated thresholds and weight parameters were synchronously written into the decision rule base to achieve global effect.

[0168] By verifying and archiving compression results, accumulating and reviewing historical data, adapting to path differences, and using a two-layer adaptive iterative optimization of threshold weights, the problems of poor adaptation of static decision parameters, inaccurate prediction of heterogeneous path performance, and decay of long-term execution effect in data fusion compression scenarios have been solved.

[0169] Step F13 includes steps G11 to G13:

[0170] Step G11: Compare the performance of the two types of paths and the failure rate of heterogeneous paths from the effective historical task data, and adjust the boundary constraints according to the threshold optimization rules to obtain the target decision threshold.

[0171] In this embodiment, the threshold optimization process can be triggered in four ways. First, it is automatically triggered after the effective historical data filtering is completed. Once the most recent 100 sets of effective historical task data have been filtered, the threshold optimization process is automatically initiated to achieve real-time threshold iteration in small steps. Second, it is triggered by the cumulative number of tasks. When the cumulative number of newly added effective task data in the historical task library reaches a preset threshold, the batch threshold optimization process is automatically triggered to achieve precise threshold iteration supported by batch data. Third, it is triggered by a timed period. The system automatically triggers the threshold optimization process at specified time points according to preset periodic scheduling rules to achieve periodic decision threshold updates. Fourth, it is triggered by performance degradation. When the system detects that the average execution performance of recent compression tasks is lower than a preset performance threshold, the threshold optimization process is automatically triggered to correct performance degradation issues.

[0172] Once the decision optimization module receives valid historical task data, it initiates the pre-statistical process for threshold optimization, extracting core information such as path type, execution time, and execution results from the valid historical task data.

[0173] For example, threshold optimization can be implemented in two ways. The first is a sequential, progressive threshold adjustment. First, all valid historical task data is traversed sequentially, and the compression execution efficiency of the two types of paths is statistically analyzed to calculate performance comparisons. Based on the calculation results, the decision threshold for file-related tasks is adjusted. Then, the task execution results of heterogeneous paths are traversed and statistically analyzed to calculate the failure rate. Based on the calculation results, the decision threshold for load-related tasks is adjusted. Finally, boundary constraint checks are performed on the two adjusted thresholds sequentially. If the threshold exceeds the boundary, it is corrected to within the boundary range to obtain the target decision threshold. This method has a simple and clear execution logic, and the sequential advancement of each adjustment step avoids parameter adjustment conflicts. It consumes relatively few processing resources during operation and is suitable for typical small-batch historical data optimization scenarios.

[0174] The second method is parallel synchronous threshold adjustment. It simultaneously extracts performance and execution result information from all valid historical task data, concurrently performs performance comparison calculations for the two types of paths and failure rate calculations for heterogeneous paths, synchronously obtains the adjustment amounts for the two decision thresholds, and adjusts both thresholds simultaneously. Subsequently, it performs boundary constraint verification on the two adjusted thresholds in batches, synchronously correcting any deviations from the boundary thresholds, and finally obtains the target decision threshold. This method has a highly efficient adjustment process, can simultaneously handle the synergistic effects of the two types of adjustments, quickly completes the overall threshold optimization, and is suitable for rapid optimization scenarios with large amounts of historical data.

[0175] In an exemplary threshold optimization scheme, 100 sets of filtered valid historical task data are first loaded, and the path type, execution time, and execution result fields are extracted. First, the average compression speed of all CPU paths and GPU paths is calculated, yielding a performance ratio of 2.3. Based on this ratio, the original file size threshold is adjusted from 1GB to 512MB. Next, the execution results of all GPU paths are analyzed, and the failure rate of heterogeneous paths is calculated to be 2.1%. Based on this failure rate, the original GPU load threshold is adjusted from 70% to 65%. Then, boundary constraint checks are performed to verify whether the adjusted file size threshold of 512MB is within the 100MB~10GB boundary range, and whether the adjusted GPU load threshold of 65% is within the 50%~80% boundary range. Both thresholds are confirmed to meet the boundary constraint requirements. Finally, the two adjusted thresholds are integrated to obtain the target decision threshold for this optimization, which is used for subsequent path selection and determination.

[0176] Step G12: Based on the effective historical task data and weight optimization rules, iteratively calculate the target weight parameters and update the triggering rules for the refined decision markers.

[0177] Once the threshold optimization process is complete, the decision optimization module initiates the preparatory process for weight optimization, extracting core information such as feature parameters, decision results, and performance data from valid historical task data.

[0178] For example, there are two implementation methods for weight optimization and labeling rule updates. The first is a serial progressive weight adjustment, which first traverses all weight parameters to be optimized in dimensional order, extracts historical performance data for each corresponding dimension, and iteratively calculates the optimization direction of each parameter in round-by-round iterations. After the adjustment of a single weight parameter is completed, the iterative calculation of the next weight parameter begins. After all weight parameters have been adjusted, the triggering rules for decision labels are refined one by one based on the updated weight parameters, and the triggering conditions for each decision label are adjusted sequentially. This method has a simple execution logic, the optimization process of each parameter is independent and controllable, there is no optimization interference between parameters, and it consumes less processing resources during operation, making it suitable for common small-batch historical data optimization scenarios.

[0179] The second method is parallel batch weight adjustment. It simultaneously extracts historical performance data corresponding to all weight parameters to be optimized, initiates iterative calculations for all weight parameters simultaneously, and completes the adjustment of all weight parameters in batches. Subsequently, it batch extracts historical trigger data for all refined decision markers and simultaneously updates the trigger rules for all decision markers. This method has a highly efficient optimization process, fully utilizes parallel computing power to complete batch optimization, significantly reduces the overall time consumption of weight optimization, and is suitable for rapid optimization scenarios with large amounts of historical data.

[0180] In an exemplary weight optimization scheme, 100 sets of selected valid historical task data are first loaded. A loss objective function is constructed with the reciprocal of compression speed and the reciprocal of compression ratio as its core, and the weights of two indicators are balanced by a fixed performance hyperparameter. Iterative calculation is then initiated, using a gradient descent algorithm with a set learning rate to iteratively update the three core weight parameters within the decision function: data adaptation benefit, computational load benefit, and transmission cost benefit. After 15 iterations, the loss function converges, yielding optimized target weight parameters of 0.5, 0.3, and 0.2. Next, for refined decision marking, performance data of cross-PCIe link transmission paths in historical tasks is statistically analyzed, and the original PCIe link bandwidth utilization trigger threshold is adjusted from 50% to 60%, completing the update of the refined decision marking trigger rules.

[0181] Step G13: Update the target decision threshold, target weight parameters, and updated refined decision marker trigger rules to the rule base to optimize the next compression task.

[0182] As an optional implementation, the target decision threshold, target weight parameters, and refined decision marker trigger rules are extracted sequentially according to parameter type. Each rule is then matched against the corresponding stored item in the rule base, and incremental updates are performed on each stored item. Once an update of a stored item is completed, the rule item is marked as active. After all updates are completed, the optimization is marked as complete, providing an optimized decision basis for the next compression task. This method offers traceability during the update process, independent and controllable update status for each rule item, rapid problem localization in case of update anomalies, and minimal impact on other completed updates. Furthermore, it consumes relatively few processing resources during operation.

[0183] Exemplarily, the threshold adjustment and weight adaptive update process based on historical performance is as follows. First, the performance data of the last N compression tasks is statistically analyzed, where N represents the number of tasks in a single optimization batch. In this case, N = 100. For the graphics processing unit (GPU) path and the central processing unit (CPU) path, first perform threshold adjustment based on performance comparison. Calculate the performance ratio of the average compression speeds of the two types of paths. If the average compression speed of the GPU path > 1.5 × the CPU path, then reduce the file size threshold T, where T represents the file size determination threshold in the compression path decision-making link. The adjusted new threshold is T_new = T × 0.8. If the average compression speed of the GPU path < 0.8 × the CPU path, then increase the file size threshold T, and the adjusted new threshold is T_new = T × 1.2. Subsequently, perform threshold adjustment based on the failure rate. Statistically analyze the failure rate of the GPU path, which is the ratio of the number of failures to the total number of executions. If the failure rate of the GPU path > 10%, then increase the GPU load threshold G_threshold, where G_threshold represents the graphics processing unit load determination threshold in the compression path decision-making link. The adjusted new threshold is G_threshold_new = G_threshold - 5%. If the failure rate of the GPU path < 2% and the current G_threshold > 50%, then moderately reduce the load threshold, and the adjusted new threshold is G_threshold_new = G_threshold + 3%. All threshold adjustments need to satisfy boundary constraints. The file size threshold needs to satisfy 200MB ≤ T ≤ 10GB, and the GPU load threshold needs to satisfy 30% ≤ G_threshold ≤ 90%. After completing the threshold adjustment, start the adaptive update of the weights of the decision function. The weight parameters α, β, γ in the decision function, where α represents the graphics processing unit adaptation weight, β represents the entropy weight, and γ represents the bandwidth weight, are adaptively updated through the gradient descent method. First, define the optimization objective loss function J, where J represents the loss function used to measure the optimization effect of the decision parameters, and its expression is J = α_perf × (1 / speed) + β_perf × (1 / compression_ratio), where α_perf represents the performance hyperparameter that controls the importance of the compression speed, β_perf represents the performance hyperparameter that controls the importance of the compression ratio, speed represents the compression speed in MB / s, and the larger the value, the higher the compression efficiency, and compression_ratio represents the compression rate, which is the ratio of the original size to the compressed size, and the larger the value, the better the compression reduction effect. After each batch of tasks (N = 100) is completed, perform weight update. First, calculate the average loss J_avg under the current parameters, where J_avg represents the average loss value of the current batch of tasks, that is, J_avg = mean(J_1, J_2,..., J_N). Subsequently, apply the gradient descent update formula, where η is the learning rate of the gradient descent. In this case, η = 0.01, J / α、 J / β、 J / γ represents the partial derivative of the loss function with respect to each weight parameter, and the updated weight parameters are α_new = α - η × J / α、β_new=β-η× J / β、γ_new=γ-η× J / γ is used to achieve adaptive optimization of the weights.

[0184] By employing a two-layer adaptive iterative optimization based on threshold weights, the problems of poor adaptation of static decision parameters and inaccurate prediction of heterogeneous path performance in data fusion and compression scenarios are resolved, thereby improving the accuracy of path decision-making and the efficiency of compression execution.

[0185] This application provides a data fusion compression device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data fusion compression control method in the first embodiment described above.

[0186] The following is for reference. Figure 6 The diagram illustrates a structural schematic of a data fusion compression device suitable for implementing embodiments of this application. The data fusion compression device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, data fusion compression storage all-in-one machines, personal digital assistants (PDAs), heterogeneous computing power data compression acceleration terminals, etc., as well as fixed terminals such as data fusion compression servers, desktop computers, etc. Figure 6 The data fusion and compression device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0187] like Figure 6As shown, the data fusion compression device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the data fusion compression device. The processing unit 1001, the ROM 1002, and the RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the data fusion compression device to communicate wirelessly or wiredly with other devices to exchange data. Although a data fusion compression device with various systems is shown in the figure, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0188] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0189] The data fusion compression device provided in this application, employing the data fusion compression control method described in the above embodiments, can solve the technical problem of low data compression efficiency. Compared with the prior art, the beneficial effects of the data fusion compression device provided in this application are the same as those of the data fusion compression control method described in the above embodiments, and other technical features of this data fusion compression device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0190] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0191] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0192] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the data fusion compression control method described in the above embodiments.

[0193] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0194] The aforementioned computer-readable storage medium may be included in the data fusion compression device; or it may exist independently and not be assembled into the data fusion compression device.

[0195] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the data fusion compression device, the data fusion compression device: responds to a data compression command by determining candidate compression paths that match the inherent attribute characteristics of the data to be compressed and the computing power status parameters of the current system; uses the inherent attribute characteristics and the computing power status parameters as independent variables of a decision function to determine the benefit score corresponding to each candidate compression path; selects the candidate compression path with the highest benefit score as the target execution path; and, during the compression process, performs resource scheduling actions based on the resource scheduling rules corresponding to the target execution path and performs compression actions based on the target execution path to obtain a compressed file.

[0196] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0197] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation that may be implemented in systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0198] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0199] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described data fusion compression control method, thereby solving the technical problem of low data compression efficiency. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the data fusion compression control method provided in the above embodiments, and will not be repeated here.

[0200] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A control method for data fusion compression, characterized in that, The method includes: In response to a data compression command, a candidate compression path is determined that matches the inherent attribute characteristics of the data to be compressed and the computing power status parameters of the current system. The inherent attribute characteristics of the data to be compressed are proprietary attribute indicators that describe the size and redundancy of the data to be compressed. The inherent attribute features and the computing power state parameters are normalized according to the input feature rules to generate feature input parameters; For each of the candidate compression paths, configure the corresponding revenue weight parameters according to the weight library; The feature input parameters are input into the decision function, and the data adaptation benefit, computing load benefit, and transmission cost benefit of each candidate compression path are calculated by combining the benefit sub-item parameters of each candidate compression path. The decision function corresponds one-to-one with each candidate compression path. The revenue score corresponding to the candidate compression path is obtained by weighting and summing the data adaptation revenue, computing load revenue, and transmission cost revenue of the candidate compression path according to the revenue weight. The candidate compression path with the highest profit score is selected as the target execution path; During the compression process, resource scheduling actions are performed based on the resource scheduling rules corresponding to the target execution path, and compression actions are performed based on the target execution path to obtain a compressed file.

2. The data fusion compression control method as described in claim 1, characterized in that, The step of determining candidate compression paths that match the inherent attribute characteristics of the data to be compressed and the computing power status parameters of the current system in response to a data compression command includes: In response to the data compression command, an initial path that meets the attribute requirements and load requirements is determined based on the inherent attribute characteristics and the computing power status parameters; Analyze the block continuity characteristics of the data to be compressed to determine the block adaptation parameters of the data to be compressed; The initial path that matches the segmentation adaptation parameters is selected as the candidate compression path.

3. The data fusion compression control method as described in claim 2, characterized in that, The step of parsing the block continuity characteristics of the data to be compressed and determining the block-specific adaptation parameters of the data to be compressed includes: The block continuity features of each segment in the data to be compressed are analyzed to obtain a block-level feature sequence; Calculate the local information entropy of all block-level features in the block-level feature sequence, and arrange the local information entropy according to the order of the block-level features to obtain a local entropy value sequence; Based on the local entropy value sequence, the entropy value difference between adjacent blocks is compared, the total proportion of consecutive blocks that meet the requirements is counted, and the block adaptation parameters of the data to be compressed are calculated.

4. The data fusion compression control method as described in claim 1, characterized in that, The step of selecting the candidate compression path with the highest profit score as the target execution path includes: Each candidate compression path is paired with its corresponding revenue score to obtain a candidate revenue list; Based on the refined decision markers in the candidate revenue list, the transmission cost of the corresponding candidate path is adjusted to obtain the revised candidate revenue list. The revised candidate payout list is sorted from high to low according to the payout score to obtain an ordered candidate sequence; The highest-scoring candidate path in the ordered candidate sequence is selected, and the candidate path corresponding to the highest-scoring candidate path is determined as the target execution path for this compression task.

5. The data fusion compression control method as described in claim 1, characterized in that, The steps of performing resource scheduling actions based on the resource scheduling rules corresponding to the target execution path and performing compression actions based on the target execution path to obtain a compressed file during the compression process include: During the compression process, the resource scheduling rules corresponding to the target execution path are matched, and the appropriate compression algorithm is matched based on the redundancy of the data to be compressed. The data to be compressed is preprocessed according to the resource scheduling rules. If it is a heterogeneous path, the data to be compressed is asynchronously transmitted to the heterogeneous video memory. The computing unit corresponding to the target execution path is invoked to load the compression algorithm matching the data to be compressed, and the preprocessed data to be compressed is compressed to obtain segmented compressed data. The segmented compressed data is standardized and a file checksum header is added to obtain the compressed file.

6. The data fusion compression control method as described in claim 1, characterized in that, After the steps of performing resource scheduling actions based on the resource scheduling rules corresponding to the target execution path and performing compression actions based on the target execution path to obtain a compressed file during the compression process, the data fusion compression control method further includes: The compression speed and compression ratio of this compression process are correlated with the integrity verification result of the compressed file to obtain the performance data of this compression process; The performance data, feature parameters, and path selection structure of this compression process are stored in the historical task library, and the most recent valid historical task data are filtered out. Based on the effective historical task data, adaptive optimization of decision parameters is performed, and the updated decision parameters are synchronously updated to the decision rule base.

7. The data fusion compression control method as described in claim 6, characterized in that, The step of adaptively optimizing decision parameters based on the effective historical task data and synchronously updating the updated decision parameters to the decision rule base includes: The performance comparison of the two types of paths and the failure rate of heterogeneous paths are statistically analyzed from the effective historical task data, and the boundary constraints are adjusted according to the threshold optimization rules to obtain the target decision threshold. Based on the effective historical task data and weight optimization rules, the target weight parameters are iteratively calculated, and the triggering rules for refined decision markers are updated. The target decision threshold, target weight parameters, and updated refined decision marker trigger rules are updated in the rule base to optimize the next compression task.

8. A data fusion and compression device, characterized in that, The data fusion compression device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the control method for data fusion compression as described in any one of claims 1 to 7.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the data fusion compression control method as described in any one of claims 1 to 7.

Images