Data processing method and device, electronic equipment, storage medium and computer program product

Abstract

The application provides a data processing method and device, electronic equipment, storage medium and computer program product, relates to the technical field of data processing technology, and the method comprises the following steps: acquiring to-be-processed data; determining a target algorithm corresponding to the to-be-processed data based on multi-dimensional data characteristics of the to-be-processed data and real-time performance parameters; and processing the to-be-processed data based on the target algorithm to determine target data. In this way, the multi-dimensional data characteristics of the data and the real-time performance parameters are fully considered when the target algorithm is determined, so that the target algorithm that matches the data and the current performance can be determined when facing large data volume or abnormal performance. Moreover, the target algorithm is dynamically determined, so that if the performance changes during the processing of the to-be-processed data, the target algorithm that matches the changed performance can be dynamically determined according to the real-time performance parameters, so that the determined target algorithm can always adapt to the data and the performance, thereby improving the data read-write performance of the equipment.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus, electronic device, storage medium, and computer program product. Background Technology

[0002] Currently, the primary Online Analytical Processing (OLAP) engine for logistics data products is ClickHouse. The current data analysis method involves offline data warehouse pre-aggregation processing, which involves performing cube operations on various dimensions to generate application result tables, importing them into ClickHouse, and then performing single-point queries on the indicator results under different dimension combinations. In ClickHouse, the compression algorithm is statically specified by the codec (CODEC) during table creation and cannot be dynamically adjusted afterwards. However, a fixed compression algorithm cannot adapt to large data volumes or high resource pressure, resulting in poor read and write performance. Summary of the Invention

[0003] The present application provides a data processing method, apparatus, electronic device, storage medium, and computer program product that can improve the performance of reading and writing data.

[0004] The technical solution of this application is implemented as follows:

[0005] This application provides a data processing method, including:

[0006] Obtain the data to be processed;

[0007] The target algorithm corresponding to the data to be processed is dynamically determined based on the multidimensional data characteristics of the data to be processed and the real-time performance parameters.

[0008] The target data is determined by processing the data to be processed based on the target algorithm.

[0009] In the above scheme, the step of dynamically determining the target algorithm corresponding to the data to be processed based on the multidimensional data features and real-time performance parameters of the data to be processed includes:

[0010] Feature extraction is performed on the preprocessed data to be processed to determine the multidimensional data features; wherein, the preprocessing is used to characterize the data cleaning and formatting processing performed on the data to be processed;

[0011] Real-time monitoring of local performance metrics to determine the real-time performance parameters; wherein the real-time performance parameters include at least one of the following: system resource performance parameters, network condition performance parameters, and query load performance parameters;

[0012] The target algorithm for the data to be processed is dynamically determined based on the multidimensional data features, the real-time performance parameters, and the algorithm library; wherein, the algorithm library includes: multiple data compression algorithms, and data processing feature labels corresponding to each data compression algorithm.

[0013] In the above scheme, the target algorithm that dynamically determines the data to be processed based on the multidimensional data features, the real-time performance parameters, and the algorithm library includes:

[0014] Based on the multidimensional data features, the real-time performance parameters, and the algorithm library, the target algorithm is dynamically determined that matches the multidimensional data features and whose corresponding data processing feature label matches the real-time performance parameters.

[0015] In the above scheme, the step of dynamically determining the target algorithm that matches the multidimensional data features and whose corresponding data processing feature label matches the real-time performance parameters based on the multidimensional data features, the performance parameters, and the algorithm library includes:

[0016] Based on the multidimensional data features, a first feature label corresponding to the multidimensional data features is determined according to a preset data feature rule; wherein, the preset data feature rule is used to limit the correspondence between the data features and the corresponding requirement processing feature labels;

[0017] Based on the real-time performance parameters, a second feature label corresponding to the real-time performance parameters is dynamically determined according to preset performance parameter rules; wherein, the preset performance parameter rules are used to limit the correspondence between performance parameters and corresponding demand processing feature labels;

[0018] The target data processing feature label is determined in the algorithm library that matches at least one of the first feature label and the second feature label; wherein the determination of the target data processing feature label takes into account the performance pressure represented by the performance parameter.

[0019] Determine the target algorithm corresponding to the target data processing feature label.

[0020] In the above scheme, determining the target algorithm in the algorithm library that matches the multidimensional data features and whose corresponding data processing feature label matches the real-time performance parameters, based on the multidimensional data features and the real-time performance parameters, includes:

[0021] The multidimensional data features and the real-time performance parameters are input into a preset model to dynamically determine the target algorithm; wherein, the preset model is determined based on historical algorithms that process historical data in the algorithm library and the data processing feature labels corresponding to each historical algorithm; wherein, the process of determining the target algorithm by the preset model takes into account the performance pressure represented by the real-time performance parameters.

[0022] In the above scheme, the multidimensional data features include at least one of the following: data type, data size, statistical features, distribution features, and correlation features.

[0023] The method in the above scheme further includes:

[0024] During the process of processing the data to be processed using the target algorithm, the algorithm parameters of the target algorithm are dynamically adjusted based on the performance parameters obtained from real-time monitoring.

[0025] The method in the above scheme further includes:

[0026] Based on the multidimensional data characteristics and the query requirements for the data to be processed, the target data is stored in a storage space that meets the query requirements.

[0027] The method in the above scheme further includes:

[0028] During the process of processing the data to be processed using the target algorithm, a performance evaluation is performed based on the performance parameters obtained from real-time monitoring, and the performance evaluation result is determined.

[0029] A new target algorithm matching the performance evaluation results is determined, and the new target algorithm is used to process the data to be processed to determine the target data.

[0030] This application also provides a data processing apparatus, including:

[0031] The data acquisition unit is used to acquire data to be processed.

[0032] The determining unit is used to dynamically determine the target algorithm corresponding to the data to be processed based on the multidimensional data features of the data to be processed and the real-time performance parameters.

[0033] A data processing unit is used to process the data to be processed based on the target algorithm to determine the target data.

[0034] This application also provides an electronic device, including a memory and a processor. The memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the steps in the above-described method.

[0035] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the above-described method.

[0036] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps in the above-described method.

[0037] In this embodiment, the process involves acquiring data to be processed; dynamically determining the target algorithm corresponding to the data based on its multidimensional data characteristics and real-time performance parameters; and processing the data based on the target algorithm to determine the target data. This approach fully considers the multidimensional data characteristics and real-time performance parameters when determining the target algorithm, enabling the identification of a target algorithm that matches the data and current performance even when faced with large data volumes or performance anomalies. Furthermore, because the target algorithm is dynamically determined, if performance changes during data processing, a target algorithm matching the changed performance can be dynamically determined based on real-time performance parameters. This ensures that the determined target algorithm always adapts to both the data and performance, thereby improving the device's data read / write performance. Attached Figure Description

[0038] Figure 1 Flowchart of the data detection method provided in the embodiments of this application Figure 1 ;

[0039] Figure 2 This application provides an example of the effectiveness of the data detection method. Figure 1 ;

[0040] Figure 3 Flowchart of the data detection method provided in the embodiments of this application Figure 2 ;

[0041] Figure 4 Flowchart of the data detection method provided in the embodiments of this application Figure 3 ;

[0042] Figure 5 Flowchart of the data detection method provided in the embodiments of this application Figure 4 ;

[0043] Figure 6 Flowchart of the data detection method provided in the embodiments of this application Figure 5 ;

[0044] Figure 7 Flowchart of the data detection method provided in the embodiments of this application Figure 5 ;

[0045] Figure 8Flowchart of the data detection method provided in the embodiments of this application Figure 7 ;

[0046] Figure 9 Flowchart of the data detection method provided in the embodiments of this application Figure 8 ;

[0047] Figure 10 Flowchart of the data detection method provided in the embodiments of this application Figure 9 ;

[0048] Figure 11 This is a schematic diagram of the structure of the data detection device provided in the embodiments of this application;

[0049] Figure 12 This is a schematic diagram of a hardware entity of an electronic device provided in an embodiment of this application. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0051] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0052] If the application documents contain similar descriptions such as "first / second", the following explanation shall be added: In the following description, the terms "first / second / third" are used only to distinguish similar objects and do not represent a specific order of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0054] Among related technologies, ClickHouse is a high-performance columnar database management system widely used in large-scale data storage and real-time analysis scenarios. However, in actual production environments, ClickHouse often faces the following problems when handling peak data write times (such as 8 AM):

[0055] High CPU usage, especially during peak data write periods, significantly increases the CPU utilization of the ClickHouse cluster, leading to decreased query performance and potentially impacting system stability. In ClickHouse, the compression algorithm is statically specified by the CODEC during table creation and cannot be dynamically adjusted afterwards. This static compression algorithm selection mechanism has the following drawbacks in practical applications:

[0056] 1. Inability to adapt to resource pressure during peak data write periods. Disadvantage description: During peak data write periods (e.g., 8 AM), the CPU utilization of the ClickHouse cluster increases significantly, leading to decreased query performance, severely degraded user experience, and even system failures. Static compression algorithms cannot dynamically adjust based on real-time resource usage, causing high-compression-ratio algorithms (such as Zstandard) to consume excessive CPU resources during peak periods, while high-speed compression algorithms (such as LZ4) may not fully utilize storage space. Cause: ClickHouse's compression algorithm selection is static and cannot dynamically adjust compression strategies based on real-time metrics such as CPU utilization and data write rate.

[0057] 2. Uneven Resource Allocation. Disadvantage Description: Static compression algorithms cannot dynamically adjust based on data characteristics and system load, leading to uneven resource allocation. For example, when CPU resources are scarce, a high compression ratio algorithm may result in excessive computational overhead; conversely, when CPU resources are plentiful, a high-speed compression algorithm may not fully utilize system resources. Cause: ClickHouse's compression algorithm selection mechanism lacks dynamic awareness of data characteristics and system resources.

[0058] 3. Degraded Query Performance. Disadvantage Description: During peak data write periods, static compression algorithms may excessively consume CPU resources, thus impacting query performance. For example, high compression ratio algorithms may lead to increased query latency or even query timeouts during peak periods. Cause: Static compression algorithms cannot dynamically adjust their compression strategy based on query load, resulting in unoptimized query performance.

[0059] 4. Insufficient trade-off between storage and computational efficiency. Disadvantage description: Static compression algorithms cannot dynamically adjust based on real-time storage requirements and computational resource status, resulting in an insufficient trade-off between storage and computational efficiency. For example, when storage space is limited, high compression ratio algorithms may not be prioritized; conversely, when computational resources are limited, high-speed compression algorithms may not be prioritized. Cause: ClickHouse's compression algorithm selection mechanism lacks dynamic awareness of storage requirements and computational resources.

[0060] To address the aforementioned technical problems, this application provides a data processing method. Please refer to [link to relevant documentation]. Figure 1 The following is a flowchart illustrating the data processing method provided in the embodiments of this application. Figure 1 , will combine Figure 1 The steps shown are explained below:

[0061] S101. Obtain the data to be processed.

[0062] In this embodiment, the data processing device receives a data write request sent by a client or other terminal. The data write request includes data to be processed.

[0063] In this embodiment of the application, the process of determining the target algorithm is triggered after receiving the data write request.

[0064] In this embodiment of the application, after acquiring the data to be processed, the data processing device can perform data cleaning and formatting on the data to be processed, in order to remove redundant information and unify the data format.

[0065] The data processing device may include a server, cloud server, or terminal with corresponding data processing functions. In some embodiments, the data processing device may be configured with the ClickHouse analytics engine. The data to be processed may include logistics data, item transaction data, logistics volume data, item classification data, item ownership data, etc.

[0066] S102. Based on the multidimensional data features of the data to be processed and the real-time performance parameters, dynamically determine the target algorithm corresponding to the data to be processed.

[0067] In this embodiment, the data processing device analyzes and determines the corresponding multidimensional data features of the data to be processed, and determines the real-time performance parameters based on real-time monitoring performance indicators. Based on the multidimensional data features and real-time performance parameters, a target algorithm that matches the multidimensional data features and real-time performance parameters is dynamically determined from multiple data compression algorithms.

[0068] In this embodiment of the application, after determining a target algorithm, the data processing device can also monitor and acquire new performance parameters in real time, and determine a new target algorithm based on the new performance parameters and multidimensional data characteristics.

[0069] The real-time performance parameters include at least one of the following: system resource performance parameters, network condition performance parameters, and query load performance parameters; the multidimensional data features include at least one of the following: data type, data size, statistical features, distribution features, and correlation features.

[0070] S103. Based on the target algorithm, process the data to be processed to determine the target data.

[0071] In this embodiment, the data processing device uses a target algorithm to compress the data to be processed, thereby determining the compressed target data. The target data is then stored in a storage space that matches the query requirements corresponding to the data to be processed.

[0072] In this embodiment, the target algorithm may include: a high compression ratio algorithm, a high-speed compression algorithm, and a general algorithm. The high compression ratio algorithms include Zstandard and LZMA (Lempel-Ziv-Markov chain-Algorithm). The high-speed compression algorithms include Snappy and LZ4 (Lempel-Ziv-Markov chain-4). The general algorithm is Gzip (GNU Zip Compression Algorithm).

[0073] This application proposes a dynamic compression algorithm selection method based on ClickHouse to optimize CPU usage and query performance during peak data write periods. The following, in conjunction with the appendix... Figure 2 The technical solution will be described in detail, including its specific implementation details.

[0074] Data Feature Analysis Module 101: Analyzes the characteristics of the written data.

[0075] Compression algorithm selection module 102: Dynamically selects the optimal compression algorithm based on data characteristics, system resources, network conditions and query load.

[0076] System resource monitoring module 103: Real-time monitoring of system resource usage.

[0077] Network Condition Monitoring Module 104: Real-time monitoring of network conditions.

[0078] Query load monitoring module 105: Real-time monitoring of query load.

[0079] Dynamic compression algorithm execution module 106: Compresses and stores data according to the selected compression algorithm.

[0080] Performance Feedback and Optimization Module 107: Monitors system performance in real time and optimizes compression algorithm selection strategies.

[0081] In this embodiment, the process involves acquiring data to be processed; dynamically determining the target algorithm corresponding to the data based on its multidimensional data characteristics and real-time performance parameters; and processing the data based on the target algorithm to determine the target data. This approach fully considers the multidimensional data characteristics and real-time performance parameters when determining the target algorithm, enabling the identification of a target algorithm that matches the data and current performance even when faced with large data volumes or performance anomalies. Furthermore, because the target algorithm is dynamically determined, if performance changes during data processing, a target algorithm matching the changed performance can be dynamically determined based on real-time performance parameters. This ensures that the determined target algorithm always adapts to both the data and performance, thereby improving the device's data read / write performance.

[0082] Please see Figure 3 The following is a flowchart illustrating the data processing method provided in the embodiments of this application. Figure 2 , Figure 1 The illustrated S102 can also be implemented via S201 to S203, which will be explained in conjunction with the illustrated steps:

[0083] S201. Feature extraction is performed on the preprocessed data to be processed to determine the multidimensional data features; wherein, the preprocessing is used to characterize the data cleaning and formatting processing performed on the data to be processed.

[0084] In this embodiment, after acquiring the data to be processed, the data processing device can perform data cleaning to remove redundant information and format the data. Then, based on the preset feature types of multidimensional data features, multidimensional data features are extracted from the data to be processed.

[0085] In this embodiment, the multidimensional data features include at least one of the following: data type, data size, statistical features, distribution features, and correlation features. Data type includes structured data, semi-structured data, and unstructured data. Data size includes "long text" and "short text". Statistical features include entropy, mean, and variance. Distribution features include peak values, outliers, extreme values, and frequency distribution. Correlation features include autocorrelation and cross-correlation.

[0086] For example, the data feature analysis module 101 is one of the core modules of the dynamic compression algorithm selection system. It is responsible for classifying and extracting features from the written data, providing a scientific basis for the selection of compression algorithms. The following is a detailed description of this module.

[0087] Data preprocessing is the first step in data feature analysis. Its purpose is to clean and organize the raw data, remove redundant information, and categorize it into types suitable for subsequent processing. The specific steps are as follows:

[0088] Step 1: Data cleaning:

[0089] Removing redundant information: Identify and delete duplicate records, null values or invalid data in the data. For example, in text data, remove stop words (such as "de", "shi", etc.) and punctuation marks; in numerical data, remove outliers or values beyond a reasonable range. Data formatting: Convert the data into a unified format for subsequent processing. For example, unify timestamps to Coordinated Universal Time, convert text data to UTF-8 (8-bit, Universal Character Set / Unicode Transformation Format) encoding, and standardize numerical data to a unified unit.

[0090] Step 2: Data classification:

[0091] Dividing data types: According to the structure and characteristics of the data, divide it into the following categories.

[0092] Structured data: Data with a fixed format, such as table data in a relational database (navigation data falls into this category).

[0093] Semi-structured data: Partially structured data, such as JS key-value pair data (JavaScript Object Notation, JSON) and Extensible Markup Language (XML) files.

[0094] Unstructured data: Data without a fixed format, such as text, logs, audio, etc.

[0095] Data labeling: Add labels to each type of data to facilitate subsequent feature extraction and compression algorithm selection. For example, add the label "long text" or "short text" to text data, and add the label "high density" or "low density" to numerical data.

[0096] Feature extraction:

[0097] Feature extraction is to extract key statistical features from the preprocessed data, which is used to describe the characteristics of the data and provide a basis for selecting a compression algorithm. The specific methods are as follows:

[0098] Step 3: Calculate statistical features:

[0099] Entropy value: Used to measure the randomness or information content of data.

[0100] It can be calculated by formula (1):

[0101]

[0102] Where p(xi) is the probability of the (i)th symbol appearing in the data to be processed, H(X) is the entropy value, and n is the total number of characters in the data to be processed.

[0103] Mean: Used to describe the central tendency of data.

[0104] It can be calculated using formula (2):

[0105]

[0106] Where, x i It is the i-th value in the data, and n is the total number of data. It is the mean

[0107] Variance: Used to measure the dispersion of data.

[0108] It can be calculated using formula (3):

[0109]

[0110] Where, σ 2 It is variance.

[0111] Step 4: Extract data distribution features:

[0112] Histogram analysis: By drawing histograms of data, we can analyze the distribution characteristics of the data. For example, for numerical data, histograms can reflect the frequency distribution of the data; for text data, histograms can reflect the distribution of word frequencies.

[0113] Peak detection: Identifying peak values ​​or outliers in data. For example, detecting abnormal fluctuations in time series data; detecting extreme values ​​in numerical data.

[0114] Step 5: Extract data correlation features

[0115] Autocorrelation: Used to measure the correlation of data across different times or spaces.

[0116] It can be calculated using formula (4):

[0117]

[0118] Where, x i For the i-th observation, Let n be the mean, n be the total number, k be the lag order, and Rk be the autocorrelation.

[0119] Cross-correlation: Used to measure the correlation between different data. For example, in multidimensional data, cross-correlation can reflect the relationship between different dimensions.

[0120] It can be calculated using formula (5):

[0121]

[0122] The normalization formula (5) yields:

[0123]

[0124] Step 6: Generate feature vectors (multidimensional data features)

[0125] The extracted data type, size, statistical characteristics, distribution characteristics, and correlation characteristics are combined into a feature vector to describe the overall characteristics of the data. For example, for text data, the feature vector may include entropy, word frequency distribution, keyword density, etc.; for numerical data, the feature vector may include mean, variance, autocorrelation, etc. The data feature analysis module generates feature vectors describing the data characteristics through data preprocessing and feature extraction, providing a scientific basis for the selection of compression algorithms.

[0126] S202. Monitor local performance indicators in real time and determine the real-time performance parameters; wherein the real-time performance parameters include at least one of the following: system resource performance parameters, network condition performance parameters, and query load performance parameters.

[0127] In this embodiment of the application, the data processing device can monitor local system resources, network conditions and query load in real time through monitoring tools, and determine system resource performance parameters, network condition performance parameters and query load performance parameters.

[0128] Monitoring tools may include Prometheus and Grafana.

[0129] System resource performance parameters may include: CPU utilization, memory utilization, and disk input / output (I / O). Network condition performance parameters may include: network bandwidth, network latency, and network load. Query load performance parameters may include: query latency, query concurrency, and query error rate.

[0130] S203. Based on the multidimensional data features, the real-time performance parameters, and the algorithm library, dynamically determine the target algorithm for the data to be processed; wherein, the algorithm library includes: multiple data compression algorithms, and data processing feature labels corresponding to each data compression algorithm.

[0131] In this embodiment of the application, the data processing device can dynamically determine a target algorithm from the algorithm library that matches the multidimensional data features and whose corresponding data processing feature label matches the real-time performance parameters, based on multidimensional data features and real-time performance parameters.

[0132] In this embodiment, the data processing device can dynamically determine a first target algorithm from the algorithm library that matches the multidimensional data features and whose corresponding data processing feature label matches the real-time performance parameters, based on multidimensional data features and real-time performance parameters at a first time. Furthermore, when the real-time performance parameters change at a second time, the device can dynamically determine a second target algorithm from the algorithm library that matches the multidimensional data features and whose corresponding data processing feature label matches the real-time performance parameters, based on both the multidimensional data features and the real-time performance parameters at the second time.

[0133] In this embodiment, the data processing device can further analyze the multidimensional data features and real-time performance parameters according to preset performance indicator rules, identify indicators that match the multidimensional data features and real-time performance parameters, determine the corresponding data processing feature label in the algorithm library based on the indicator, and then determine the target algorithm based on the label. The preset performance indicator rules may include the correspondence between multidimensional data features and performance parameters and their corresponding performance indicators.

[0134] The algorithm library includes multiple data compression algorithms and corresponding data processing feature tags for each algorithm. These data processing feature tags define the performance metrics of the corresponding algorithms, including compression ratio, compression speed, decompression speed, and CPU utilization.

[0135] The matching of the target algorithm with multidimensional data features may include: the corresponding target algorithm is applicable to processing at least one of the following data to be processed: data type, data distribution features, data statistical features, and data correlation features.

[0136] The matching of data processing feature labels with real-time performance parameters can include: the performance indicators corresponding to the target algorithm can be used to alleviate the performance pressure represented by the real-time performance parameters when processing the data to be processed.

[0137] For example, the compression algorithm selection module 102 is one of the core modules of the dynamic compression algorithm selection system, responsible for dynamically selecting the optimal compression algorithm based on data characteristics, system resources, network conditions, and query load. The following is a detailed description of this module. The main functions of the compression algorithm selection module 102 include:

[0138] Dynamic compression algorithm selection: Select the optimal compression algorithm based on real-time data characteristics, system resources, network conditions, and query load.

[0139] Optimize resource utilization: Reduce CPU usage and improve query performance while ensuring data compression efficiency.

[0140] Adaptable to diverse scenarios: Supports multiple compression algorithms (such as Zstandard, LZ4, Snappy, etc.) to adapt to different data characteristics and network conditions.

[0141] In this embodiment, feature extraction is performed on the preprocessed data to be processed to determine multidimensional data features. Preprocessing characterizes the data cleaning and formatting processes performed on the data. Real-time monitoring of local performance metrics determines real-time performance parameters, including at least one of the following: system resource performance parameters, network condition performance parameters, and query load performance parameters. A target algorithm is dynamically determined based on the multidimensional data features, real-time performance parameters, and an algorithm library. The algorithm library includes multiple data compression algorithms and corresponding data processing feature tags for each data compression algorithm. This allows for the identification of a target algorithm that matches the data and current performance when dealing with large data volumes or performance anomalies. Furthermore, because the target algorithm is dynamically determined, if performance changes during data processing, a target algorithm matching the changed performance can be dynamically determined based on the real-time performance parameters. This ensures that the determined target algorithm always adapts to the data and performance, thereby improving the device's data read / write performance.

[0142] Please see Figure 4 The following is a flowchart illustrating the data processing method provided in the embodiments of this application. Figure 3 , Figure 3 The illustrated S203 can also be implemented via S301, which will be explained in conjunction with the illustrated steps:

[0143] S301. Based on the multidimensional data features, the performance parameters, and the algorithm library, dynamically determine the target algorithm that matches the multidimensional data features and whose corresponding data processing feature label matches the real-time performance parameters.

[0144] In this embodiment, the data processing device analyzes multidimensional data features and real-time performance parameters according to preset performance indicator rules to obtain performance indicators corresponding to the multidimensional data features and real-time performance parameters. Then, based on these two indicators, it dynamically determines a data processing feature label corresponding to at least one of the two indicators from the algorithm library, and subsequently determines the target algorithm based on this label.

[0145] In this embodiment, when the performance parameters only include network condition performance parameters, the performance indicators can be dynamically determined based solely on the network bandwidth and network latency characterized by the network conditions. Then, the target algorithm is determined based on the statistical characteristics in the multidimensional data features and / or the determined performance indicators.

[0146] For example, the implementation of the compression algorithm selection module 102 includes the following steps:

[0147] Step 1: Build the algorithm library:

[0148] Algorithm library maintenance: Maintain an algorithm library containing various compression algorithms, including high compression ratio algorithms (such as Zstandard, LZMA), high-speed compression algorithms (such as Snappy, LZ4), and general-purpose algorithms (Gzip).

[0149] Performance metrics definition: Define performance metrics (data processing feature labels) for each algorithm, including compression ratio, compression speed, decompression speed, and CPU utilization.

[0150] Step 2: Data characteristics and network condition analysis.

[0151] Data feature analysis—Based on the results of the data classification and feature extraction modules, the characteristics of the data are obtained, including the following:

[0152] Data size: Total data size and block size.

[0153] Data types: text, images, time series, etc.

[0154] Statistical characteristics: entropy, mean, variance.

[0155] Distribution characteristics: histogram, peak detection.

[0156] Correlation characteristics: autocorrelation, cross-correlation.

[0157] Network condition analysis: Real-time monitoring of the network environment to obtain parameters such as network bandwidth, network latency, and network load.

[0158] Step 3: Strategy Model or Rule Base

[0159] Rule base definition: Using a predefined rule base, a compression algorithm is selected based on data characteristics and network conditions. The rule base includes the following rules.

[0160] Network condition rules:

[0161] When the network bandwidth is greater than 100Mbps and the latency is less than 50ms, choose a high compression ratio algorithm (such as Zstandard).

[0162] When the network bandwidth is less than 50Mbps and the latency is greater than 100ms, choose a high-speed compression algorithm (such as Snappy).

[0163] Data feature rules:

[0164] When the data entropy is high and the variance is large, choose a high compression ratio algorithm (such as Zstandard).

[0165] When the data entropy is low and the variance is small, choose a high-speed compression algorithm (such as LZ4).

[0166] In this embodiment, a target algorithm is dynamically determined based on the multidimensional data features, the performance parameters, and the algorithm library. This target algorithm matches the multidimensional data features, and the corresponding data processing feature label matches the real-time performance parameters. This allows for the identification of a target algorithm that matches both the data volume and current performance when faced with large datasets or performance anomalies. Furthermore, because the target algorithm is dynamically determined, if performance changes during data processing, a target algorithm matching the changed performance can be dynamically determined based on the real-time performance parameters. This ensures that the determined target algorithm always adapts to both the data and performance, thereby improving the device's data read / write performance.

[0167] Please see Figure 5 The following is a flowchart illustrating the data processing method provided in the embodiments of this application. Figure 4 , Figure 4 The illustrated S301 can also be implemented via S401 to S404, which will be explained in conjunction with the illustrated steps:

[0168] S401. Based on the multidimensional data features, analyze according to preset data feature rules to determine the first feature label corresponding to the multidimensional data features.

[0169] In this embodiment, the preset data feature rule is used to define the correspondence between data features and corresponding requirement processing feature labels. After determining the multidimensional data features, a first feature label corresponding to the multidimensional data features can be determined in the preset data feature rule based on the multidimensional data features.

[0170] For example, a high compression ratio label can be selected from a preset data feature rule base based on the high data entropy value and large variance in multidimensional data features. Then, a high compression ratio algorithm (such as Zstandard) can be determined based on the high compression ratio label.

[0171] For example, a high-speed compression label can be selected from a preset data feature rule base based on the low data entropy and small variance in multidimensional data features, and then a high-speed compression algorithm (such as LZ4) can be determined based on the high-speed compression label.

[0172] S402. Based on the real-time performance parameters, analyze according to preset performance parameter rules, and dynamically determine the second feature label corresponding to the real-time performance parameters.

[0173] In this embodiment, the preset performance parameter rule is used to define the correspondence between performance parameters and corresponding demand processing feature tags. The data processing device can analyze the real-time performance parameters according to the preset performance parameter rule to determine the second feature tag corresponding to the performance parameters.

[0174] For example, when the network bandwidth is greater than 100Mbps and the latency is less than 50ms, a high compression ratio label is selected. A high compression ratio algorithm (such as Zstandard) can then be determined based on this label. When the network bandwidth is less than 50Mbps and the latency is greater than 100ms, a high-speed compression label is selected, and a high-speed compression algorithm (such as Snappy) can then be determined based on this label.

[0175] S403. Determine a target data processing feature label in the algorithm library that matches at least one of the first feature label and the second feature label.

[0176] In this embodiment of the application, after determining the first label and the second label, the data processing device can comprehensively consider the system resource performance pressure, network condition performance pressure and query load performance pressure represented by the performance parameters, and determine the target data processing feature label that matches at least one of the first feature label and the second feature label in the algorithm library.

[0177] In this embodiment of the application, the data processing device can comprehensively consider the system resource performance pressure, network condition performance pressure and query load performance pressure represented by the performance parameters, and determine the target data processing feature label in the algorithm library that matches at least one of the first feature label and the second feature label and can be used to reduce the excessive performance pressure.

[0178] For example, the system resource performance pressure, network condition performance pressure, and query load performance pressure represented by the performance parameters can be achieved through the following:

[0179] The system resource monitoring module 103 is one of the core modules of the dynamic compression algorithm selection system. It is responsible for real-time monitoring of system resources (such as CPU utilization, memory usage, disk I / O, etc.) to provide a basis for compression algorithm selection. The following is a detailed description of this module. The main functions of the system resource monitoring module include: real-time monitoring of system resources, including CPU utilization, memory usage, disk I / O, etc.

[0180] Resource pressure assessment: Assess system resource pressure based on real-time monitoring data.

[0181] Provide a basis for compression algorithm selection: Feed back system resource usage to the compression algorithm selection module to help dynamically select the optimal compression algorithm.

[0182] The implementation of the system resource monitoring module 103 is divided into the following steps:

[0183] Step 1: CPU occupancy monitoring

[0184] Data collection: Collect CPU occupancy data in real time through system monitoring tools (such as Prometheus, Grafana).

[0185] Data processing: Calculate the average value, peak value and fluctuation range of the CPU occupancy.

[0186] Threshold setting: Set the threshold of CPU occupancy for evaluating system resource pressure. For example: CPU occupancy > 80%: high resource pressure. 50% < CPU occupancy ≤ 80%: medium resource pressure. CPU occupancy ≤ 50%: low resource pressure.

[0187] Step 2: Memory usage monitoring

[0188] Data collection: Collect memory usage data in real time through system monitoring tools.

[0189] Data processing: Calculate the average value, peak value and fluctuation range of the memory usage.

[0190] Threshold setting: Set the threshold of memory usage for evaluating system resource pressure. For example: memory usage > 80%: high resource pressure. 50% < memory usage ≤ 80%: medium resource pressure. memory usage ≤ 50%: low resource pressure.

[0191] Step 3: Disk I / O monitoring

[0192] Data collection: Collect disk I / O data in real time through system monitoring tools, including read / write speed and I / O waiting time.

[0193] Data processing: Calculate the average value, peak value and fluctuation range of the disk I / O.

[0194] Threshold setting: Set the threshold of disk I / O for evaluating system resource pressure. For example: I / O waiting time > 50ms: high resource pressure. 20ms < I / O waiting time ≤ 50ms: medium resource pressure. I / O waiting time ≤ 20ms: low resource pressure.

[0195] Step 4: Resource pressure assessment

[0196] Comprehensive assessment: Comprehensively evaluate system resource pressure based on the monitoring data of CPU occupancy, memory usage and disk I / O.

[0197] Feedback mechanism: Feed back the resource pressure assessment result to the compression algorithm selection module to help dynamically select the optimal compression algorithm.

[0198] Assume the real-time monitoring data from the system resource monitoring module is as follows: CPU utilization: 75% (under resource pressure). Memory utilization: 65% (under resource pressure). Disk I / O: I / O wait time 30ms (under resource pressure).

[0199] The processing steps of the system resource monitoring module are as follows:

[0200] Data Acquisition: Collect CPU utilization, memory usage, and disk I / O data through system monitoring tools.

[0201] Data processing: Calculate the average value, peak value, and fluctuation range of each indicator.

[0202] Threshold setting: Assess system resource pressure based on preset thresholds.

[0203] Overall assessment: Based on the combined assessment results of CPU utilization, memory usage, and disk I / O, the system resource pressure is determined to be moderate.

[0204] Feedback mechanism: The resource pressure assessment results are fed back to the compression algorithm selection module to help dynamically select the optimal compression algorithm.

[0205] The network condition monitoring module 104 is one of the core modules of the dynamic compression algorithm selection system. It is responsible for real-time monitoring of network conditions (such as bandwidth, latency, network load, etc.) to provide a basis for compression algorithm selection. The following is a detailed description of this module.

[0206] The main functions of the network condition monitoring module include:

[0207] Real-time monitoring of network conditions, including network bandwidth, network latency, and network load.

[0208] Network stress assessment: Assess network stress based on real-time monitoring data.

[0209] Provide a basis for compression algorithm selection: Feed back the network condition monitoring results to the compression algorithm selection module 102 to help dynamically select the optimal compression algorithm.

[0210] The implementation of the network condition monitoring module 104 involves the following steps:

[0211] Step 1: Network bandwidth monitoring

[0212] Data Acquisition: Use network monitoring tools (such as NetFlow, sFlow) to collect network bandwidth data in real time.

[0213] Data processing: Calculate the average, peak, and fluctuation range of network bandwidth.

[0214] Threshold Setting: Sets a network bandwidth threshold to assess network stress. For example: Network bandwidth < 50Mbps: High network stress. 50Mbps ≤ Network bandwidth < 100Mbps: Medium network stress. Network bandwidth ≥ 100Mbps: Low network stress.

[0215] Step 2: Network latency monitoring

[0216] Data Acquisition: Use Ping or Traceroute tools to collect network latency data in real time.

[0217] Data processing: Calculate the average, peak, and fluctuation range of network latency.

[0218] Threshold setting: Set a threshold for network latency to assess network stress. For example: Network latency > 100ms: High network stress.

[0219] Network latency 50ms < 100ms: Medium network load. Network latency ≤ 50ms: Low network load.

[0220] Step 3: Network load monitoring

[0221] Data Acquisition: Use network monitoring tools to collect network load data in real time, including packet loss rate and network utilization.

[0222] Data processing: Calculate the average, peak, and fluctuation range of network load.

[0223] Threshold Setting: Sets a threshold for network load to assess network stress. For example: Packet loss rate > 5%: High network stress. 1% < Packet loss rate ≤ 5%: Medium network stress. Packet loss rate ≤ 1%: Low network stress.

[0224] Step 4: Network Stress Assessment

[0225] Comprehensive assessment: Based on monitoring data of network bandwidth, network latency, and network load, a comprehensive assessment of network stress is conducted.

[0226] Feedback mechanism: The network pressure assessment results are fed back to the compression algorithm selection module 102 to help dynamically select the optimal compression algorithm.

[0227] Assume the real-time monitoring data from network condition monitoring module 104 is as follows: Network bandwidth: 80Mbps (medium network load). Network latency: 60ms (medium network load). Network load: Packet loss rate 2% (medium network load). The processing steps of the network condition monitoring module are as follows:

[0228] Data Acquisition: Collect network bandwidth, network latency, and network load data using network monitoring tools.

[0229] Data processing: Calculate the average value, peak value, and fluctuation range of each indicator.

[0230] Threshold setting: Assess network stress based on preset thresholds.

[0231] Overall assessment: Based on the combined assessment results of network bandwidth, network latency, and network load, the network pressure is determined to be moderate.

[0232] Feedback mechanism: The network stress assessment results are fed back to the compression algorithm selection module to help dynamically select the optimal compression algorithm.

[0233] The query load monitoring module 105 is one of the core modules of the dynamic compression algorithm selection system. It is responsible for real-time monitoring of query load (such as query latency and query concurrency) to provide a basis for compression algorithm selection. The following is a detailed description of this module.

[0234] The main functions of the load monitoring module include:

[0235] Real-time monitoring of query load, including query latency, query concurrency, and query error rate.

[0236] Query load assessment: Assess query load based on real-time monitoring data.

[0237] Provide a basis for compression algorithm selection: Feed back the query load monitoring results to the compression algorithm selection module 102 to help dynamically select the optimal compression algorithm.

[0238] The implementation of the load monitoring module involves the following steps:

[0239] Step 1: Query latency monitoring

[0240] Data Acquisition: Query latency data is collected in real time through ClickHouse's query logs.

[0241] Data processing: Calculate the average, peak, and fluctuation range of query latency.

[0242] Threshold setting: Sets a threshold for query latency to assess query pressure. For example: Query latency > 500ms: High query pressure.

[0243] Query latency 200ms < 500ms: Medium query load. Query latency ≤ 200ms: Low query load.

[0244] Step 2: Query concurrency monitoring

[0245] Data Acquisition: Real-time collection of query concurrency data through system monitoring tools (such as Prometheus and Grafana).

[0246] Data processing: Calculate the average, peak, and fluctuation range of query concurrency.

[0247] Threshold setting: Sets a threshold for query concurrency to assess query pressure. For example: Query concurrency > 100: High query pressure. 50 < Query concurrency ≤ 100: Medium query pressure. Query concurrency ≤ 50: Low query pressure.

[0248] Step 3: Query error rate monitoring

[0249] Data collection: Query error rate data is collected in real time through ClickHouse's query logs.

[0250] Data processing: Calculate the average, peak, and fluctuation range of the query error rate.

[0251] Threshold setting: Sets a threshold for the query error rate, used to assess query pressure. For example:

[0252] Query error rate > 5%: High query pressure. Query error rate ≤ 5% (1% < 5%): Medium query pressure. Query error rate ≤ 1%: Low query pressure.

[0253] Step 4: Check the stress assessment

[0254] Comprehensive assessment: Based on monitoring data of query latency, query concurrency, and query error rate, a comprehensive assessment of query pressure is conducted.

[0255] Feedback mechanism: The query pressure assessment results are fed back to the compression algorithm selection module to help dynamically select the optimal compression algorithm.

[0256] Assume the real-time monitoring data from load monitoring module 105 is as follows: Query latency: 300ms (under query pressure). Query concurrency: 80 (under query pressure). Query error rate: 2% (under query pressure).

[0257] The processing steps for querying the load monitoring module are as follows:

[0258] Data collection: Query latency, query concurrency, and query error rate data are collected through ClickHouse's query logs and system monitoring tools.

[0259] Data processing: Calculate the average value, peak value, and fluctuation range of each indicator.

[0260] Threshold setting: Evaluate query pressure based on preset thresholds.

[0261] Overall assessment: Based on the combined assessment results of query latency, query concurrency, and query error rate, the query pressure is determined to be moderate.

[0262] Feedback mechanism: The query pressure assessment results are fed back to the compression algorithm selection module 102 to help dynamically select the optimal compression algorithm.

[0263] S404. Determine the target algorithm corresponding to the target data processing feature label.

[0264] In this embodiment of the application, the target algorithm corresponding to the target data processing feature label is determined.

[0265] For example, suppose we have a set of text data in a network environment with a bandwidth of 80Mbps, a latency of 60ms, high data entropy, and large variance. The decision-making process for the compression algorithm selection module is as follows:

[0266] Data characteristic analysis: Data size is 1GB, high entropy, large variance. Network condition analysis: Bandwidth 80Mbps, latency 60ms, medium load.

[0267] Rule base matching: Select a high compression ratio algorithm (such as Zstandard) based on the rule base. Dynamic adjustment: Dynamically switch to a high-speed compression algorithm (such as Snappy) when the network bandwidth drops to 30Mbps and the latency increases to 120ms.

[0268] In this embodiment, based on multidimensional data features and according to preset data feature rules, a first feature label corresponding to the multidimensional data features is determined. The preset data feature rules define the correspondence between data features and algorithm processing feature labels. Based on real-time performance parameters and according to preset performance parameter rules, a second feature label corresponding to the real-time performance parameters is dynamically determined. The preset performance parameter rules define the correspondence between performance parameters and algorithm processing feature labels. A target data processing feature label matching at least one of the first and second feature labels is determined from the algorithm library. The determination of the target data processing feature label considers the performance pressure represented by the performance parameters. A target algorithm corresponding to the target data processing feature label is determined. This allows for the determination of a target algorithm that matches the data and current performance when facing large amounts of data or performance anomalies. Furthermore, because the target algorithm is dynamically determined, if performance changes during the processing of data, a target algorithm matching the changed performance can be dynamically determined based on real-time performance parameters. This ensures that the determined target algorithm always adapts to the data and performance, thereby improving the device's data read / write performance.

[0269] Please see Figure 6 The following is a flowchart illustrating the data processing method provided in the embodiments of this application. Figure 5 , Figure 4 The illustrated S301 can also be implemented via S501, which will be explained in conjunction with the illustrated steps:

[0270] S501. Input the multidimensional data features and the real-time performance parameters into a preset model to dynamically determine the target algorithm; wherein, the preset model is determined based on historical algorithms that process historical data in the algorithm library and the data processing feature labels corresponding to each historical algorithm.

[0271] In this embodiment, a preset model can be determined based on historical algorithms for processing historical data and the data processing feature labels corresponding to each historical algorithm. Then, multidimensional data features and real-time performance parameters are input into the preset model to dynamically determine the target algorithm. The process of determining the target algorithm using the preset model takes into account the performance pressure represented by the real-time performance parameters.

[0272] In this embodiment of the application, the process of determining the target algorithm by the preset model, which takes into account the performance pressure represented by the real-time performance parameters, may include: the preset model can comprehensively consider the system resource performance pressure, network condition performance pressure, and query load performance pressure represented by the performance parameters, and determine the target algorithm that matches the multidimensional data features and the real-time performance parameters and can be used to reduce the performance pressure exceeding the limit.

[0273] For example, a pre-defined model (such as a decision tree) is used to predict the optimal compression algorithm. Training data is derived from compression algorithm performance metrics in historical data. Input features include data size, entropy, mean, bandwidth, and latency; the output is a recommended target algorithm.

[0274] Algorithm selection: Based on the output of the preset model or rule base, select the optimal compression algorithm. Performance verification: Test the performance of the selected algorithm in a real-world environment to verify whether it meets the expected goals. If the performance does not meet the requirements, select a different algorithm.

[0275] In this embodiment, the multidimensional data features and the real-time performance parameters are input into a preset model to dynamically determine the target algorithm. The preset model is trained based on historical algorithms that process historical data and the corresponding data processing feature labels for each historical algorithm. Thus, when performance parameters are unstable or performance pressure is high, the preset model can output a matching target algorithm to alleviate performance pressure and improve the device's data read / write performance.

[0276] Please see Figure 7 The following is a flowchart illustrating the data processing method provided in the embodiments of this application. Figure 6 The following will be explained in conjunction with the steps shown:

[0277] S601. During the process of processing the data to be processed using the target algorithm, the algorithm parameters of the target algorithm are dynamically adjusted based on the performance parameters obtained from real-time monitoring.

[0278] In this embodiment, during the processing of the data to be processed using the target algorithm, the data processing device dynamically adjusts the algorithm parameters of the target algorithm based on the performance parameters obtained in real time through system monitoring tools. This alleviates the performance pressure represented by the performance parameters.

[0279] For example, the dynamic compression algorithm execution module 106 is one of the core modules of the dynamic compression algorithm selection system, responsible for compressing and storing data according to the output of the compression algorithm selection module. The following is a detailed description of this module.

[0280] The main functions of the dynamic compression algorithm execution module include:

[0281] Data compression: Compress the data according to the selected compression algorithm.

[0282] Data storage: Store the compressed data in ClickHouse.

[0283] Data Query: Supports efficient querying of compressed data.

[0284] The implementation of the dynamic compression algorithm execution module 106 consists of the following steps:

[0285] Step 1: Data Compression

[0286] Compression Algorithm Selection: Based on the output of the compression algorithm selection module 102, the optimal compression algorithm is selected. Data Compression: The selected compression algorithm is used to compress the data. For example, for text data, Zstandard is used for high compression ratio compression. For time series data, LZ4 is used for high-speed compression.

[0287] Compression parameter adjustment: Dynamically adjust compression parameters based on system resources and network conditions. For example, when CPU resources are plentiful, use a higher compression level to increase the compression ratio. When CPU resources are limited, use a lower compression level to reduce computational overhead.

[0288] In this embodiment, during the processing of the data to be processed using the target algorithm, the algorithm parameters of the target algorithm are dynamically adjusted based on performance parameters obtained through real-time monitoring. This allows for real-time adjustment of the corresponding target algorithm parameters when performance pressure exceeds limits, thereby alleviating performance pressure and improving the device's data read / write performance.

[0289] Please see Figure 8 The following is a flowchart illustrating the data processing method provided in the embodiments of this application. Figure 7 The following will be explained in conjunction with the steps shown:

[0290] S701. Based on the multidimensional data characteristics and the query requirements for the data to be processed, the target data is stored in a storage space that meets the query requirements.

[0291] In this embodiment of the application, the data processing device can store the target data in a storage space that meets the query requirements based on the multidimensional data features and the query requirements for the data to be processed.

[0292] For example, the implementation of the dynamic compression algorithm execution module 106 includes the following steps:

[0293] Data Partitioning: Based on data characteristics and query requirements, the compressed data is stored in appropriate partitions. For example, time-series data is partitioned by time period, and geographic data is partitioned by geographic location. Data Storage: The compressed data is stored in ClickHouse, supporting efficient querying and analysis.

[0294] Query optimization: Optimize query performance based on compression algorithms and data partitioning. For example, optimize decompression performance for high compression ratio algorithms and query response time for high-speed compression algorithms. Query execution: Support efficient querying of compressed data, ensuring query performance meets requirements.

[0295] Assuming the output of the compression algorithm selection module is high compression ratio compression using Zstandard, the processing steps of the dynamic compression algorithm execution module are as follows:

[0296] Data compression: Use Zstandard to compress the data at a high compression ratio, with the compression level set to 5.

[0297] Data storage: The compressed data is partitioned by time and stored in ClickHouse.

[0298] Data Query: Optimize query performance to ensure efficient querying of compressed data.

[0299] In this embodiment, based on the multidimensional data characteristics and the query requirements for the data to be processed, the target data is stored in a storage space that meets the query requirements. Because the target data is stored in a space that meets the query requirements, the target data can be retrieved quickly and conveniently when querying it, thus improving data query performance.

[0300] Please see Figure 9 The following is a flowchart illustrating the data processing method provided in the embodiments of this application. Figure 8 The following will be explained in conjunction with the steps shown:

[0301] S801. During the process of processing the data to be processed using the target algorithm, a performance evaluation is performed based on the performance parameters obtained from real-time monitoring, and the performance evaluation result is determined.

[0302] In this embodiment of the application, during the process of processing the data to be processed using the target algorithm, the data processing device can monitor and obtain performance parameters in real time based on system monitoring tools, and then perform performance evaluation on the performance parameters to determine the performance evaluation result.

[0303] In this embodiment, the data processing device can compare system resource performance parameters with a first preset value, network condition performance parameters with a second preset value, and query load performance parameters with a third preset value to determine the performance evaluation result. The performance evaluation result is used to characterize the performance pressure represented by the corresponding performance parameters.

[0304] S802. Determine a new target algorithm that matches the performance evaluation result, and use the new target algorithm to process the data to be processed to determine the target data.

[0305] In this embodiment of the application, the data processing device determines a new target algorithm that matches the performance evaluation result based on the performance evaluation result, that is, determines a new target algorithm to alleviate the performance pressure represented by the performance evaluation result, and uses the new target algorithm to process the data to be processed to determine the target data.

[0306] For example, the performance feedback and optimization module 107 is one of the core modules of the dynamic compression algorithm selection system. It is responsible for real-time monitoring of system performance (such as compression ratio, query latency, CPU utilization, etc.) and optimizing the compression algorithm selection strategy based on the monitoring results. The following is a detailed description of this module.

[0307] The main functions of the performance feedback and optimization module 107 include:

[0308] Real-time monitoring of system performance, including compression ratio, query latency, CPU utilization, and memory usage.

[0309] Performance evaluation: Based on monitoring data, evaluate whether the system performance meets the expected goals.

[0310] Optimize compression algorithm selection strategy: Based on performance evaluation results, dynamically adjust the compression algorithm selection strategy to improve system performance.

[0311] The implementation of the performance feedback and optimization module involves the following steps:

[0312] Step 1: Performance Monitoring

[0313] Compression ratio monitoring: Monitor the compression ratio to evaluate the efficiency of the compression algorithm. For example: Compression ratio = Data size before compression / Data size after compression.

[0314] Query latency monitoring: Monitor query latency and evaluate query performance. For example: Query latency = Query completion time - Query start time.

[0315] CPU utilization monitoring: Monitor CPU utilization to assess system resource usage. For example: CPU utilization = CPU usage time / total time.

[0316] Memory usage monitoring: Monitor memory usage to assess system resource utilization. For example: Memory usage = Used memory / Total memory.

[0317] Step 2: Performance Evaluation

[0318] Compression ratio evaluation: Evaluate the efficiency of the compression algorithm based on compression ratio monitoring data. For example: Compression ratio > 5: High compression efficiency. 2 < Compression ratio ≤ 5: Medium compression efficiency. Compression ratio ≤ 2: Low compression efficiency.

[0319] Query latency assessment: Evaluate query performance based on query latency monitoring data. For example: Query latency < 200ms: High query performance. 200ms ≤ Query latency < 500ms: Medium query performance. Query latency ≥ 500ms: Low query performance.

[0320] CPU utilization assessment: Based on CPU utilization monitoring data, assess system resource usage. For example: CPU utilization < 50%: Resource usage is reasonable. 50% ≤ CPU utilization < 80%: Resource usage is high. CPU utilization ≥ 80%: Resource usage is excessive.

[0321] Memory usage assessment: Based on memory usage monitoring data, assess system resource usage. For example: Memory usage < 50%: Resource usage is reasonable. 50% ≤ Memory usage < 80%: Resource usage is high. Memory usage ≥ 80%: Resource usage is excessive.

[0322] Step 3: Optimize the compression algorithm selection strategy

[0323] Strategy Adjustment: Dynamically adjust the compression algorithm selection strategy based on performance evaluation results. For example, if the compression ratio is low and CPU utilization is high, prioritize high-speed compression algorithms (such as LZ4). If query latency is high and memory utilization is high, prioritize low-memory compression algorithms (such as Snappy).

[0324] Feedback mechanism: The performance evaluation results are fed back to the compression algorithm selection module 102 to help dynamically select the optimal compression algorithm.

[0325] Assume the real-time monitoring data of the performance feedback and optimization module is as follows: Compression ratio: 3 (medium compression efficiency). Query latency: 00ms (medium query performance). CPU utilization: 70% (high resource usage). Memory utilization: 60% (high resource usage). The processing steps of the performance feedback and optimization module are as follows:

[0326] Performance monitoring: Monitor compression ratio, query latency, CPU utilization, and memory usage.

[0327] Performance assessment: Based on monitoring data, the system performance is assessed as moderate.

[0328] Strategy Adjustment: Based on the performance evaluation results, dynamically adjust the compression algorithm selection strategy, giving priority to compression algorithms with low CPU usage (such as LZ4).

[0329] Feedback mechanism: The performance evaluation results are fed back to the compression algorithm selection module to help dynamically select the optimal compression algorithm.

[0330] In this embodiment, during the processing of the data to be processed using the target algorithm, a performance evaluation is performed based on performance parameters obtained from real-time monitoring, and the performance evaluation result is determined. A new target algorithm matching the performance evaluation result is determined, and the new target algorithm is used to process the data to be processed to determine the target data. Thus, when the performance evaluation result indicates that a certain performance parameter exceeds its limit, a new target algorithm that can alleviate this performance pressure can be determined, thereby reducing performance pressure and improving the data read / write performance of the device.

[0331] Please see Figure 10 The following is a flowchart illustrating the data processing method provided in the embodiments of this application. Figure 9 The following will be explained in conjunction with the steps shown:

[0332] S11, Data write request.

[0333] When a data write request enters the system, it triggers the dynamic compression algorithm selection process.

[0334] S12, Data Feature Analysis.

[0335] The characteristics of the written data are analyzed, including data size, data type, entropy value, and distribution characteristics.

[0336] S13. Compression algorithm selection.

[0337] Based on the data feature analysis results, combined with system resources, network conditions, and query load, the optimal compression algorithm is dynamically selected.

[0338] S14, System Resource Monitoring.

[0339] Real-time monitoring of system resources (such as CPU utilization and memory usage) provides a basis for selecting compression algorithms.

[0340] S15. Is the network condition good?

[0341] Real-time monitoring of network conditions (such as bandwidth and latency) provides a basis for selecting compression algorithms.

[0342] S16, Query load monitoring.

[0343] If the network conditions are good in real time, then the query load (such as query latency and query concurrency) can be monitored in real time to provide a basis for the selection of compression algorithms.

[0344] S17, Dynamic compression algorithm execution.

[0345] The data is compressed and stored according to the selected compression algorithm.

[0346] S18. Data storage and retrieval.

[0347] The compressed data is stored in ClickHouse and supports efficient querying. System performance (such as compression ratio and query latency) is monitored in real time, and feedback information is used to optimize the compression algorithm selection strategy.

[0348] This application discloses a dynamic compression algorithm selection method and system for ClickHouse, used to optimize CPU usage and query performance during peak data write periods. This application is not specifically effective for logistics navigation data, but has universal applicability. It is important to emphasize that the following provides methods for processing various data types (including data types consistent with navigation).

[0349] This invention aims to solve the performance bottleneck problem faced by ClickHouse during peak data write periods (such as 8 AM), particularly the high CPU usage and decreased query performance caused by the static compression algorithm selection mechanism. Specifically, this invention proposes solutions to the following technical problems:

[0350] 1. Dynamically adapt to resource pressure during peak data write periods.

[0351] Problem Description: During peak data write periods, the CPU utilization of the ClickHouse cluster increases significantly, leading to a decrease in query performance. In existing technologies, static compression algorithms cannot dynamically adjust based on real-time resource usage. This results in high-compression-ratio algorithms (such as Zstandard) potentially consuming excessive CPU resources during peak periods, while high-speed compression algorithms (such as LZ4) may not fully utilize storage space.

[0352] Solution: This invention introduces a dynamic compression algorithm selection mechanism, which dynamically adjusts the compression algorithm based on real-time CPU utilization, data write rate, and query load. During peak periods, compression algorithms with low CPU utilization (such as LZ4) are prioritized to reduce computational overhead; during off-peak periods, algorithms with high compression ratios (such as Zstandard) are prioritized to optimize storage space.

[0353] 2. Optimize resource allocation.

[0354] Problem Description: Static compression algorithms cannot dynamically adjust based on data characteristics and system load, leading to uneven resource allocation. For example, when CPU resources are scarce, high compression ratio algorithms may result in excessive computational overhead; while when CPU resources are plentiful, high-speed compression algorithms may not fully utilize system resources.

[0355] Solution: This invention achieves efficient resource allocation by dynamically selecting the optimal compression algorithm based on real-time perception of system resources (such as CPU utilization and memory usage) and data characteristics (such as entropy and distribution characteristics).

[0356] 3. Improve query performance.

[0357] Problem Description: During peak data write periods, static compression algorithms may excessively consume CPU resources, thus impacting query performance. For example, high compression ratio algorithms may increase query latency or even cause query timeouts during peak periods.

[0358] Solution: This invention improves query performance by dynamically adjusting the compression algorithm to reduce CPU usage during peak data write periods. Simultaneously, it further reduces query latency by dynamically optimizing the compression strategy based on real-time query load monitoring.

[0359] 4. Achieve a balance between storage efficiency and computing efficiency.

[0360] Problem Description: Static compression algorithms cannot dynamically adjust based on storage requirements and the real-time status of computing resources, resulting in an insufficient trade-off between storage efficiency and computational efficiency. For example, when storage space is limited, high compression ratio algorithms may not be the preferred choice; conversely, when computing resources are limited, high-speed compression algorithms may not be the preferred choice.

[0361] Solution: This invention achieves the optimal balance between storage efficiency and computing efficiency by dynamically sensing storage needs and computing resource status and adjusting compression algorithms and parameters in real time.

[0362] The main objective of this invention is:

[0363] 1. Implement dynamic compression algorithm selection in ClickHouse to adapt to resource pressure during peak data writing periods.

[0364] 2. While ensuring data writing efficiency, reduce CPU usage and improve query performance.

[0365] 3. By dynamically adjusting the compression algorithm and parameters, the optimal balance between storage efficiency and computational efficiency is achieved.

[0366] Correspondence with the shortcomings of existing technologies.

[0367] 1. Dynamically adapt to resource pressure: Solves the problem that static compression algorithms cannot adapt to resource pressure during peak data writing periods.

[0368] 2. Optimize resource allocation: Solve the problem of uneven resource allocation caused by static compression algorithms.

[0369] 3. Improve query performance: Solve the problem of decreased query performance caused by static compression algorithms.

[0370] 4. Achieve a balance between storage and computation efficiency: Solve the problem that static compression algorithms cannot balance storage efficiency and computation efficiency.

[0371] This application aims to address the performance bottleneck problem faced by ClickHouse during peak data write periods (such as 8 AM), particularly the high CPU usage and decreased query performance caused by the static compression algorithm selection mechanism. The following are the technical innovations and principles that this invention seeks to protect:

[0372] 1. Dynamic compression algorithm selection mechanism.

[0373] Key technical points: Dynamically select the optimal compression algorithm based on real-time data characteristics, system resources, network conditions, and query load.

[0374] Technical principle: By combining rule base matching and machine learning model prediction, and considering data features (such as entropy, mean, variance), system resources (such as CPU utilization, memory utilization), network conditions (such as bandwidth, latency), and query load (such as query latency, query concurrency), the optimal compression algorithm (such as Zstandard, LZ4, Snappy, etc.) is dynamically selected.

[0375] During peak data writing periods, prioritize compression algorithms with low CPU usage (such as LZ4); during off-peak periods, prioritize algorithms with high compression ratios (such as Zstandard).

[0376] 2. Real-time monitoring of system resources and network conditions.

[0377] Key technical points: Real-time monitoring of system resources (such as CPU utilization and memory usage) and network conditions (such as bandwidth and latency) provides a basis for selecting compression algorithms.

[0378] Technical principle: System monitoring tools (such as Prometheus and Grafana) and network monitoring tools (such as NetFlow and sFlow) are used to collect system resource and network condition data in real time.

[0379] Based on preset thresholds, the system resources and network pressure are assessed, and the assessment results are fed back to the compression algorithm selection module.

[0380] 3. Real-time monitoring and optimization of query load.

[0381] Key technical points: Real-time monitoring of query load (such as query latency and query concurrency) provides a basis for selecting compression algorithms.

[0382] Technical principle: Query load data is collected in real time through ClickHouse's query logs and system monitoring tools. Based on preset thresholds, query pressure is assessed, and the assessment results are fed back to the compression algorithm selection module.

[0383] 4. Performance feedback and optimization loop.

[0384] Key technical points: Real-time monitoring of system performance (such as compression ratio, query latency, and CPU utilization), and optimization of compression algorithm selection strategies.

[0385] Technical principle: The compression algorithm selection strategy is dynamically adjusted through performance monitoring and evaluation. The performance evaluation results are fed back to the compression algorithm selection module, forming a performance optimization feedback loop.

[0386] 5. Compression parameters are dynamically adjusted.

[0387] Key technical point: Dynamically adjust compression parameters (such as compression level and data block size) based on system resources and network conditions.

[0388] Technical principle: When CPU resources are sufficient, a higher compression level is used to increase the compression ratio; when CPU resources are scarce, a lower compression level is used to reduce computational overhead.

[0389] In high-bandwidth, low-latency environments, larger data blocks are used to improve compression efficiency; in low-bandwidth, high-latency environments, smaller data blocks are used to reduce transmission time.

[0390] Please see Figure 11 This is a schematic diagram of the structure of the data processing device provided in the embodiments of this application.

[0391] This application also provides a data processing device 800, including: a data acquisition unit 801, a determination unit 802, and a data processing unit 803.

[0392] Data acquisition unit 801 is used to acquire data to be processed;

[0393] The determining unit 802 is used to dynamically determine the target algorithm corresponding to the data to be processed based on the multidimensional data features of the data to be processed and the real-time performance parameters.

[0394] The data processing unit 803 is used to process the data to be processed based on the target algorithm to determine the target data.

[0395] In this embodiment of the application, the determining unit 802 in the data processing device 800 is used to extract features from the preprocessed data to be processed and determine the multidimensional data features; wherein, the preprocessing is used to characterize the data cleaning and formatting processing performed on the data to be processed;

[0396] Real-time monitoring of local performance metrics to determine the real-time performance parameters; wherein the real-time performance parameters include at least one of the following: system resource performance parameters, network condition performance parameters, and query load performance parameters;

[0397] The target algorithm for the data to be processed is dynamically determined based on the multidimensional data features, the real-time performance parameters, and the algorithm library; wherein, the algorithm library includes: multiple data compression algorithms, and data processing feature labels corresponding to each data compression algorithm.

[0398] In this embodiment of the application, the determining unit 802 in the data processing device 800 is used to dynamically determine the target algorithm that matches the multidimensional data features and whose corresponding data processing feature label matches the real-time performance parameters based on the multidimensional data features, the real-time performance parameters, and the algorithm library.

[0399] In this embodiment of the application, the determining unit 802 in the data processing device 800 is used to analyze the multidimensional data features according to preset data feature rules and determine the first feature label corresponding to the multidimensional data features; wherein, the preset data feature rules are used to limit the correspondence between the data features and the corresponding demand processing feature labels;

[0400] Based on the real-time performance parameters, a second feature label corresponding to the real-time performance parameters is dynamically determined according to preset performance parameter rules; wherein, the preset performance parameter rules are used to limit the correspondence between performance parameters and corresponding demand processing feature labels;

[0401] The target data processing feature label is determined in the algorithm library that matches at least one of the first feature label and the second feature label; wherein the determination of the target data processing feature label takes into account the performance pressure represented by the performance parameter.

[0402] Determine the target algorithm corresponding to the target data processing feature label.

[0403] In this embodiment of the application, the determining unit 802 in the data processing device 800 is used to input the multidimensional data features and the real-time performance parameters into a preset model to dynamically determine the target algorithm; wherein, the preset model is determined based on historical algorithms that process historical data in the algorithm library and the data processing feature labels corresponding to each historical algorithm; wherein, the process of determining the target algorithm by the preset model takes into account the performance pressure represented by the real-time performance parameters.

[0404] In this embodiment of the application, the multidimensional data features include at least one of the following: data type, data size, statistical features, distribution features, and correlation features.

[0405] In this embodiment of the application, the determining unit 802 in the data processing device 800 is used to dynamically adjust the algorithm parameters of the target algorithm based on the performance parameters obtained by real-time monitoring during the process of processing the data to be processed using the target algorithm.

[0406] In this embodiment of the application, the determining unit 802 in the data processing device 800 is used to store the target data in a storage space that meets the query requirements based on the multidimensional data features and the query requirements for the data to be processed.

[0407] In this embodiment of the application, the determining unit 802 in the data processing device 800 is used to perform performance evaluation based on performance parameters obtained by real-time monitoring and determine the performance evaluation result during the process of processing the data to be processed using the target algorithm.

[0408] A new target algorithm matching the performance evaluation results is determined, and the new target algorithm is used to process the data to be processed to determine the target data.

[0409] It should be noted that, in the embodiments of this application, if the above-described data processing method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a data processing device (which may be a personal computer, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0410] Correspondingly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the method on one side of the data processing apparatus 800.

[0411] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0412] It should be noted that, Figure 12 A schematic diagram of a hardware entity of an electronic device provided in an embodiment of this application, such as... Figure 12 As shown, this application embodiment provides an electronic device 900, including a memory 902 and a processor 901. The memory 902 stores a computer program that can run on the processor 901. When the processor 901 executes the program, it implements the steps in the above-described method, wherein;

[0413] The processor 901 typically controls the overall operation of the electronic device 900.

[0414] The memory 902 is configured to store instructions and applications executable by the processor 901, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data and video communication data) in the processor 901 and various modules in the electronic device 900. It can be implemented by flash memory or random access memory (RAM).

[0415] Correspondingly, this application also provides a computer program product, including a computer program that can be executed by the processor 901 of the electronic device 900 to complete the steps in the method on one side of the data processing device 800.

[0416] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0417] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0418] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the apparatus or units can be electrical, mechanical, or other forms.

[0419] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0420] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0421] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0422] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0423] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, include: Obtain the data to be processed; The target algorithm corresponding to the data to be processed is dynamically determined based on the multidimensional data characteristics of the data to be processed and the real-time performance parameters. The target data is determined by processing the data to be processed based on the target algorithm.

2. The data processing method according to claim 1, characterized in that, The target algorithm for dynamically determining the data to be processed based on the multidimensional data features and real-time performance parameters includes: Feature extraction is performed on the preprocessed data to be processed to determine the multidimensional data features; wherein, the preprocessing is used to characterize the data cleaning and formatting processing performed on the data to be processed; Real-time monitoring of local performance metrics to determine the real-time performance parameters; wherein the real-time performance parameters include at least one of the following: system resource performance parameters, network condition performance parameters, and query load performance parameters; The target algorithm for the data to be processed is dynamically determined based on the multidimensional data features, the real-time performance parameters, and the algorithm library; wherein, the algorithm library includes: multiple data compression algorithms, and data processing feature labels corresponding to each data compression algorithm.

3. The data processing method according to claim 2, characterized in that, The target algorithm for dynamically determining the data to be processed based on the multidimensional data features, the real-time performance parameters, and the algorithm library includes: Based on the multidimensional data features, the real-time performance parameters, and the algorithm library, the target algorithm is dynamically determined that matches the multidimensional data features and whose corresponding data processing feature label matches the real-time performance parameters.

4. The data processing method according to claim 3, characterized in that, The target algorithm, which dynamically determines the matching data processing feature label and the real-time performance parameter based on the multidimensional data features, the performance parameters, and the algorithm library, includes: Based on the multidimensional data features, a first feature label corresponding to the multidimensional data features is determined according to a preset data feature rule; wherein, the preset data feature rule is used to limit the correspondence between the data features and the corresponding requirement processing feature labels; Based on the real-time performance parameters, a second feature label corresponding to the real-time performance parameters is dynamically determined according to preset performance parameter rules; wherein, the preset performance parameter rules are used to limit the correspondence between performance parameters and corresponding demand processing feature labels; In the algorithm library, a target data processing feature label is determined that matches at least one of the first feature label and the second feature label; wherein the determination of the target data processing feature label takes into account the performance pressure represented by the performance parameter. Determine the target algorithm corresponding to the target data processing feature label.

5. The data processing method according to claim 3, characterized in that, The target algorithm, which dynamically determines the data processing feature label that matches the multidimensional data features and the corresponding data processing feature label that matches the real-time performance parameters based on the multidimensional data features, the real-time performance parameters, and the algorithm library, includes: The multidimensional data features and the real-time performance parameters are input into a preset model to dynamically determine the target algorithm; wherein, the preset model is determined based on historical algorithms that process historical data in the algorithm library and the data processing feature labels corresponding to each historical algorithm; wherein, the process of determining the target algorithm by the preset model takes into account the performance pressure represented by the real-time performance parameters.

6. The data processing method according to any one of claims 1 to 5, characterized in that, The multidimensional data features include at least one of the following: data type, data size, statistical features, distribution features, and correlation features.

7. The data processing method according to any one of claims 1 to 5, characterized in that, The method further includes: During the process of processing the data to be processed using the target algorithm, the algorithm parameters of the target algorithm are dynamically adjusted based on the performance parameters obtained from real-time monitoring.

8. The data processing method according to any one of claims 1 to 5, characterized in that, The method further includes: Based on the multidimensional data characteristics and the query requirements for the data to be processed, the target data is stored in a storage space that meets the query requirements.

9. The data processing method according to any one of claims 1 to 5, characterized in that, The method further includes: During the process of processing the data to be processed using the target algorithm, a performance evaluation is performed based on the performance parameters obtained from real-time monitoring, and the performance evaluation result is determined. A new target algorithm matching the performance evaluation results is determined, and the new target algorithm is used to process the data to be processed to determine the target data.

10. A data processing apparatus, characterized in that, include: The data acquisition unit is used to acquire data to be processed. The determining unit is used to dynamically determine the target algorithm corresponding to the data to be processed based on the multidimensional data features of the data to be processed and the real-time performance parameters. A data processing unit is used to process the data to be processed based on the target algorithm to determine the target data.

11. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program that can run on the processor, the processor executing the computer program to implement the steps of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

Images