Data compression method and device, storage medium and program product

Abstract

The invention provides a data compression method and device, a storage medium and a program product, and the method comprises the steps: obtaining a plurality of data blocks from a plurality of data sources; determining compression information of each data block according to the data source type of each data block; determining a target compression strategy corresponding to each data block from at least one selectable compression strategy of each data block according to the compression parameters under each selectable compression strategy of each data block; and based on the target compression strategy of each data block, compressing each data block to obtain a compression result of each data block. The data from different data sources can be subjected to cross-modal differential compression, key information loss or structure damage caused by unified compression is avoided, the internal structure and key elements of each data block are protected, semantic breakage caused by universal processing is avoided, the availability and semantic fidelity of a compression result are improved, and the data compression efficiency is improved. And meanwhile, the method has good expandability and is easy to adapt to new data types and business scenes.

Description

Technical Field

[0001] This application relates to the field of data compression technology, and more specifically, to a data compression method, apparatus, storage medium, and program product. Background Technology

[0002] With the widespread application of Large Language Models (LLMs) in complex tasks, prompting engineering has become a crucial bridge connecting user intent and model capabilities. Especially in high-level application scenarios such as Retrieval-Augmented Generation (RAG) and multi-tool collaborative invocation, it is necessary to integrate heterogeneous information from multiple sources, including Application Programming Interface (API) responses, database queries, knowledge base documents, time-series data streams, and structured reports, as input prompts. These complex prompts often contain various data formats such as text paragraphs, tabular data, code snippets, and time series, leading to a rapid expansion of prompt length that easily exceeds the limited context window of the LLM. Therefore, how to efficiently compress multi-source heterogeneous data and adapt to model context constraints while ensuring semantic integrity and task performance has become one of the core bottlenecks restricting the large-scale deployment of large models.

[0003] Existing cue compression techniques mainly fall into two categories: soft cue compression and hard cue compression. Soft cue compression methods map natural language cues into low-dimensional continuous vector representations using an encoder to achieve compression. Hard cue compression, on the other hand, directly filters or rewrites the original text, preserving the natural language form to improve compatibility. Additionally, existing techniques also include cue compression methods based on reinforcement learning, which optimize preservation strategies by designing reward functions.

[0004] However, existing cue compression methods generally employ a uniform token-level or phrase-level compression strategy for multi-source heterogeneous data during the compression process. This can easily disrupt row and column relationships, omit key values, or damage the syntactic structure. Furthermore, they cannot intelligently determine the importance of each data block, often resulting in the accidental deletion of high-value information while inefficient redundancy is retained. Consequently, the usability and semantic fidelity of the compression results are poor. Summary of the Invention

[0005] The purpose of this application is to provide a data compression method, device, storage medium, and program product to address the shortcomings of the prior art, thereby solving the problems of poor usability and semantic fidelity in the prior art.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, one embodiment of this application provides a data compression method, the method comprising: Retrieve multiple data blocks from multiple data sources; Based on the data source type of each data block, the compression information of each data block is determined, and the compression information includes: at least one optional compression strategy for the data block and compression parameters under each optional compression strategy; Based on the compression parameters under each optional compression strategy of each data block, determine the target compression strategy corresponding to each data block from at least one optional compression strategy of each data block; Based on the target compression strategy for each data block, each data block is compressed to obtain the compression result of each data block.

[0007] Secondly, another embodiment of this application provides a data compression apparatus, the apparatus comprising: The acquisition module is used to acquire multiple data blocks from multiple data sources; The first determining module is used to determine the compression information of each data block according to the data source type of each data block. The compression information includes: at least one optional compression strategy of the data block and compression parameters under each optional compression strategy. The second determining module is used to determine the target compression strategy corresponding to each data block from at least one optional compression strategy of each data block based on the compression parameters under each optional compression strategy of each data block. The compression module is used to compress each data block based on the target compression strategy of each data block, so as to obtain the compression result of each data block.

[0008] Thirdly, another embodiment of this application provides an electronic device, including: a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of any of the methods described in the first aspect above.

[0009] Fourthly, another embodiment of this application provides a storage medium storing a computer program, which, when executed by a processor, performs the steps of any of the methods described in the first aspect above.

[0010] Fifthly, another embodiment of this application provides a computer program product, the computer program product including computer program code, wherein when the computer program code is executed by a computing device, the computing device performs the steps of any of the methods described in the first aspect above.

[0011] The beneficial effects of this application are as follows: By acquiring multiple data blocks from multiple data sources, determining the compression information of each data block according to its data source type, and determining the target compression strategy for each data block from at least one optional compression strategy based on the compression parameters of each data block's optional compression strategies, the compression results of each data block are obtained by compressing each data block based on its target compression strategy. This enables cross-modal differentiated compression of data from different data sources, avoiding the loss of key information or structural damage caused by uniform compression, protecting the internal structure and key elements of each data block, avoiding semantic breaks caused by generalized processing, and improving the usability and semantic fidelity of the compression results. At the same time, the compression strategy and data source type are loosely coupled, so that when adding a new data source, there is no need to modify the core decision engine, which has good scalability and is easy to adapt to new data types and business scenarios. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram illustrating an application scenario of the data compression method provided in the embodiments of this application; Figure 2 A schematic flowchart of a data compression method provided in an embodiment of this application; Figure 3 This is a flowchart illustrating the process of determining compression information for each data block in the data compression method provided in this application embodiment. Figure 4 This is a schematic flowchart illustrating the process of determining the second compression parameter under each optional compression strategy in the data compression method provided in the embodiments of this application. Figure 5 A schematic flowchart illustrating the process of determining the cost-effectiveness of an optional compression strategy in the data compression method provided in this application embodiment; Figure 6 Another flowchart illustrating the determination of the cost-effectiveness of an optional compression strategy in the data compression method provided in the embodiments of this application; Figure 7 A schematic flowchart illustrating the data filling process in the data compression method provided in this application embodiment; Figure 8 A flowchart illustrating the process of determining the values ​​at each position in the cost-effectiveness array in the data compression method provided in this application embodiment; Figure 9 This is a flowchart illustrating the process of determining the cost-effectiveness of alternative compression strategies in the data compression method provided in this application embodiment. Figure 10 This is a flowchart illustrating the process of determining the target compression strategy for each data block in the data compression method provided in this application embodiment. Figure 11 This is a schematic diagram of the electronic device structure provided in an embodiment of this application. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0015] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0016] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0017] Existing cue compression techniques mainly fall into two categories: soft cue compression and hard cue compression. Soft cue compression methods map natural language cues into low-dimensional continuous vector representations using an encoder to achieve compression. Hard cue compression, on the other hand, directly filters or rewrites the original text, preserving the natural language form to improve compatibility. Additionally, existing techniques also include cue compression methods based on reinforcement learning, which optimize preservation strategies by designing reward functions.

[0018] However, existing cue compression methods generally adopt a uniform token-level or phrase-level compression strategy for multi-source heterogeneous data during the compression process. This can easily disrupt row and column relationships, omit key values, or damage the syntactic structure. Furthermore, the compression priority decision is static and fixed, making it impossible to intelligently determine the importance of each data block. This often results in the accidental deletion of high-value information while inefficient redundancy is retained, leading to problems with poor usability and semantic fidelity.

[0019] Based on the aforementioned problems, this application proposes a data compression method. This method acquires multiple data blocks from multiple data sources, determines the compression information of each data block according to its data source type, and determines the target compression strategy for each data block from at least one optional compression strategy based on the compression parameters of each data block's optional compression strategies. Then, based on the target compression strategy, each data block is compressed to obtain its compression result. This method enables cross-modal differentiated compression of data from different data sources, avoiding the loss of key information or structural damage caused by uniform compression. It improves the usability and semantic fidelity of the compression result, while also possessing good scalability and easy adaptation to new data types and business scenarios.

[0020] It is understood that the data compression method provided in this application embodiment can be applied to any scenario that requires compression of multi-source heterogeneous data, such as the scenario of compressing prompt words of a large model.

[0021] For example, Figure 1 This is a schematic diagram illustrating an application scenario of the data compression method provided in this application embodiment, with reference to... Figure 1 As shown, taking the data compression method provided in this application embodiment as an example in the scenario of compressing prompt words for a large model, the electronic device can first find multiple data blocks from multiple data sources according to the user's input instructions, and execute the data compression method provided in this application embodiment to compress the multiple data blocks from multiple data sources to obtain the compression result of each data block. The compression result of each data block is then used as a prompt word, and the prompt word is input into the large model. The large model performs inference based on the prompt word to obtain the inference result, and then returns the inference result to the user.

[0022] It is worth noting that, in order to improve the compression effect, the data compression method provided in this application embodiment can be repeatedly executed multiple times in practical applications. Furthermore, during the repeated execution of the data compression method provided in this application embodiment, the compression result of the previous execution can be used as multiple data blocks for the next compression iteration. This application embodiment will be described using the execution of one compression as an example.

[0023] The data compression method provided in this application will be described in detail below with reference to several embodiments.

[0024] Figure 2 This is a schematic flowchart of a data compression method provided in an embodiment of this application, referring to... Figure 2 As shown, the subject executing this method can be any electronic device with processing capabilities, and the method includes: S201. Obtain multiple data blocks from multiple data sources.

[0025] Optionally, multiple data blocks from multiple data sources can be retrieved.

[0026] Multiple data sources can be different data sources, and multiple data blocks can include, for example, text data blocks, time-series data blocks, code data blocks, table data blocks, etc.

[0027] For example, a data block can be a piece of JSON text returned by an API, a database query result table, or a piece of code.

[0028] S202. Determine the compression information for each data block based on the data source type of each data block.

[0029] The compression information includes: at least one optional compression strategy for the data block and compression parameters under each optional compression strategy.

[0030] Specifically, an optional compression strategy refers to a compression method that can be applied to the data block, and the compression parameters under the optional compression strategy refer to the compression-related performance indicators under that optional compression strategy.

[0031] For example, compression parameters under an optional compression strategy may include: time cost, quality coefficient, compression priority, compressible length, and cost-effectiveness under the optional compression strategy.

[0032] In one example, each data block can be traversed, and for the current data block being traversed, at least one optional compression strategy for the data block can be determined based on the data source type of the data block, and the compression parameters under each optional compression strategy can be calculated.

[0033] In another example, each data block and its data source type can be input into a pre-trained policy generation model. The policy generation model then infers and generates at least one optional compression policy for each data block, along with compression parameters for each optional compression policy. The policy generation model can be a Transformer-based sequence-to-sequence (Seq2Seq) model.

[0034] For example, taking text data blocks as an example, optional compression strategies may include: text relevance filtering, text summarization based on large models, and token classification models.

[0035] Among them, text relevance screening refers to the compression of the volume of text data, which can be accomplished through a three-stage pipeline. First, in the recall stage, vector retrieval or keyword indexing is used to quickly screen candidate documents, greatly narrowing the processing scope. Then, in the re-ranking stage, a lightweight model is used to score the relevance of candidate results. Finally, in the adoption stage, only the large model is called to conduct a final review of the top 10% of high-quality content, balancing effectiveness and efficiency.

[0036] Among them, text summarization based on large models refers to dynamically adjusting the processing method according to the text length. Short texts directly call the large model to generate summaries in real time, while long documents are processed in blocks using a sliding window and then the key information is integrated.

[0037] The Token classification model refers to hierarchical processing based on semantic importance. Specifically, core entities such as personal names and place names are kept as is, high-frequency functional words are replaced with abbreviations, and modifying components are deleted with differentiated probabilities based on the depth of the syntax tree.

[0038] For example, taking time-series data blocks as an example, optional compression strategies may include: data frequency conversion. Data frequency conversion refers to reducing the frequency of data compression calculations based on the release frequency in the time dimension, performing peak sampling on special data, and retaining only key data points.

[0039] S203. Based on the compression parameters under each optional compression strategy of each data block, determine the target compression strategy corresponding to each data block from at least one optional compression strategy of each data block.

[0040] In one example, the optional compression strategies for each data block can be combined to obtain multiple combination schemes. The compression performance index corresponding to each combination scheme is calculated according to the compression parameters under each optional compression strategy. Then, the compression strategy corresponding to each data block in the combination scheme with the best compression performance index is taken as the target compression strategy for each data block.

[0041] The compression performance indicators corresponding to the combined scheme may include one or more of the following: overall cost-effectiveness, overall quality coefficient, overall time cost, overall compression length, and overall priority score.

[0042] In another example, at least one optional compression strategy for each data block can be filtered based on the compression parameters under each optional compression strategy and the preset expected compression index to obtain the target compression strategy corresponding to each data block.

[0043] The expected compression metric can be the expected compression metric for each data block. For example, the expected compression metric can include the time compression threshold of the data block. When the time cost under a certain optional compression strategy exceeds the time compression threshold, the optional compression strategy can be eliminated.

[0044] S204. Based on the target compression strategy for each data block, compress each data block to obtain the compression result of each data block.

[0045] Optionally, after obtaining the target compression strategy for each data block, each data block can be compressed separately using the target compression strategy to obtain the compression result of each data block.

[0046] In this embodiment, multiple data blocks from multiple data sources are acquired, and the compression information of each data block is determined according to the data source type of each data block. Based on the compression parameters under each optional compression strategy of each data block, the target compression strategy corresponding to each data block is determined from at least one optional compression strategy of each data block. Thus, each data block is compressed based on the target compression strategy of each data block to obtain the compression result of each data block. This enables cross-modal differentiated compression of data from different data sources, avoiding the loss of key information or structural damage caused by uniform compression, protecting the internal structure and key elements of each data block, avoiding semantic breaks caused by generalized processing, and improving the usability and semantic fidelity of the compression result. At the same time, the compression strategy and the data source type are loosely coupled, so that when adding a new data source, there is no need to modify the core decision engine, which has good scalability and is easy to adapt to new data types and business scenarios.

[0047] In one possible implementation, the process of determining the compression information of each data block is illustrated by example. Figure 3 This is a flowchart illustrating the process of determining compression information for each data block in the data compression method provided in this application embodiment, with reference to... Figure 3 As shown, S202 above includes: S301. Based on the data source type of the data block, obtain at least one optional compression strategy corresponding to the data block and the first compression parameter corresponding to each optional compression strategy.

[0048] Optionally, with the first Taking a data block as an example, it can be based on the data block Data source type, retrieve data blocks At least one corresponding optional compression strategy and various optional compression strategies The corresponding first compression parameter.

[0049] The first compression parameter is used to indicate the data block. In this optional compression strategy The basic performance indicators, the first compression parameter includes: compression ratio, relevance, and time cost.

[0050] Specifically, compression ratio refers to the expected size of the data block after applying the optional compression strategy; it measures the space efficiency during compression. Relevance refers to the semantic connection or importance between the retained content and the uncompressed data block or the final task objective after applying the optional compression strategy; it measures the loss or semantic fidelity during compression. Time cost refers to the computation time or resources required to apply the optional compression strategy; it measures the processing efficiency or latency during compression.

[0051] Optionally, the first compression parameter may also include reproducibility and compatibility. Specifically, reproducibility refers to whether applying the same optional compression strategy to the same input data always yields the exact same result. Compatibility refers to the degree to which the optional compression strategy depends on a specific external system or model; compatibility is used to measure the cost, reliability, and deployment flexibility during compression.

[0052] In one example, it can be based on data blocks. Data source type Data blocks are matched from a pre-built pool of compression methods. At least one corresponding optional compression strategy and various optional compression strategies The corresponding first compression parameter. The compression method pool stores each data source type and at least one corresponding optional compression strategy. and various optional compression strategies The corresponding first compression parameter.

[0053] For example, the compression method pool can be referenced as shown in Table 1 below:

[0054] In this table, A1, A2, A3, A4, and A5 are pre-determined compression ratio values ​​for each data type. These compression ratio values ​​may be the same or different for each data type, and their values ​​range from 0 to 1. B1, B2, B3, B4, and B5 are pre-determined relevance values ​​for each data type. These relevance values ​​may be the same or different for each data type, and their values ​​range from 0 to 1. C1, C2, C3, C4, and C5 are pre-determined estimated time cost values ​​for each data type. These estimated time cost values ​​may be the same or different for each data type. This is the input length of the model, i.e., the expected length of the compressed result. This refers to the data block length. When the compression method pool stores a range of compression ratios, the compression ratio of an optional compression strategy can be obtained by taking a value from that range. For example, a random value or the median value within that range can be used.

[0055] For example, in this implementation, the compression ratio, relevance, and time cost can all be obtained through pre-calculation or pre-configuration.

[0056] In another example, the data source type of the data block can be input into a pre-trained policy parameter generation model, which then performs inference to generate the data block. At least one optional compression strategy and various optional compression strategies The corresponding first compression parameter.

[0057] S302. Determine the second compression parameter under each optional compression strategy based on the first compression parameter corresponding to each optional compression strategy and the data block length of the data block.

[0058] Optionally, after obtaining each optional compression strategy After obtaining the corresponding first compression parameter, various optional compression strategies can be used. The corresponding first compression parameters and data blocks Data block length The second compression parameters under each optional compression strategy are calculated, and each optional compression strategy is then... and various optional compression strategies Add the second compression parameter below to the compression information.

[0059] For example, it can be based on an optional compression strategy Corresponding compression ratio and data blocks Data block length The optional compression strategies are calculated. The compressible length below And based on the relevance of the optional compression strategy and the data blocks Data block length The optional compression strategies are calculated. The quality coefficient below And according to the optional compression strategy Corresponding relevance, time cost, compressible length and data blocks Data block length The data block was calculated. In optional compression strategies The cost performance .

[0060] The second compression parameter includes: compressible length. quality coefficient and cost-effectiveness .

[0061] Specifically, compressible length refers to the amount of space saved after compression using optional compression strategies. Quality coefficient refers to the information loss or semantic fidelity of the compressed data block compared to its uncompressed state after compression using optional compression strategies. Cost-effectiveness refers to the overall score after compression using optional compression strategies; this overall score can be obtained by comprehensively considering dimensions such as space saving, processing speed, and information quality.

[0062] By identifying the data source type of the data block, at least one optional compression strategy and the first compression parameter corresponding to each optional compression strategy can be obtained. This allows for the determination of the original performance of compression strategies from different data sources using different principles. Furthermore, based on the first compression parameter corresponding to each optional compression strategy and the data block length, the second compression parameter under each optional compression strategy can be determined. This enables the comprehensive analysis of the specific circumstances of the data block, transforming the original performance of the compression strategy into a unified and quantifiable evaluation metric. Consequently, it allows for the precise calculation of how to select the optimal compression strategy for each data block, achieving an overall balance between space, time, and quality, and ultimately, the best compression effect.

[0063] In one possible implementation, the process of determining the second compression parameter under each optional compression strategy based on the first compression parameter corresponding to each optional compression strategy and the data block length of the data block is illustrated by an example. Figure 4 This is a schematic flowchart illustrating the process of determining the second compression parameter under each optional compression strategy in the data compression method provided in the embodiments of this application, with reference to... Figure 4 As shown, S302 above includes: S401. Determine the compressible length of the optional compression strategy based on the data block length and compression ratio.

[0064] For example, data blocks can be computed. Data block length The product of the compression ratio and the compression strategy yields the available compression strategies. Compressible length .

[0065] S402. Determine the quality coefficient of the optional compression strategy based on the relevance.

[0066] In one example, a predefined optional compression strategy can be obtained. authority And calculate the authority of the optional compression strategy. Relevance The product of these factors yields the quality coefficients of the optional compression strategy. .

[0067] In another example, relevance can be... Inputting the pre-trained quality prediction model yields an optional compression strategy. The quality coefficient.

[0068] S403. Determine the cost-effectiveness of the optional compression strategy based on the compressible length of the optional compression strategy.

[0069] Optionally, it can be based on an optional compression strategy. Compressible length The optional compression strategies are calculated. Cost-effectiveness .

[0070] In one example, it can be based on an optional compression strategy. Compressible length The reproducibility and compatibility of the optional compression strategies were considered, and the optional compression strategies were calculated. Cost-effectiveness .

[0071] In another example, an optional compression strategy can be used. Compressible length Inputting the pre-trained cost-effectiveness prediction model yields an optional compression strategy. Cost-effectiveness .

[0072] In one possible implementation, an illustrative description is provided of the process for determining the cost-effectiveness of an optional compression strategy. Figure 5 This is a flowchart illustrating the process of determining the cost-effectiveness of selectable compression strategies in the data compression method provided in this application embodiment, with reference to... Figure 5 As shown, S403 above includes: S501. Calculate the length cost-effectiveness of the optional compression strategy based on the compressible length of the optional compression strategy, the data block length, and the preset length weight.

[0073] Optionally, it can be based on an optional compression strategy. The compressible length below Data blocks Data block length and preset length weights The optional compression strategies are calculated. The length and cost-effectiveness.

[0074] Among them, optional compression strategies Length, cost-effectiveness, and optional compression strategies The compressible length below Proportional.

[0075] In one example, it can be based on an optional compression strategy. The compressible length below Data blocks Data block length The length factor is calculated. And calculate the length factor. With preset length weight The product of these factors yields the optional compression strategies. The length and cost-effectiveness.

[0076] In another example, an optional compression strategy can be computed. The compressible length below Data blocks Data block length and preset length weights The product of these factors yields the optional compression strategies. The length and cost-effectiveness.

[0077] S502. Calculate the time cost-effectiveness of the optional compression strategy based on the time cost of the optional compression strategy and the preset time weight.

[0078] Optionally, it can be based on an optional compression strategy. Time cost and preset time weights The optional compression strategies are calculated. Time-efficient.

[0079] Among them, optional compression strategies Time efficiency and optional compression strategies Time cost Inversely proportional.

[0080] In one example, it can be based on an optional compression strategy. Time cost and the preset standard time cost The time factor was calculated. And calculate the time factor. With preset time weights The product of these factors yields the optional compression strategies. Time-efficient.

[0081] In another example, an optional compression strategy can be computed. Time cost With preset time weights The product of these factors yields the optional compression strategies. Time-efficient.

[0082] S503. Calculate the quality cost-effectiveness of the optional compression strategy based on the quality coefficient of the optional compression strategy and the preset quality weight.

[0083] Optionally, it can be based on an optional compression strategy. quality coefficient and preset quality weights The optional compression strategies are calculated. The quality and cost-effectiveness.

[0084] Among them, optional compression strategies Quality, cost-effectiveness, and optional compression strategies quality coefficient Proportional.

[0085] For example, optional compression strategies can be calculated. quality coefficient and preset quality weights The product of these factors yields the optional compression strategies. The quality and cost-effectiveness.

[0086] S504. Based on the cost-effectiveness of length, time, and quality, the cost-effectiveness of the optional compression strategy is calculated.

[0087] Optionally, after obtaining the length cost-effectiveness, time cost-effectiveness, and quality cost-effectiveness, these factors can be summed to obtain an optional compression strategy. Cost-effectiveness .

[0088] For example, optional compression strategies Cost-effectiveness The following formula can be used to calculate it:

[0089] in, For data blocks In optional compression strategies Cost-effectiveness For data blocks In optional compression strategies The compressible length, For data blocks In optional compression strategies The quality coefficient, For data blocks In optional compression strategies Time cost The preset standard time cost, For the preset length weight, As preset time weights, For the preset quality weights, , , , All of these are hyperparameters.

[0090] For example, in data blocks An optional compression strategy For example, this optional compression strategy The compression parameters are shown in Table 2 below:

[0091] By determining the cost-effectiveness of length, time, and quality, the cost-effectiveness of optional compression strategies can be obtained. After comprehensively considering the three key factors of space saving, processing speed, and information quality, different types and principles of optional compression strategies can be mapped into a quantifiable and comparable comprehensive scoring system. This enables accurate scoring of optional compression strategies, thereby improving the usability and semantic fidelity of compression results.

[0092] The above provides an illustrative explanation of the process for determining the compression information of each data block. It can be understood that after obtaining the compression information of each data block, the target compression strategy corresponding to each data block can be determined from at least one optional compression strategy of each data block based on the compression parameters under each optional compression strategy of each data block. The following provides an illustrative explanation.

[0093] In one possible implementation, Figure 6 This is another flowchart illustrating the determination of the cost-effectiveness of optional compression strategies in the data compression method provided in the embodiments of this application, referring to... Figure 6 As shown, in step S203 above, determining the target compression strategy corresponding to each data block from at least one optional compression strategy for each data block based on the compression parameters under each optional compression strategy for each data block includes: S601. Determine the required compression length for multiple data blocks.

[0094] Optionally, the required compression lengths for multiple data blocks can be determined. The required compression length refers to the total expected compression length of multiple data blocks.

[0095] In one example, taking the scenario of compressing prompts for a large model as an example, the upper limit threshold of the length of prompts for the large model can be obtained, and the required compression length of multiple data blocks can be calculated based on the upper limit threshold of the length of prompts and the total length of multiple data blocks.

[0096] In another example, the required compression lengths for multiple data blocks input by the user can also be obtained directly.

[0097] S602. Initialize the cost-performance array and the strategy record array.

[0098] Optionally, the cost-effectiveness array and the strategy record array can be initialized according to the number of data blocks and the required compression length of multiple data blocks.

[0099] Among them, the cost-effectiveness array is used to store the maximum cost-effectiveness of each data block under different compression lengths, and the strategy record array is used to store the strategy index when the corresponding maximum cost-effectiveness is reached.

[0100] For example, the cost-effectiveness array and the strategy record data can be two-dimensional arrays of the same size. Specifically, the size of the first dimension of the two-dimensional array is the number of data blocks, and the size of the second dimension of the two-dimensional array is the length to be compressed corresponding to multiple data blocks.

[0101] The cost-effectiveness array allows us to trade space for time by storing the optimal solutions for compression strategies for each data block, avoiding redundant calculations. The strategy record array enables the creation of a reverse index from the state value of maximum cost-effectiveness to the compression strategy that generated that state value.

[0102] S603. Based on the compression parameters and required compression length under each optional compression strategy for each data block, fill data into each position of the cost-effectiveness array and each position of the strategy record array.

[0103] Optionally, the compression parameters and the required compression length can be determined based on the optional compression records for each data block. The search algorithm fills data into each position of the cost-effectiveness array and each position of the strategy record array.

[0104] The data filled in the cost-effectiveness array can be the maximum cost-effectiveness of each data block under the current compression length, and the data filled in the strategy record array can be the compression strategy selected when each data block achieves the maximum cost-effectiveness under the current compression length.

[0105] For example, the optional compression strategies of each data block are combined to obtain multiple combination schemes. According to the compression parameters under each optional compression strategy, the maximum cost-effectiveness corresponding to each combination scheme is searched through a search algorithm. The compression strategy selected by each data block to achieve the maximum cost-effectiveness under the current compression length is determined, so as to fill data into each position of the cost-effectiveness array and each position of the strategy record array respectively.

[0106] For example, the search algorithm may include dynamic programming algorithms and greedy algorithms. Dynamic programming algorithms refer to search algorithms that search the optimal solution space.

[0107] S604. Based on the cost-effectiveness array and the strategy record array, determine the target compression strategy corresponding to each data block.

[0108] Optionally, after obtaining the cost-effectiveness array and the strategy record array, the maximum cost-effectiveness of each data block can be found from the cost-effectiveness array, and the strategy selected by each data block when it has the maximum cost-effectiveness can be found from the strategy record data, thereby obtaining the target compression strategy corresponding to each data block.

[0109] By initializing the cost-effectiveness array and the strategy record array, and filling data into each position of the cost-effectiveness array and the strategy record array according to the compression parameters and the required compression length under each optional compression strategy of each data block, the target compression strategy corresponding to each data block can be determined based on the cost-effectiveness array and the strategy record array. This enables a complete mapping from the abstract maximum cost-effectiveness to the specific combined compression scheme. At the same time, it avoids costly secondary calculations and achieves ultimate optimization of space and time.

[0110] In one possible implementation, the process of filling data into each position of the cost-effectiveness array and each position of the strategy record array according to the compression parameters and the required compression length under each optional compression strategy for each data block is illustrated by an example. Figure 7 This is a schematic flowchart illustrating the data filling process in the data compression method provided in this application embodiment, with reference to... Figure 7 As shown, S603 above includes: S701. Using each optional compression strategy, the compression parameters under each optional compression strategy, and the length to be compressed as algorithm parameters, the value at each position in the cost-effectiveness array is determined based on the dynamic optimization algorithm.

[0111] Optionally, the values ​​at each position in the cost-effectiveness array can be determined based on a dynamic optimization algorithm, using each optional compression strategy, the compression parameters under each optional compression strategy, and the length to be compressed as algorithm parameters.

[0112] Specifically, the dynamic optimization algorithm can be a dynamic optimization algorithm based on the grouped knapsack problem.

[0113] For example, the required compression length can be used as the backpack capacity, each data block as an item group, the compressible length of each data block under each optional compression strategy as the weight of each item, and the compression cost-effectiveness of each data block under each optional compression strategy as the value of each item. The goal is to maximize the total cost-effectiveness with the required compression length as the backpack capacity. The total cost-effectiveness can be specifically expressed as the following formula:

[0114] in, Refers to decision variables. Refers to data blocks Use the Several optional compression strategies 0 indicates a data block Do not use optional compression strategies. Refers to data blocks The number of corresponding optional compression strategies, For data blocks Quantity, For data blocks In optional compression strategies The cost-effectiveness.

[0115] For example, when determining the values ​​at each position in the cost-effectiveness array based on a dynamic optimization algorithm using each optional compression strategy, the compression parameters under each optional compression strategy, and the length to be compressed as algorithm parameters, the following constraints also exist:

[0116] The first constraint is that the total compressible length of the selected data blocks using the available compression strategies must be greater than or equal to the required compression length K of the data blocks. The second constraint is that when there are multiple available compression strategies for the same data block, at most one of the available compression strategies can be selected.

[0117] S702. Based on the values ​​at each position in the cost-effectiveness array, determine the strategy to record the values ​​at each position in the array.

[0118] Optionally, when storing the values ​​at each position in the cost-effectiveness array, the values ​​at each position in the strategy record array can be determined and stored according to the index of each position in the cost-effectiveness array.

[0119] For example, when storing the maximum cost-effectiveness of a data block at a certain compression level in the cost-effectiveness array, the corresponding optional compression strategy number can be stored in the strategy record array.

[0120] By using each optional compression strategy, the compression parameters under each optional compression strategy, and the length to be compressed as algorithm parameters, and based on a dynamic optimization algorithm, the values ​​at each position in the cost-effectiveness array are determined. Based on the values ​​at each position in the cost-effectiveness array, the values ​​at each position in the strategy record array are determined. This can decompose the complex global optimization problem into a series of superpositions of local decisions. By recording and comparing in real time, a globally optimal decision chain from the initial state to the final goal is constructed. Furthermore, it can also achieve real-time mapping from the decision to the final combination of compression strategies.

[0121] In one possible implementation, Figure 8 This is a flowchart illustrating the process of determining the values ​​at each position in the cost-effectiveness array in the data compression method provided in this application embodiment, with reference to... Figure 8 As shown, in S701 above, each optional compression strategy, the compression parameters under each optional compression strategy, and the length to be compressed are used as algorithm parameters. Based on a dynamic optimization algorithm, the values ​​at each position in the cost-effectiveness array are determined, including: S801. For each data block, obtain the preset compression margin.

[0122] Optionally, a preset compression margin can be obtained for each data block. The preset compression margin refers to the amount of excess compression allowed to achieve a higher cost-performance ratio; that is, it allows for compression beyond the required length. Additional compressed space.

[0123] S802. Determine the current cumulative compression amount based on the required compression length and compression margin.

[0124] Optionally, depending on the required compression length and compression margin The maximum compression can be calculated. ).

[0125] Optionally, for each data block, the compression amount can be set from 0 to the maximum. ) serves as the numerical range for the cumulative compression of data blocks, and the cumulative compression ranges from 0 to the maximum compression. Iterate through the data and record the current cumulative compression amount. The value is used as the current cumulative compression amount of the data block. .

[0126] Among them, the current cumulative compression amount This refers to the expected compression amount achieved by compressing the current data block and all data blocks preceding it.

[0127] S803. Based on the current cumulative compression amount and the compression parameters of the data block under each optional compression strategy, determine the maximum cost-effectiveness of the data block under the current cumulative compression amount.

[0128] Optionally, the current cumulative compression amount can be... Based on the compression parameters of the data block under each optional compression strategy, the maximum cost-effectiveness of the data block under the current cumulative compression amount is obtained by searching.

[0129] The maximum cost-effectiveness of a data block under the current cumulative compression amount refers to the maximum cost-effectiveness that can be achieved after compressing the current data block and all data blocks before it.

[0130] S804. Store the maximum cost-effectiveness under the current cumulative compression amount to the corresponding position of the data block in the cost-effectiveness array.

[0131] Optionally, the maximum cost-effectiveness under the current cumulative compression amount can be stored in the corresponding position of the data block in the cost-effectiveness array.

[0132] For example, the size of the cost-performance array can be The corresponding position of the data block in the cost-effectiveness array refers to the cost-effectiveness array. Middle characterization data block At the current cumulative compression amount The lower position, that is, .

[0133] Specifically, when data blocks At the current cumulative compression amount The best cost performance Stored in the cost-performance array In position.

[0134] In one possible implementation, S803 determines the maximum cost-effectiveness of the data block at the current cumulative compression level based on the current cumulative compression amount and the compression parameters of the data block under each optional compression strategy, including: Iterate through the available compression strategies corresponding to the data block. For the current available compression strategy, determine the alternative cost-effectiveness of the current available compression strategy based on the compressible length of the current available compression strategy and the current cumulative compression amount. Based on the alternative cost-effectiveness of the current available compression strategy, determine the maximum cost-effectiveness of the data block under the current cumulative compression amount.

[0135] Optionally, the current cumulative compression amount can be... Next, iterate through each of the optional compression strategies corresponding to the data block. For the current optional compression strategy, calculate the alternative cost-effectiveness of the current optional compression strategy based on the compressible length of the current optional compression strategy and the current cumulative compression amount. Based on the alternative cost-effectiveness of the current optional compression strategy, determine the maximum cost-effectiveness of the data block under the current cumulative compression amount.

[0136] For example, the maximum cost-effectiveness of the data block under the current cumulative compression amount can be obtained by comparing the alternative cost-effectiveness of the current available compression strategy with the alternative cost-effectiveness of other available compression strategies.

[0137] In one possible implementation, Figure 9 This is a flowchart illustrating the process of determining the cost-effectiveness of alternative compression strategies in the data compression method provided in this application embodiment, with reference to... Figure 9 As shown, the above steps determine the cost-effectiveness of the current available compression strategies based on the compressible length and the current cumulative compression amount, including: S901. Determine the historical state compression amount based on the compressible length of the currently selectable compression strategy and the current cumulative compression amount.

[0138] Optionally, if the current available compression strategy Compressible length Less than or equal to the current cumulative compression amount Determine the compression amount in historical states Current cumulative compression amount Compressible length compared to current available compression strategies The difference .

[0139] Optionally, if the current available compression strategy Compressible length Greater than the current cumulative compression amount Determine the compression amount in historical states This is the first preset value. The first preset value can be 0.

[0140] Among them, the historical state compression amount refers to the expected compression amount corresponding to the previous data block before the data block.

[0141] In other words, taking any data block as the current data block, to achieve the current cumulative compression amount under the current available compression strategy for the current data block, it is necessary to determine the compression amount that can be achieved after compressing all the data blocks preceding the current data block. In this process, two situations may arise: First, when all data blocks preceding the current data block are compressed, the achieved compression amount does not exceed the expected compression amount for the current data block and all data blocks preceding it; that is, the current selectable compression strategy. Compressible length Less than or equal to the current cumulative compression amount .

[0142] In the first case, determine the compression amount of the historical state. Current cumulative compression amount Compressible length compared to current available compression strategies The difference That is, the expected compression amount corresponding to the previous data block before the current data block. It should be the cumulative compression amount from the current date. Subtract the current available compression strategy Compressible length The amount of compression obtained afterwards.

[0143] Second, when all data blocks preceding the current data block are compressed, the achieved compression exceeds the compression of the current data block and all data blocks preceding it. This exceeds the expected compression level, i.e., the current selectable compression strategy. Compressible length Greater than the current cumulative compression amount .

[0144] In the second case, determine the historical state compression amount. A value of 0 indicates that the compression amount of the currently available compression strategy exceeds the current cumulative compression amount. In this case, only the current cumulative compression amount is considered. The part.

[0145] S902. From the cost-effectiveness array, determine whether the previous data block of the data block is reachable under the historical state compression amount.

[0146] Optionally, after determining the historical state compression amount, the value of the previous data block under the historical state compression amount can be found from the cost-performance array, and based on the value of the previous data block under the historical state compression amount, it can be determined whether the previous data block was in a reachable state under the historical state compression amount.

[0147] Among them, reachability state refers to the existence of a compression scheme in the previous data block of the data block that can satisfy the compression amount of the historical state. Reachability state can be characterized as the existence of the maximum cost-effectiveness value in the corresponding position of the previous data block in the cost-effectiveness array of the current data block at the historical state compression amount.

[0148] For example, in data blocks For example, after determining the compression amount of the historical state... Then, we can start from the cost-performance array Searching for data blocks Previous data block Compression in historical state The value below, that is, determined Does it contain a numerical value?

[0149] S903. If the previous data block of the current data block is reachable under the historical state compression amount, obtain the historical cost-effectiveness of the previous data block of the current data block under the historical state compression amount, and calculate the alternative cost-effectiveness of the current optional compression strategy based on the cost-effectiveness of the current optional compression strategy and the historical cost-effectiveness.

[0150] Optionally, if data block Previous data block Compression in historical state The following is the reachable state. To obtain the historical cost-effectiveness of the previous data block under the historical compression amount, we need to... The value is used as a historical cost-effectiveness indicator.

[0151] Optionally, based on the currently available compression strategy Cost-effectiveness and historical cost-effectiveness The current available compression strategies are calculated. The alternative cost-effectiveness.

[0152] For example, the current available compression strategies can be calculated. Cost-effectiveness and historical cost-effectiveness The sum of these factors yields the currently available compression strategies. Alternative cost-effectiveness .

[0153] Optionally, if the current data block Previous data block Compression in historical state If the current data block is not reachable, then traverse the current data block. The next optional compression strategy.

[0154] By determining the historical state compression amount and checking whether the previous data block of the current data block was reachable under the historical state compression amount from the cost-effectiveness array, and then calculating the alternative cost-effectiveness of the current available compression strategy based on the cost-effectiveness of the current and historical options when it is reachable, feasibility pruning can be achieved. This ensures that the compression strategies for the current and previous data blocks are both feasible and logically correct, thus avoiding redundant calculations and comparisons for invalid branches and significantly improving computational efficiency. Simultaneously, it ensures that the compression strategy for each data block corresponds to a real-world strategy combination, guaranteeing the reliability of the target compression strategy for each data block.

[0155] In one possible implementation, the above steps determine the maximum cost-effectiveness of the data block under the current cumulative compression amount based on the alternative cost-effectiveness of the currently available compression strategies, including: Get the maximum cost-effectiveness of the previous data block under the current cumulative compression amount; if the alternative cost-effectiveness of the available compression strategy is greater than the maximum cost-effectiveness of the previous data block under the current cumulative compression amount, then take the alternative cost-effectiveness of the current available compression strategy as the maximum cost-effectiveness of the data block under the current cumulative compression amount.

[0156] Optionally, the maximum cost-effectiveness of the previous data block under the current cumulative compression amount can be obtained, and the maximum cost-effectiveness of the previous data block under the current cumulative compression amount can be compared with the alternative cost-effectiveness of the currently available compression strategies.

[0157] Optionally, if the current available compression strategy Alternative cost-effectiveness Larger than data block Previous data block At the current cumulative compression amount The maximum cost-effectiveness under the current cumulative compression amount is determined by selecting the alternative cost-effectiveness of the currently available compression strategies.

[0158] Optionally, if the current available compression strategy Alternative cost-effectiveness Smaller than data block Previous data block At the current cumulative compression amount To achieve the best cost-effectiveness, the data blocks... Previous data block At the current cumulative compression amount The maximum cost-effectiveness is used as the data block. At the current cumulative compression amount The best value for money.

[0159] The above provides an illustrative explanation of the process of filling data into each position of the cost-effectiveness array and each position of the strategy record array. It can be understood that after filling data into each position of the cost-effectiveness array and each position of the strategy record array, the cost-effectiveness array and the strategy record array store all feasible combinations of each optional compression strategy for each data block. Therefore, the target compression strategy corresponding to each data block can be determined from the cost-effectiveness array and the strategy record array. The following is an illustrative explanation.

[0160] In one possible implementation, Figure 10 This is a flowchart illustrating the process of determining the target compression strategy for each data block in the data compression method provided in this application embodiment, with reference to... Figure 10 As shown, in S604 above, the target compression strategy corresponding to each data block is determined based on the cost-effectiveness array and the strategy record array, including: S1001. Search the cost-performance array to obtain the target compression amount corresponding to the target data block.

[0161] Optionally, the target compression amount corresponding to the target data block can be searched from the cost-effectiveness array according to the target compression range.

[0162] The target compression range is the length to be compressed. To maximum compression ( The target data block is the last data block, and the target compression amount is the current compression length in the cost-effectiveness array minus the required compression length. To maximum compression ( When ), the compression length corresponding to the maximum cost-effectiveness.

[0163] For example, the last row in the cost-performance array Assume: , , Therefore, the target compression amount can be determined to be 6, and the maximum cost-effectiveness is 18.

[0164] S1002. Based on the target compression amount and strategy record array corresponding to the target data block, backtrack to obtain the target compression strategy corresponding to each data block.

[0165] Optionally, after obtaining the target compression amount corresponding to the target data block, the compression strategy corresponding to the target data block can be determined from the strategy record array according to the target compression amount corresponding to the target data block, and the target compression strategy corresponding to other data blocks can be obtained by backtracking.

[0166] In one possible implementation, step S1002 above, based on the target compression amount corresponding to the target data block and the strategy record array, backtracks to obtain the target compression strategy corresponding to each data block, including: Based on the target compression amount corresponding to the target data block, the target compression strategy corresponding to the target data block is found from the strategy record array; based on the compressible length of the target data block under the target compression strategy, the target compression amount corresponding to the previous data block is determined, and based on the target compression amount corresponding to the previous data block, the target compression strategy corresponding to the previous data block is found from the strategy record array. This process is repeated iteratively until the target compression strategy corresponding to each data block is obtained.

[0167] Optionally, the target compression strategy corresponding to the target data block can be found from the strategy record array according to the target compression amount corresponding to the target data block.

[0168] For example, the target data block is the third data block. Taking a target compression ratio of 6 and a maximum cost-effectiveness ratio of 18 as an example, we can start from the strategy record array. Search in The value is used to obtain the target data block. The corresponding target compression strategy, assuming Then the target data block The corresponding target compression strategy is data blocks. Optional compression strategy 1.

[0169] For example, it is possible to obtain data from the target data block. The target compression amount 6 is subtracted from the target data block. The compressible length of the target compression strategy Obtain the target data block Previous data block The corresponding target compression amount.

[0170] For example, it can be based on the previous data block The corresponding target compression amount is recorded in the strategy record array. In the middle, the previous data block can be found. The corresponding target compression strategy is executed iteratively until the target compression strategy for each data block is obtained.

[0171] By finding the target compression amount corresponding to the previous data block from the strategy record array, the target compression strategy corresponding to the previous data block is obtained. This process is repeated iteratively until the target compression strategy corresponding to each data block is obtained by backtracking. This can reproduce the state transition logic between data blocks during forward computation, but in the opposite direction. This ensures that the backtracking path and the forward computation path are completely consistent in the state space, guaranteeing the determinism and uniqueness of the target compression strategy corresponding to each data block.

[0172] Based on the same inventive concept, this application also provides a data compression device corresponding to the data compression method. Since the principle of the device in this application is similar to the data compression method described above in this application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0173] The device includes: an acquisition module, a first determination module, a second determination module, and a compression module; The acquisition module is used to acquire multiple data blocks from multiple data sources; The first determining module is used to determine the compression information of each data block according to the data source type of each data block. The compression information includes: at least one optional compression strategy for the data block and compression parameters under each optional compression strategy. The second determining module is used to determine the target compression strategy corresponding to each data block from at least one optional compression strategy of each data block based on the compression parameters under each optional compression strategy of each data block. The compression module is used to compress each data block based on the target compression strategy for each data block, and obtain the compression result of each data block.

[0174] Optionally, the first determining module is specifically used for: Based on the data source type of the data block, obtain at least one optional compression strategy corresponding to the data block and the first compression parameter corresponding to each optional compression strategy. The first compression parameter includes: compression ratio, relevance and time cost. Based on the first compression parameters corresponding to each optional compression strategy and the data block length, the second compression parameters under each optional compression strategy are determined. The second compression parameters include: compressible length, quality coefficient, and cost-effectiveness.

[0175] Optionally, the first determining module is specifically used for: Based on the data block length and compression ratio, determine the compressible length of the optional compression strategy; Based on the relevance, determine the quality coefficient of the optional compression strategy; The cost-effectiveness of the optional compression strategy is determined based on the compressible length of the optional compression strategy.

[0176] Optionally, the first determining module is specifically used for: The length cost-effectiveness of the optional compression strategy is calculated based on the compressible length of the optional compression strategy, the data block length, and the preset length weight. The time cost-effectiveness of the optional compression strategy is calculated based on the time cost of the optional compression strategy and the preset time weight. The quality cost-effectiveness of the optional compression strategy is calculated based on the quality coefficient of the optional compression strategy and the preset quality weight. The cost-effectiveness of the available compression strategies is calculated based on the cost-effectiveness of length, time, and quality.

[0177] Optionally, the second determining module is specifically used for: Determine the required compression length for multiple data blocks; Initialize the cost-performance array and the strategy record array. The cost-performance array is used to store the maximum cost-performance of each data block under different compression lengths, and the strategy record array is used to store the strategy index when the corresponding maximum cost-performance is reached. Based on the compression parameters and required compression length under each optional compression strategy for each data block, data is filled into each position of the cost-effectiveness array and each position of the strategy record array. Based on the cost-effectiveness array and the strategy record array, determine the target compression strategy for each data block.

[0178] Optionally, the second determining module is specifically used for: Using each optional compression strategy, the compression parameters under each optional compression strategy, and the length to be compressed as algorithm parameters, the values ​​at each position in the cost-effectiveness array are determined based on a dynamic optimization algorithm. Based on the values ​​at each position in the cost-effectiveness array, determine the strategy to record the values ​​at each position in the array.

[0179] Optionally, the second determining module is specifically used for: For each data block, obtain the preset compression margin; Determine the current cumulative compression amount based on the required compression length and compression margin; Based on the current cumulative compression amount and the compression parameters of the data block under each optional compression strategy, determine the maximum cost-effectiveness of the data block under the current cumulative compression amount; Store the maximum cost-effectiveness under the current cumulative compression amount to the corresponding position of the data block in the cost-effectiveness array.

[0180] Optionally, the second determining module is specifically used for: Iterate through the available compression strategies corresponding to the data block. For the current available compression strategy, determine the alternative cost-effectiveness of the current available compression strategy based on the compressible length of the current available compression strategy and the current cumulative compression amount. Based on the alternative cost-effectiveness of the current available compression strategy, determine the maximum cost-effectiveness of the data block under the current cumulative compression amount.

[0181] Optionally, the second determining module is specifically used for: The historical compression amount is determined based on the compressible length of the currently available compression strategy and the current cumulative compression amount. From the cost-effectiveness array, determine whether the previous data block of the current data block is reachable under the historical state compression amount; If so, obtain the historical cost-effectiveness of the previous data block under the historical state compression amount, and calculate the alternative cost-effectiveness of the current optional compression strategy based on the cost-effectiveness of the current optional compression strategy and the historical cost-effectiveness.

[0182] Optionally, the second determining module is specifically used for: Get the maximum cost-effectiveness of the previous data block of the current data block under the current cumulative compression amount; If the cost-effectiveness of the current optional compression strategy is greater than the maximum cost-effectiveness of the previous data block under the current cumulative compression amount, then the cost-effectiveness of the current optional compression strategy will be taken as the maximum cost-effectiveness of the data block under the current cumulative compression amount.

[0183] Optionally, the second determining module is specifically used for: The target compression amount corresponding to the target data block is obtained by searching the cost-performance array. Based on the target compression amount and strategy record array corresponding to the target data block, the target compression strategy corresponding to each data block is obtained by backtracking.

[0184] Optionally, the second determining module is specifically used for: Based on the target compression amount corresponding to the target data block, the target compression strategy corresponding to the target data block is retrieved from the strategy record array; Based on the compressible length of the target data block in the target compression policy, determine the target compression amount corresponding to the previous data block. Then, based on the target compression amount corresponding to the previous data block, find the target compression policy corresponding to the previous data block from the policy record array. Iterate until the target compression policy corresponding to each data block is obtained.

[0185] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0186] This application also provides an electronic device, such as... Figure 11 As shown, Figure 11 The schematic diagram of the electronic device structure provided in this application embodiment includes: a processor 1101 and a memory 1102, and optionally, a bus 1103. The memory 1102 stores machine-readable instructions executable by the processor 1101. When the electronic device is running, the processor 1101 and the memory 1102 communicate through the bus 1103. When the machine-readable instructions are executed by the processor 1101, the steps of the above-described data compression method are performed.

[0187] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the above-described data compression method.

[0188] This application also provides a computer program product, which includes computer program code. When the computer program code is executed by a computing device, the computing device performs the steps of the above-described data compression method.

[0189] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0190] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0191] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A data compression method, characterized in that, include: Retrieve multiple data blocks from multiple data sources; Based on the data source type of each data block, the compression information of each data block is determined, and the compression information includes: at least one optional compression strategy for the data block and compression parameters under each optional compression strategy; Based on the compression parameters under each optional compression strategy of each data block, determine the target compression strategy corresponding to each data block from at least one optional compression strategy of each data block; Based on the target compression strategy for each data block, each data block is compressed to obtain the compression result of each data block.

2. The data compression method according to claim 1, characterized in that, The step of determining the compression information of each data block based on the data source type of each data block includes: Based on the data source type of the data block, at least one optional compression strategy corresponding to the data block and a first compression parameter corresponding to each optional compression strategy are obtained. The first compression parameter includes: compression ratio, relevance and time cost. Based on the first compression parameters corresponding to each optional compression strategy and the data block length of the data block, the second compression parameters under each optional compression strategy are determined. The second compression parameters include: compressible length, quality coefficient, and cost-effectiveness.

3. The data compression method according to claim 2, characterized in that, The step of determining the second compression parameter under each of the optional compression strategies based on the first compression parameter corresponding to each optional compression strategy and the data block length of the data block includes: The compressible length of the optional compression strategy is determined based on the data block length and the compression ratio. Based on the relevance, determine the quality coefficient of the optional compression strategy; The cost-effectiveness of the optional compression strategy is determined based on the compressible length of the optional compression strategy.

4. The data compression method according to claim 3, characterized in that, Determining the cost-effectiveness of the optional compression strategy based on its compressible length includes: The length cost-effectiveness of the optional compression strategy is calculated based on the compressible length of the optional compression strategy, the data block length of the data block, and the preset length weight. The time cost-effectiveness of the optional compression strategy is calculated based on the time cost of the optional compression strategy and the preset time weight. The quality cost-effectiveness of the optional compression strategy is calculated based on the quality coefficient of the optional compression strategy and the preset quality weight. The cost-effectiveness of the optional compression strategy is calculated based on the length cost-effectiveness, the time cost-effectiveness, and the quality cost-effectiveness.

5. The data compression method according to claim 1, characterized in that, The step of determining the target compression strategy corresponding to each data block from at least one optional compression strategy for each data block based on the compression parameters under each optional compression strategy for each data block includes: Determine the required compression lengths for the plurality of data blocks; Initialize a cost-performance array and a strategy record array. The cost-performance array is used to store the maximum cost-performance of each data block under different compression lengths. The strategy record array is used to store the strategy index when the corresponding maximum cost-performance is achieved. Based on the compression parameters under each optional compression strategy of each data block and the length to be compressed, data is filled into each position of the cost-effectiveness array and each position of the strategy record array, respectively. Based on the cost-effectiveness array and the strategy record array, the target compression strategy corresponding to each data block is determined.

6. The data compression method according to claim 5, characterized in that, The step of filling data into each position of the cost-effectiveness array and each position of the strategy record array according to the compression parameters under each optional compression strategy of each data block and the length to be compressed includes: Using the optional compression strategies, the compression parameters under each optional compression strategy, and the length to be compressed as algorithm parameters, the values ​​at each position in the cost-effectiveness array are determined based on a dynamic optimization algorithm. The values ​​at each position in the strategy record array are determined based on the values ​​at each position in the cost-effectiveness array.

7. The data compression method according to claim 6, characterized in that, The step of determining the values ​​at each position in the cost-effectiveness array based on a dynamic optimization algorithm, using each of the optional compression strategies, the compression parameters under each optional compression strategy, and the length to be compressed as algorithm parameters, includes: For each of the data blocks, obtain the preset compression margin; The current cumulative compression amount is determined based on the required compression length and the compression margin. Based on the current cumulative compression amount and the compression parameters of the data block under each optional compression strategy, determine the maximum cost-effectiveness of the data block under the current cumulative compression amount; Store the maximum cost-effectiveness under the current cumulative compression amount to the corresponding position of the data block in the cost-effectiveness array.

8. The data compression method according to claim 7, characterized in that, The step of determining the maximum cost-effectiveness of the data block under the current cumulative compression amount based on the current cumulative compression amount and the compression parameters of the data block under each selectable compression strategy includes: Iterate through each of the optional compression strategies corresponding to the data block. For the currently selected optional compression strategy, determine the alternative cost-effectiveness of the current optional compression strategy based on the compressible length of the current optional compression strategy and the current cumulative compression amount. Based on the alternative cost-effectiveness of the current optional compression strategy, determine the maximum cost-effectiveness of the data block under the current cumulative compression amount.

9. The data compression method according to claim 8, characterized in that, The step of determining the alternative cost-effectiveness of the currently selectable compression strategy based on the compressible length of the currently selectable compression strategy and the current cumulative compression amount includes: Based on the compressible length of the currently selectable compression strategy and the current cumulative compression amount, determine the historical state compression amount; From the cost-effectiveness array, determine whether the previous data block of the data block is reachable under the historical state compression amount; If so, obtain the historical cost-effectiveness of the previous data block under the historical state compression amount, and calculate the alternative cost-effectiveness of the current optional compression strategy based on the cost-effectiveness of the current optional compression strategy and the historical cost-effectiveness.

10. The data compression method according to claim 8, characterized in that, The step of determining the maximum cost-effectiveness of the data block under the current cumulative compression amount based on the alternative cost-effectiveness of the currently available compression strategies includes: Obtain the maximum cost-effectiveness of the previous data block under the current cumulative compression amount; If the cost-effectiveness of the current optional compression strategy is greater than the maximum cost-effectiveness of the previous data block under the current cumulative compression amount, then the cost-effectiveness of the current optional compression strategy shall be taken as the maximum cost-effectiveness of the data block under the current cumulative compression amount.

11. The data compression method according to claim 5, characterized in that, The step of determining the target compression strategy corresponding to each data block based on the cost-effectiveness array and the strategy record array includes: The target compression amount corresponding to the target data block is obtained by searching the cost-performance array; Based on the target compression amount corresponding to the target data block and the strategy record array, the target compression strategy corresponding to each data block is obtained by backtracking.

12. The data compression method according to claim 11, characterized in that, The step of backtracking to obtain the target compression strategy corresponding to each data block based on the target compression amount corresponding to the target data block and the strategy record array includes: According to the target compression amount corresponding to the target data block, the target compression strategy corresponding to the target data block is found from the strategy record array; Based on the compressible length of the target data block in the target compression strategy, determine the target compression amount corresponding to the previous data block, and find the target compression strategy corresponding to the previous data block from the strategy record array based on the target compression amount corresponding to the previous data block. Iterate until the target compression strategy corresponding to each data block is obtained.

13. An electronic device, characterized in that, include: A processor and a memory, the memory storing machine-readable instructions executable by the processor, which, when the electronic device is in operation, are executed by the processor to perform the steps of the data compression method as described in any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the data compression method as described in any one of claims 1 to 12.

15. A computer program product, characterized in that, The computer program product includes computer program code, which, when executed by a computing device, performs the steps of the data compression method according to any one of claims 1 to 12.

Images