Data processing method and device, storage medium and electronic equipment

Abstract

The present disclosure provides a data processing method, device, storage medium and electronic equipment. The method comprises: obtaining resource transfer log data to be processed and a first compression dictionary, the first compression dictionary having a first version number, and the first compression dictionary being stored in a first storage device; obtaining first historical resource transfer log data corresponding to the resource transfer log data to be processed, and determining a target compression level according to the first historical resource transfer log data and the first compression dictionary; compressing the resource transfer log data to be processed based on the first compression dictionary and the target compression level, and generating first compressed data based on the first version number and the compressed data; encrypting the first compressed data, and sending the encrypted data to a second storage device for storage, the second storage device being different from the first storage device. The method can improve data security.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a data processing method, apparatus, storage medium and electronic device. Background Technology

[0002] In today's digital age, data processing has become a core component of information technology, widely applied in fields such as the Internet, the Internet of Things, cloud computing, and mobile communications. Data processing enables the efficient and accurate transmission of data from the sending end to the receiving end. However, with the explosive growth of data volume, existing data processing technologies face numerous challenges, particularly in terms of transmission rate, transmission efficiency, transmission cost, and storage.

[0003] In data processing, the size of the data directly affects transmission time and storage resources. To improve data transmission efficiency and save storage resources, data compression technology has emerged. Data compression encodes data using specific algorithms, removing redundant information and thus reducing storage space and transmission volume.

[0004] However, current data processing methods lack sufficient data security. Summary of the Invention

[0005] This disclosure provides a data processing method, apparatus, storage medium, and electronic device, which can improve data security.

[0006] The first aspect of this disclosure provides a data processing method, including:

[0007] Obtain the resource transfer log data to be processed and the first compressed dictionary. The first compressed dictionary has a first version number and is stored in the first storage device.

[0008] Obtain the first historical resource transfer log data corresponding to the resource transfer log data to be processed, and evaluate and determine the target compression level based on the first historical resource transfer log data and the first compression dictionary;

[0009] The resource transfer log data to be processed is compressed based on the first compression dictionary and the target compression level, and the first compressed data is generated based on the first version number and the compressed data.

[0010] The first compressed data is encrypted, and the encrypted data is sent to a second storage device for storage. The second storage device is different from the first storage device.

[0011] A second aspect of this disclosure provides a data processing apparatus, comprising:

[0012] The first acquisition unit is used to acquire resource transfer log data to be processed and a first compressed dictionary. The first compressed dictionary has a first version number and is stored in a first storage device.

[0013] The second acquisition unit is used to acquire the first historical resource transfer log data corresponding to the resource transfer log data to be processed, and to evaluate and determine the target compression level based on the first historical resource transfer log data and the first compression dictionary.

[0014] The compression unit is used to compress the resource transfer log data to be processed based on the first compression dictionary and the target compression level, and to generate the first compressed data based on the first version number and the compressed data.

[0015] The sending unit is used to encrypt the first compressed data and send the encrypted data to a second storage device for storage. The second storage device is different from the first storage device.

[0016] Optionally, in one feasible implementation, the transmitting unit includes:

[0017] The encryption subunit is used to encrypt the first compressed data to obtain encrypted data;

[0018] The encoding subunit is used to encode the encrypted data to obtain encoded data;

[0019] The transmitting subunit is used to send the encoded data to the second storage device for storage.

[0020] Optionally, in one feasible implementation, the transmitting subunit includes:

[0021] Add a module to add version protocol data in a preset format to the encoded data to obtain format verification data;

[0022] The sending module is used to send the format verification data to the second storage device for storage.

[0023] Optionally, in one feasible implementation, the compression unit includes:

[0024] The merging sub-unit is used to merge the protocol data and transfer object data in the resource transfer log data to be processed into combined data;

[0025] The first compression subunit is used to compress the combined data based on the first compression dictionary and the target compression level, and to generate the first compressed data based on the first version number and the compressed data.

[0026] Optionally, in one feasible implementation, the second acquisition unit includes:

[0027] The first evaluation subunit is used to evaluate the compression effect of different compression levels based on the first historical resource transfer log data and the first compression dictionary, and obtain compression effect evaluation indicators corresponding to multiple compression levels.

[0028] The first determining subunit is used to determine the target compression level among multiple compression levels based on compression effect evaluation indicators.

[0029] Optionally, in one feasible implementation, the compression unit includes:

[0030] The second compression subunit is used to compress the resource transfer log data to be processed based on the first compression dictionary and the target compression level, and to generate second compressed data based on the first version number and the compressed data.

[0031] The second evaluation subunit is used to evaluate the compression ratio of the second compressed data to obtain the target compression ratio.

[0032] The adjustment subunit is used to adjust the first compression dictionary when the target compression ratio is less than a preset threshold, so as to obtain the second compression dictionary and the second version number corresponding to the second compression dictionary;

[0033] The third compression subunit is used to compress the resource transfer log data to be processed based on the second compression dictionary and the target compression level, and to generate the first compressed data based on the second version number and the compressed data.

[0034] Optionally, in one feasible implementation, the adjustment subunit includes:

[0035] The determination module is used to identify duplicate data in the resource transfer log data to be processed.

[0036] The update module is used to update the first compressed dictionary based on duplicate data to obtain the second compressed dictionary and the second version number corresponding to the second compressed dictionary.

[0037] A third aspect of this disclosure provides an apparatus for constructing a first compressed dictionary, comprising:

[0038] The third acquisition unit is used to acquire multiple second historical resource transfer log data within a preset time range;

[0039] The building unit is used to construct a first compression dictionary for compressing resource transfer log data based on multiple second historical resource transfer log data.

[0040] Optionally, in one feasible implementation, the building unit includes:

[0041] The second determining subunit is used to determine the protocol data format of multiple second historical resource transfer log data;

[0042] A sub-unit is constructed to build a first compression dictionary for compressing resource transfer log data based on multiple second historical resource transfer log data and protocol data formats.

[0043] A fourth aspect of this disclosure provides an apparatus for processing encrypted data stored in a second storage device, comprising:

[0044] The decryption unit is used to decrypt the encrypted data to obtain the first version number and the first compressed data;

[0045] The query unit is used to initiate a compressed dictionary query request to the first storage device based on the first version number, and to receive the first compressed dictionary returned by the first storage device;

[0046] The decompression unit is used to decompress the first compressed data based on the first compression dictionary to obtain resource transfer log data.

[0047] The fifth aspect of this disclosure provides an apparatus for processing a compressed dictionary query request, comprising: the compressed dictionary query request including first permission information;

[0048] The permission verification unit is used to verify permissions based on the first permission information and obtain the permission verification result.

[0049] The dictionary feedback unit is used to send the first compressed dictionary to the second storage device based on the first version number when the permission verification result is successful.

[0050] The sixth aspect of this disclosure provides a storage medium storing a computer program that, when executed by a processor, implements the data processing method as described in the first aspect.

[0051] The seventh aspect of this disclosure provides an electronic device including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the data processing method as described in the first aspect.

[0052] The eighth aspect of this disclosure provides a computer program product comprising a computer program that is read and executed by a processor of an electronic device, causing the electronic device to perform the data processing method as described in the first aspect.

[0053] Therefore, the data processing method provided in this disclosure includes: acquiring resource transfer log data to be processed and a first compression dictionary, the first compression dictionary having a first version number and being stored in a first storage device; acquiring first historical resource transfer log data corresponding to the resource transfer log data to be processed, and evaluating and determining a target compression level based on the first historical resource transfer log data and the first compression dictionary; compressing the resource transfer log data to be processed based on the first compression dictionary and the target compression level, and generating first compressed data based on the first version number and the compressed data; encrypting the first compressed data, and sending the encrypted data to a second storage device for storage, the second storage device being different from the first storage device.

[0054] This method involves pre-constructing a first compression dictionary and storing it on a first storage device. Based on the historical resource transfer log data corresponding to the resource transfer log data to be processed and the first compression dictionary, a compression level is determined. The resource transfer log data to be processed is then compressed based on the first compression dictionary and the compression level, and a first version number corresponding to the first compression dictionary is encapsulated. Finally, the encapsulated compressed data is encrypted and sent to a second storage device. Only by using the same version of the compression dictionary can the decompression party correctly decompress the data, thus improving data security.

[0055] Other features and advantages of this disclosure will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the disclosure. The objectives and other advantages of this disclosure may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0056] The accompanying drawings are provided to further understand the technical solutions of this disclosure and constitute a part of the specification. They are used together with the embodiments of this disclosure to explain the technical solutions of this disclosure and do not constitute a limitation on the technical solutions of this disclosure.

[0057] Figure 1 A system architecture diagram for the data processing method used in the embodiments of this disclosure;

[0058] Figure 2 This is a flowchart illustrating a data processing method according to an embodiment of the present disclosure;

[0059] Figure 3 This is a schematic diagram illustrating the selection of the target compression level in an embodiment of this disclosure;

[0060] Figure 4 This is a schematic diagram of compressed dictionary lookup in an embodiment of this disclosure;

[0061] Figure 5 This is a schematic diagram illustrating the parsing of encoded data by the second storage device in an embodiment of this disclosure;

[0062] Figure 6 This is another schematic diagram of the data processing method in an embodiment of this disclosure;

[0063] Figure 7 This is another schematic diagram of the data processing method in an embodiment of this disclosure;

[0064] Figure 8 This is a schematic diagram of the structure of a data processing device according to an embodiment of the present disclosure;

[0065] Figure 9 This is a terminal structure diagram of each method in one embodiment of this disclosure;

[0066] Figure 10 This is a server structure diagram of various methods in one embodiment of this disclosure. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this disclosure.

[0068] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “corresponding to,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0069] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0070] In this disclosure, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0071] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0072] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. The terminology used herein is for the purpose of describing embodiments of this disclosure only and is not intended to be limiting of this disclosure.

[0073] To facilitate understanding, the terms used in the embodiments of this application will be explained below.

[0074] Zlib is a widely used lossless data compression library that provides fast compression and decompression speeds while maintaining high compression efficiency. Zlib's working principle can be divided into two main stages: data compression and data decompression. In the compression stage, Zlib first analyzes the input data using the LZ77 algorithm, looking for repeating string patterns and replacing them with pointers to their previous occurrences, thus reducing redundant information. Subsequently, it uses Huffman coding to further encode the compressed data, mapping more frequent symbols to shorter codes and less frequent symbols to longer codes, thereby further improving the compression ratio. In the decompression stage, Zlib recovers the original data through a reverse operation, first decoding the Huffman codes and then restoring the repeating string patterns based on the pointer information from the LZ77 algorithm.

[0075] ZSTD (Zstandard) is a fast, lossless data compression algorithm developed by Facebook, designed to provide high-performance compression and decompression capabilities while achieving a good balance between compression ratio and speed. It combines advanced algorithm design with efficient implementation, making it outstanding in a variety of application scenarios. ZSTD's core advantages lie in its high compression ratio and fast decompression capabilities, making it particularly efficient when handling large-scale data storage and network transmission. It supports configurable compression levels, allowing users to flexibly adjust the trade-off between speed and compression ratio according to their actual needs.

[0076] Base64 is an encoding method for converting binary data into text format. It uses 64 printable characters to represent binary data, including uppercase and lowercase letters, numbers, and two special characters (usually "+" and " / "). The core of Base64 encoding is to group the binary data into groups of 3 bytes each. Each group of 24 bits is divided into four subgroups of 6 bits each, and each subgroup is then mapped to a corresponding character in the Base64 character set.

[0077] Base122 is a non-standard binary-to-text encoding scheme that uses 7-bit ASCII characters and 28 special two-byte sequences to represent data. Base122 divides binary data into subgroups of 7 bits for encoding. This encoding method utilizes the single-byte character characteristic of UTF-8 encoding, where the first 7 bits of each byte can be used to represent data, while the highest bit is reserved as 0.

[0078] Single-encryption technology has significant limitations in data security protection. Relying on a single encryption algorithm or mode, it struggles to cope with complex security threats. When faced with targeted attacks, such as encryption algorithm vulnerability exploitation, brute-force attacks, or side-channel attacks, single-encryption systems lack redundant protection mechanisms. Once the key is leaked or the algorithm fails, the entire data protection mechanism will completely fail, resulting in insufficient data security. Furthermore, the size of the data directly affects transmission time and storage resources. To improve data transmission efficiency and save storage resources, data compression technology has emerged. Data compression encodes data using compression algorithms, removing redundant information and thus reducing storage space and transmission volume. However, current mainstream compression algorithms still have bottlenecks in compression ratio, failing to fully meet the growing demand for efficient storage and transmission. Some algorithms, in order to balance processing speed and compression effect, have to sacrifice compression ratio, failing to maximize network resource savings and reduce data storage costs.

[0079] In view of the above-mentioned technical problems, this disclosure provides a data processing method in order to solve the technical problems.

[0080] System architecture and scenario description of the embodiments disclosed herein

[0081] Figure 1 This is a system architecture diagram for a data processing method applied according to an embodiment of the present disclosure. It includes a terminal 140, an Internet 130, a gateway 120, a server 110, etc.

[0082] Terminal 140 includes various forms of devices with display screens, such as desktop computers, laptops, PDAs (personal digital assistants), mobile phones, in-vehicle terminals, home theater terminals, dedicated terminals, intelligent voice interaction devices, smart home appliances, or aircraft. Furthermore, it can be a single device or a collection of multiple devices. Terminal 140 can communicate with the Internet 130 via wired or wireless means to exchange data.

[0083] Server 110 refers to a computer system that can provide certain services to terminal 140. Compared to ordinary terminal 140, server 110 has higher requirements in terms of stability, security, and performance. Server 110 can be a single high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a single high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines).

[0084] Gateway 120, also known as an internetwork connector or protocol converter, is a computer system or device that acts as a translator, enabling network interconnection at the transport layer. It bridges the gap between two systems using different communication protocols, data formats, languages, or even completely different architectures. Gateways can also provide filtering and security functions. Messages sent from terminal 140 to server 110 are forwarded to the corresponding server 110 via gateway 120. Messages sent from server 110 to terminal 140 are also forwarded to the corresponding terminal 140 via gateway 120.

[0085] The data processing method provided in this embodiment can be implemented separately in terminal 140, separately in server 110, or partially in terminal 140 and partially in server 110.

[0086] When the data processing method provided in this embodiment is implemented solely in terminal 140, terminal 140 acquires resource transfer log data to be processed and a first compression dictionary. The first compression dictionary has a first version number and is stored in a first storage device. Terminal 140 acquires first historical resource transfer log data corresponding to the resource transfer log data to be processed and evaluates and determines a target compression level based on the first historical resource transfer log data and the first compression dictionary. Terminal 140 compresses the resource transfer log data to be processed based on the first compression dictionary and the target compression level, and generates first compressed data based on the first version number and the compressed data. Terminal 140 encrypts the first compressed data and sends the encrypted data to a second storage device for storage. The second storage device is different from the first storage device.

[0087] When the data processing method provided in this embodiment is implemented solely in server 110, server 110 obtains resource transfer log data to be processed and a first compression dictionary. The first compression dictionary has a first version number and is stored in a first storage device. Server 110 obtains first historical resource transfer log data corresponding to the resource transfer log data to be processed, and evaluates and determines a target compression level based on the first historical resource transfer log data and the first compression dictionary. Server 110 compresses the resource transfer log data to be processed based on the first compression dictionary and the target compression level, and generates first compressed data based on the first version number and the compressed data. Server 110 encrypts the first compressed data and sends the encrypted data to a second storage device for storage. The second storage device is different from the first storage device.

[0088] When the data processing method provided in this embodiment is partially implemented in the terminal 140 and partially implemented in the server 110, the terminal 140 obtains the resource transfer log data to be processed and a first compression dictionary, and sends the resource transfer log data to be processed and a first version number of the first compression dictionary to the server. The first compression dictionary has a first version number and is stored in a first storage device. The server 110 obtains the first historical resource transfer log data corresponding to the resource transfer log data to be processed, and evaluates and determines a target compression level based on the first historical resource transfer log data and the first compression dictionary. The server 110 compresses the resource transfer log data to be processed based on the first compression dictionary and the target compression level, and generates first compressed data based on the first version number and the compressed data. The server 110 encrypts the first compressed data and sends the encrypted data to the terminal 140. The terminal 140 stores the encrypted data in a second storage device, which is different from the first storage device.

[0089] The electronic devices for data processing provided in this disclosure can be various types of terminal devices or servers. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal devices and servers can be directly or indirectly connected via wired or wireless communication, and this disclosure does not impose any limitations on this.

[0090] Taking servers as an example, such as server clusters deployed in the cloud, AI as a Service (AIaaS) is provided to objects. The AIaaS platform will break down several common AI services and provide independent or packaged services in the cloud. This service model is similar to an AI-themed marketplace. All objects can access and use one or more artificial intelligence services provided by the AIaaS platform through application programming interfaces.

[0091] This disclosure can be applied to various scenarios. Taking the transmission of transaction log data as an example, ensuring a balance between transmission efficiency and data security is a core challenge in building a reliable transaction system. For instance, transaction logs containing sensitive information such as transaction amount, timestamp, account information, and transaction type are crucial for transaction tracing and also involve the privacy and financial security of transaction parties. Therefore, the transmission process must simultaneously meet stringent requirements for both efficiency and confidentiality. Data redundancy leads to high bandwidth consumption and slow transmission speeds, and may pose a risk of leakage due to insufficient protection of sensitive information. Malicious interception or tampering could trigger financial disputes, credit crises, or even legal consequences. The data processing method of this disclosure can compress data using a target version of a compression dictionary. Only the corresponding version of the compression dictionary can decompress the data, improving data security. Furthermore, the compressed data structure has been altered and encrypted, making the data even more difficult to crack during transmission.

[0092] The above examples do not limit the scope of protection in this case.

[0093] General Description of Embodiments in this Disclosure

[0094] The data processing method provided in the embodiments of this disclosure will be described below with reference to the accompanying drawings. The execution subject of the data processing method described below is a terminal device, which can be implemented by the terminal device by running the various computer programs described above. Of course, based on the understanding of the following text, it is not difficult to see that the data processing method provided in the embodiments of this disclosure can also be implemented by the terminal device and the server in collaboration.

[0095] Please see Figure 2 ,like Figure 2 The diagram shown is a flowchart illustrating a data processing method provided in an embodiment of this disclosure. The method includes:

[0096] Step 201: Obtain the resource transfer log data to be processed and the first compressed dictionary.

[0097] Resource transfer log data refers to detailed information recording the process of transferring resources between different entities (such as the resource transfer object, system, account, etc.), and can be used in fields such as financial transactions and data transmission. Taking resource transfer log data in financial transactions as an example, it is necessary to ensure the security of the transmission of resource transfer log data.

[0098] The first compressed dictionary can be a mapping table (a set of key-value pairs) data structure that stores common data patterns or recurring content and their corresponding codes. For example, if "time" and "amount" frequently appear in the resource transfer log data, they can be mapped to shorter codes. The first compressed dictionary can have a first version number, which can be a string or a sequence of numbers used to distinguish different versions of the compressed dictionary. Assuming the version number of the first compressed dictionary is v1.0, it indicates that this is the first official version; when the dictionary is updated, the version number can become v1.1. The first compressed dictionary can be stored on a first storage device, which can be a medium or system for storing data. It can be a local physical storage device (such as a hard drive or solid-state drive) or a remote virtual storage service (such as cloud storage). The resource transfer log data to be processed and the first compressed dictionary can be obtained by the system actively fetching it or by executing a fetch command.

[0099] In one implementation, the process of constructing the first compressed dictionary includes:

[0100] Acquire multiple second historical resource transfer log data within a preset time range;

[0101] A first compression dictionary is constructed based on multiple second historical resource transfer log data to compress the resource transfer log data.

[0102] In this embodiment, the preset time range can refer to a specific time period that is pre-set according to specific needs in data processing and analysis. This time period can limit the scope of data collection, processing or analysis, and ensure that the processed data is timely and relevant.

[0103] The second historical resource transfer log data refers to historical data used to analyze and extract duplicate content when constructing the first compressed dictionary. This data typically originates from resource transfer operations already recorded in the system, reflecting resource transfer activities within a preset time frame. The construction of the first compressed dictionary can involve statistically analyzing the frequency of each field in the second historical resource transfer log data and its repetition frequency across different log data, such as the log format (title, time, event) and the number of times the template appears. Through frequency analysis, frequently occurring content can be identified, and the compressed dictionary can be constructed. Each entry in the compressed dictionary contains a frequently occurring content and a corresponding code. Selecting resource transfer log data from a recent period, such as the past month, three months, or one year, as the second historical resource transfer data can improve the timeliness of duplicate content in the constructed first compressed dictionary.

[0104] In one implementation, a first compression dictionary for compressing the resource transfer log data is constructed based on multiple second historical resource transfer log data, including:

[0105] Determine the protocol data format for multiple second historical resource transfer log data;

[0106] A first compression dictionary is constructed based on multiple second historical resource transfer log data and protocol data formats to compress the resource transfer log data.

[0107] In this implementation, the protocol data format refers to the structure and specifications followed by the data during data transmission and storage. It defines the data organization method, the order of fields, data types, and the relationships between fields. These fields appear repeatedly in each record. For example, resource transfer log data typically includes fields such as timestamp, resource type, resource quantity, resource source, and target. These fields exist in every record and have a consistent format. By assigning encodings to these repeating fields, data redundancy can be significantly reduced. It is understood that the first compression dictionary can configure encodings not only for the protocol data format but also for other frequently occurring content in the second historical resource transfer log data, as well as for some common data; this is not limited here.

[0108] Step 202: Obtain the first historical resource transfer log data corresponding to the resource transfer log data to be processed, and evaluate and determine the target compression level based on the first historical resource transfer log data and the first compression dictionary.

[0109] The first historical resource transfer log data refers to information related to resource transfer operations recorded and stored prior to the current processing cycle; that is, data of the same type as the resource transfer log data to be processed. Historical resource transfer log data can be stored in a database and organized and managed through table structures. For example, a dedicated table can be used to record resource transfer serial numbers within historical cycles.

[0110] The target compression level refers to the pre-set compression effect level during data compression based on specific needs and conditions. It is a parameter that guides the strategy employed by the compression algorithm during compression operations, and can be a strategy that balances compression efficiency (the degree of reduction in data size) and compression cost (computational resources, time, and complexity). Understandably, a high compression level will minimize data size and achieve a higher compression ratio, but may consume more computational resources and time. A low compression level can employ simpler and faster compression strategies, resulting in a relatively lower compression ratio, but with less computational resource consumption and processing time. It can also be a compression level determined by evaluating the compression results of a first historical resource transfer log data based on a first compression dictionary.

[0111] In one implementation, determining the target compression level based on first historical resource transfer log data and a first compression dictionary evaluation includes:

[0112] Based on the first historical resource transfer log data and the first compression dictionary, the compression effect of different compression levels is evaluated, and multiple compression effect evaluation indicators corresponding to the compression levels are obtained.

[0113] The target compression level is determined from multiple compression levels based on compression performance evaluation indicators.

[0114] In this implementation, compression performance evaluation metrics refer to a series of quantitative standards used to measure and evaluate the performance of compression algorithms during data compression. These metrics determine the most suitable compression level for a specific application scenario by evaluating the performance of the compression algorithm at different compression levels. These metrics may include compression ratio, compression time, decompression time, resource consumption, data integrity, compressed data size, compression ratio, compression efficiency, and reliability. Here, the compression performance evaluation metrics can be obtained by applying different compression levels to the first historical resource transfer log data and evaluating the compression effect at each level. Analyzing the recorded evaluation metrics and comparing the compression ratio, compression time, decompression time, and resource consumption at different compression levels can help find a compression level that achieves the best balance between compression efficiency and computational cost as the target compression level. For example, if the application scenario has high requirements for storage space or transmission bandwidth, a compression level with a higher compression ratio can be selected; if the application scenario has high real-time requirements, a compression level with a shorter compression time can be selected. Determining the target compression level based on multiple metrics in the compression performance evaluation metrics allows for selection based on application requirements, improving the accuracy of compression level selection.

[0115] In some embodiments, please refer to Figure 3 The diagram shown illustrates the selection of the target compression level. Figure 3 The process compresses historical resource transfer log data at different levels and selects the final target compression level based on compression result metrics. First, the process begins with historical resource transfer log data 301 as input, which needs to be compressed to reduce storage space or improve transmission efficiency. The data is processed through three different compression levels 302 (compression level 1, compression level 2, and compression level 3; the number of compression levels here is just an example). Each compression level can use different compression algorithms or parameter settings to achieve different degrees of data compression. After compression, each level generates a compression result metric 303 (compression result metric 1, compression result metric 2, and compression result metric 3). These metrics can include compression ratio, compressed file size, compression speed, etc., used to evaluate the effectiveness of different compression levels. Next, an optimal target compression level 304 can be selected from multiple compression levels based on these compression result metrics. This can be a comprehensive evaluation of various compression result metrics to determine which compression level meets specific needs (such as minimizing storage space and maximizing compression speed) while maintaining data integrity and availability.

[0116] Step 203: Compress the resource transfer log data to be processed based on the first compression dictionary and the target compression level, and generate the first compressed data based on the first version number and the compressed data.

[0117] The resource transfer log data to be processed can be parsed and broken down into individual fields. For each field, it can be matched against a first compression dictionary. If a match is found, the field is replaced with the corresponding code in the first dictionary. If a match fails, the original field is retained or another compression strategy is applied. Then, based on the target compression level, a suitable compression algorithm is selected to further compress the replaced data, resulting in compressed data. For example, taking the "resource type" field in a log record as an example, if the code for "resource type" in the first compression dictionary is "0x01", then in the compressed data, this field will be replaced with "0x01". To ensure that the decompression device can correctly identify the compression dictionary used for code replacement, the first version number corresponding to the first compression dictionary can be combined with the compressed data. This can be achieved by encapsulating the first version number into the header of the compressed data; the encapsulated compressed data can be called the first compressed data.

[0118] In one implementation, the resource transfer log data to be processed is compressed based on a first compression dictionary and a target compression level, and first compressed data is generated based on a first version number and the compressed data, including:

[0119] Merge the protocol data and transfer object data in the resource transfer log data to be processed into combined data;

[0120] The combined data is compressed based on the first compression dictionary and the target compression level, and the first compressed data is generated based on the first version number and the compressed data.

[0121] In this implementation, protocol data can refer to metadata used to describe the data structure and format during data transmission and storage. It defines the organization, field order, data type, and relationships between fields in the resource transfer log data to be processed.

[0122] Transfer object data can refer to various information related to the resource transfer object in the resource transfer log data to be processed. Here, it can be data related to the resource transfer operation or behavior of the resource transfer object, such as the account information of the resource transfer object, operation time, operation type, operation amount, etc.

[0123] Combined data refers to merging multiple related data elements into a single data structure. Here, protocol data and transfer object data are merged into a single data structure. The system can compress the combined data based on a first compression dictionary and a target compression level. Specifically, based on the predefined mapping rule set provided by the first compression dictionary, the system accurately identifies repetitive patterns, redundant fields, or frequently occurring features in the combined data by combining them with the target compression level. This data volume is reduced by replacing or encoding these elements. Because the combined data deeply integrates protocol data and transfer object data, the compression algorithm can process the entire data structure at once, avoiding the additional overhead of compressing each sub-data separately and significantly improving compression efficiency. After compression, the system further encapsulates the first version number with the compressed data to generate the first compressed data.

[0124] In one implementation, the resource transfer log data to be processed is compressed based on a first compression dictionary and a target compression level, and first compressed data is generated based on a first version number and the compressed data, including:

[0125] The resource transfer log data to be processed is compressed based on the first compression dictionary and the target compression level, and second compressed data is generated based on the first version number and the compressed data.

[0126] The compression ratio of the second compressed data is evaluated to obtain the target compression ratio.

[0127] When the target compression ratio is less than a preset threshold, the first compression dictionary is adjusted to obtain the second compression dictionary and the second version number corresponding to the second compression dictionary;

[0128] The resource transfer log data to be processed is compressed based on the second compression dictionary and the target compression level, and the first compressed data is generated based on the second version number and the compressed data.

[0129] In this embodiment, the second compressed data can refer to the data obtained after compressing the resource transfer log data to be processed based on the first compression dictionary and the target compression level, which can be called the initial compression result.

[0130] The target compression ratio can be expressed as a percentage, representing the ratio of the compressed data size to the original data size. It's used to evaluate the efficiency and effectiveness of the compression algorithm. A higher compression ratio indicates smaller compressed data and better compression performance. Here, it could be the ratio between the second-compressed data and the resource transfer log data to be processed.

[0131] The second compression dictionary can be an optimized version of the first compression dictionary, generated by adjusting it. It is generated when the target compression ratio of the initial compression result does not reach a preset threshold, by analyzing log data characteristics and adjusting the mapping relationships in the first compression dictionary. The preset threshold can be a predefined expected compression ratio; that is, when the target compression ratio is not met, the first compression dictionary is adjusted to achieve the target compression ratio. The second version number can be a unique identifier associated with the second compression dictionary, that is, it can uniquely identify the second compression dictionary. Understandably, when the target compression ratio of the initial compression result does not reach the preset threshold, the characteristics of the resource transfer log data can be analyzed in depth, such as frequently occurring strings, data patterns, or structural regularities. Based on these strings, data patterns, and structural regularities, the mapping relationships in the first compression dictionary are adjusted in a targeted manner, thereby generating the second compression dictionary and its uniquely identifying second version number. Through optimized mapping rules, the second compression dictionary can more accurately identify and compress redundant information in the log data, providing more efficient matching capabilities for subsequent compression. Based on this, the system compresses the resource transfer log data to be processed again based on the updated second compression dictionary and target compression level, and generates the final first compressed data by combining the second version number. This allows the compression process to dynamically adapt to the characteristics of the log data. By continuously adjusting the mapping strategy of the compression dictionary, the compression rate is significantly improved, ensuring that the same information is carried with a smaller amount of data during storage or transmission.

[0132] In one implementation, adjusting the first compressed dictionary to obtain a second compressed dictionary and a second version number corresponding to the second compressed dictionary includes:

[0133] Identify duplicate data in the resource transfer log data to be processed;

[0134] The first compressed dictionary is updated based on the duplicate data to obtain the second compressed dictionary and the second version number corresponding to the second compressed dictionary.

[0135] In this implementation, duplicate data can be data that appears repeatedly in a resource transfer log to be processed. This could include common log formats, duplicate error messages, frequently occurring user operation records, etc. Updating the first compression dictionary can be done by adding or optimizing the mapping relationships in the dictionary to more effectively identify and replace duplicate data in the log data. The updated mapping relationships can then be called the second compression dictionary.

[0136] In some embodiments, the system can perform a comprehensive scan of the resource transfer log data to be processed, identifying duplicate data. This duplicate data can manifest as fixed log format templates, frequently occurring error codes, or regular operation records. By accurately locating this duplicate data, the first compression dictionary can be updated accordingly. The update can be achieved by optimizing the mapping relationships in the dictionary, enabling it to more efficiently match and replace duplicate content in the resource transfer log data. For example, for frequently occurring specific log format strings, the system can add new mapping entries to the compression dictionary or adjust the priority of existing entries to ensure that these duplicate parts can be quickly identified and compressed during compression. This optimization not only improves the adaptability of the compression dictionary to the current resource transfer log data but also enhances the overall efficiency of the compression algorithm. The second compression dictionary, through mapping rules that better match the characteristics of the log data, can improve the compression ratio.

[0137] Step 204: Encrypt the first compressed data and send the encrypted data to the second storage device for storage. The second storage device is different from the first storage device.

[0138] Before transmitting the resource transfer log data to be processed, in addition to data compression, the compressed data can also be encrypted. This can be done by selecting a suitable encryption algorithm based on security requirements for the first compressed data, and then sending the encrypted data to the second storage device. The encryption algorithm can be a symmetric encryption algorithm. In symmetric encryption, the sender uses a key to encrypt the data and then sends the encrypted data to the receiver. The receiver uses the same key to decrypt the data, thereby recovering the original data. Examples include Advanced Encryption Standard (AES) and Triple Data Encryption Algorithm (3DES). The second storage device can be a different storage medium or system than the first storage device. The first storage device ensures secure storage and fast access to the compressed dictionary, while the second storage device can be used for data backup, redundant storage, or long-term archiving.

[0139] In one embodiment, the process of processing encrypted data stored in the second storage device includes:

[0140] Decrypt the encrypted data to obtain the first version number and the first compressed data;

[0141] A compressed dictionary query request is initiated to the first storage device based on the first version number, and the first compressed dictionary returned by the first storage device is received.

[0142] The first compressed data is decompressed based on the first compression dictionary to obtain resource transfer log data.

[0143] In this embodiment, after receiving the encrypted data, the second storage device can first decrypt the encrypted data, restoring it to a readable format. After decryption, the obtained data may include a first version number and first compressed data. The first version number indicates the version of the compression dictionary used during the compression process.

[0144] Understandably, the second storage device may or may not store a compression dictionary locally. When a compression dictionary is stored locally, the second storage device first checks if the version number of its local compression dictionary matches the first version number. If the version number matches, the second storage device can directly use its local compression dictionary to decompress the first compressed data. If the version numbers are different or the compression dictionary is not stored locally, a compression dictionary query request carrying the first version number can be sent to the first storage device. This query request instructs the first storage device to retrieve the compression dictionary corresponding to the first version number. The first storage device then sends the first compression dictionary corresponding to the first version number back to the second storage device. The second storage device then decompresses the first compressed data according to the received first compression dictionary to obtain the resource transfer log data to be processed before compression in the system. The compression dictionary query request ensures the consistency of dictionary versions during compression and decompression, supports dynamic dictionary management, and efficient data processing.

[0145] In one implementation, the process of the first storage device processing a compressed dictionary lookup request includes:

[0146] Based on the first permission information, perform permission verification to obtain the permission verification result;

[0147] When the permission verification result is successful, the first compression dictionary is sent to the second storage device based on the first version number.

[0148] In this embodiment, the compressed dictionary query request may further include first permission information. This first permission information verifies whether the requester has permission to obtain the specified compressed dictionary. Here, the control system of the second storage device verifies with the management system whether it has permission to obtain the first compressed dictionary corresponding to the first version number. The first permission information may include authentication information (such as requester name, password, API key, token, etc.) and authorization information (such as role, permission level, etc.) to ensure that only authorized requesting objects or systems can access the compressed dictionary in the first storage device.

[0149] The permission verification result can be the output of the permission verification process, indicating whether the requester has passed the permission verification. It can be a Boolean value (such as "Pass" or "Fail") or a status code indicating whether the requester has the right to access the compressed dictionary. In some embodiments, an authentication mechanism can be used to verify the legitimacy of the requester. The requester's permission level is checked to confirm whether they have the right to obtain the specified version of the compressed dictionary. If the permission verification passes, the query request continues to be processed; otherwise, an error message (such as "Insufficient permissions") is returned.

[0150] When the permission verification result indicates that the second storage device has passed verification, the control system can retrieve the first compressed dictionary corresponding to the first version number from the first storage device. By generating and processing the permission verification result, the system can ensure that only authorized requesting objects or systems can perform specific operations, thereby protecting the confidentiality and integrity of data.

[0151] In some embodiments, please refer to Figure 4 The diagram shown illustrates a compressed dictionary lookup. Figure 4 The diagram illustrates how a requester retrieves compressed data from storage device 1, parses the version number from the compressed data, generates a compressed dictionary query request based on the version number and the requester's permission information, and sends it to the receiver. The receiver can then verify the requester's permissions for that version number. If the verification process finds insufficient permissions to access the requested resource, the receiver can return an error message to the requester, explicitly indicating the permission issue. If the permission verification passes, the receiver can send a valid verification message back to the requesting method. The requester can then retrieve the compressed dictionary corresponding to that version number from storage device 2 based on this valid information. This feedback mechanism ensures that only devices with appropriate permissions can access the required compressed dictionary, thereby protecting data security and privacy to a certain extent.

[0152] In one implementation, the first compressed data is encrypted, and the encrypted data is sent to a second storage device for storage, including:

[0153] The first compressed data is encrypted to obtain encrypted data;

[0154] Encode the encrypted data to obtain encoded data;

[0155] The encoded data is sent to a second storage device for storage.

[0156] In this implementation, encrypted data refers to data processed by an encryption algorithm, the content of which is unreadable before decryption. Encryption can protect the confidentiality, integrity, and availability of data, preventing unauthorized access or tampering, and requires the use of specific keys or algorithms to restore the original data.

[0157] Encoded data can refer to data that has been processed by encoding algorithms, which is the process of converting data from one format or representation to another, making the data more suitable for transmission, storage, or processing.

[0158] This process involves applying an encryption algorithm to the first compressed data. Using a key or mathematical transformation of the algorithm, the first compressed data is converted into encrypted data, making its content unreadable gibberish before decryption. This effectively protects the confidentiality, integrity, and availability of the data, preventing unauthorized access, tampering, or damage. The encryption algorithm disrupts the original structure of the data through complex mathematical operations. Only authorized users with the corresponding key can decrypt and restore the data, ensuring data security during storage or transmission. The resulting binary encrypted data can then be encoded, converting it into a text-formatted data suitable for transmission and storage. The encoding process maps binary data into printable character sequences using specific rules, eliminating any invisible characters or special symbols that may exist in the binary data. This allows for stable transmission and storage across various network protocols and storage systems. This conversion not only solves the compatibility issues of binary data in text environments but also improves the flexibility of data storage, enabling the encoded data to adapt to various types of storage media and avoiding data corruption or read failures due to format incompatibility.

[0159] In one embodiment, sending the encoded data to a second storage device for storage includes:

[0160] Add version protocol data in a preset format to the encoded data to obtain format verification data;

[0161] The format verification data is sent to a second storage device for storage.

[0162] In this implementation, the version protocol data can be metadata appended to the data packet, used to identify the data version, format, verification information, and other relevant information, ensuring that the receiver can correctly parse and process the data. The version protocol data can be generated according to a preset format based on the compression type, encryption type, and encoding type of the resource transfer log data to be processed.

[0163] Format verification data can be added to the header of encoded data to specify the version protocol data. This allows the second storage device to determine the required decoding, decryption, and decompression types for the encoded data within the format verification data, and then perform the corresponding decoding, decryption, and decompression to obtain the resource transfer log data. It's understandable that version protocol data, as metadata, carries crucial contextual information during data processing. By embedding version protocol data in the header of encoded data, the format verification data not only preserves the content of the original encoded data but also provides the second storage device with a "parse key." When the second storage device receives the format verification data, it first reads the version protocol data and extracts complete metadata for data processing. This metadata acts like a detailed operating guide, instructing the storage device to automatically identify the decoding type, decryption algorithm, and decompression method used in the encoded data. For example, if the version protocol data indicates that the encryption algorithm is Algorithm 1, the encoding method is Algorithm 2, and the compression type is Algorithm 3, the storage device can process the data in the order of "Algorithm 2 decoding → Algorithm 1 decryption → Algorithm 3 decompression," ultimately restoring the original resource transfer log data. This design allows the second storage device to flexibly adapt to data in different processing formats, and can complete automatic parsing without manual intervention. This improves the efficiency of data processing and avoids data parsing errors caused by format mismatch.

[0164] In some embodiments, please refer to Figure 5 The diagram illustrates the second storage device parsing encoded data, processing and converting the encoded data to ultimately generate resource transfer log data. Encoded data 501 can be obtained from the diagram. This data first needs to undergo version protocol data extraction. Version protocol data 502 can be extracted from the header of encoded data 501. The type of version protocol data 502 can include version protocol data 1, version protocol data 2, and version protocol data 3 (this is just an example). These version protocol data can contain different encoding, encryption, and compression information. Version protocol data 502 can be mapped to corresponding decoding, decryption, and decompression types 503. Specifically, version protocol data 1 can be mapped to decoding, decryption, and decompression type 1; version protocol data 2 can be mapped to decoding, decryption, and decompression type 2; and version protocol data 3 can be mapped to decoding, decryption, and decompression type 3. After mapping, encoded data 501 can be processed based on decoding, decryption, and decompression types 503 to obtain resource transfer log data 504.

[0165] This disclosure provides a data processing method, which includes acquiring resource transfer log data to be processed and a first compression dictionary, the first compression dictionary having a first version number and being stored in a first storage device; acquiring first historical resource transfer log data corresponding to the resource transfer log data to be processed, and evaluating and determining a target compression level based on the first historical resource transfer log data and the first compression dictionary; compressing the resource transfer log data to be processed based on the first compression dictionary and the target compression level, and generating first compressed data based on the first version number and the compressed data; encrypting the first compressed data, and sending the encrypted data to a second storage device for storage, the second storage device being different from the first storage device.

[0166] This method involves pre-constructing a first compression dictionary and storing it on a first storage device. A compression level is determined based on the historical resource transfer log data corresponding to the resource transfer log data to be processed and the first compression dictionary. The resource transfer log data to be processed is then compressed based on the first compression dictionary and the compression level, and a first version number corresponding to the first compression dictionary is encapsulated. The encapsulated compressed data is then encrypted and sent to a second storage device. Only by using the same version of the compression dictionary can the decompression party correctly decompress the data, improving data security. Since historical resource transfer log data is related to the resource transfer log data to be processed, analyzing the compression effect of historical resource transfer log data based on the compression dictionary can more accurately predict the compression effect of the resource transfer log data to be processed, improving the accuracy of compression level selection. The compressed data structure has been altered and further encrypted, making the data more difficult to crack during transmission. Furthermore, because compression reduces the data volume, the encryption process can be completed faster, reducing the burden on the encryption algorithm.

[0167] This disclosure provides a detailed description of embodiments in conjunction with specific application scenarios.

[0168] like Figure 6 The diagram shown is another flowchart illustrating the data processing method provided in this disclosure. The method specifically includes the following steps:

[0169] Step 601: Obtain historical transaction log data within a preset time range.

[0170] The preset time range can be a specific time period that is set in advance. This specific time period can be a recent period of time to ensure that the data within this time range is timely.

[0171] Historical transaction log data refers to detailed records of all transaction activities recorded by the system within a preset time frame, such as transaction log data for the most recent three months, six months, or one year. This data may include the transaction time, amount, participants, transaction status, and other relevant information. This historical transaction log data can be obtained by decoding, decrypting, and decompressing a database that has already been compressed, encrypted, and encoded, or it can be directly obtained from the original transaction log database.

[0172] Step 602: Construct a compressed dictionary based on the duplicate data between the log data of the historical transaction log data, and generate the version number corresponding to the compressed dictionary.

[0173] Historical transaction log data can include a large number of historical transaction logs, which may contain repeated content, such as duplicate strings or templates. This duplicate data can be compressed by mapping these duplicates to shorter codes. The frequency of each data segment can be statistically analyzed, and high-frequency segments can be selected as compression targets. These high-frequency segments can then be mapped to shorter codes or symbols. A compression dictionary can then be constructed based on the mapped codes or symbols and the compression targets. This dictionary can be a collection of key-value pairs, where the keys are high-frequency segments and the values ​​are the mapped short codes. A unique version number can be generated for each version of the compression dictionary.

[0174] In some embodiments, 50,000 records of the top 19 protocols and 50,000 records of other protocols can be obtained from historical transaction log data. First, the encrypted fields in the protocols are decrypted. Then, the 100,000 decrypted records are used to construct 20 dictionary files. Each of the top 19 protocols is used to construct one dictionary, and the other protocols are used to construct one dictionary. The top 19 protocols refer to the 19 most frequently used or most important communication protocol types in the system. The structured fields of the top 19 protocols are highly standardized, while other protocols may contain more heterogeneous or sparse fields.

[0175] Step 603: Obtain the target transaction log data.

[0176] Target transaction log data can refer to the transaction log data that needs to be processed currently. This can be real-time generated transaction records, such as order details from e-commerce platforms, transfer details from financial systems, or transaction receipts from payment gateways. This data carries key information such as transaction time, amount, participants, and operation type, serving as the foundation for subsequent analysis, storage, and secure processing. There are various ways to obtain this data. It can be exported directly from the transaction system's database or through real-time interaction with external systems via API interfaces to obtain structured data streams. For example, in high-concurrency online transaction scenarios, the system can continuously capture transaction logs through real-time data stream pipelines to ensure data timeliness; while for batch processing scenarios, historical transaction data can be periodically extracted from various business systems.

[0177] Step 604: Decrypt the target transaction log data and merge the decrypted protocol data and transaction object data in the transaction log data to obtain merged data.

[0178] Protocol data refers to protocol information related to a transaction, which describes the context of the transaction and the details of the transmission protocol. This may include: transaction time: recording the specific time each transaction occurred; transaction type: describing the type of transaction, such as purchase, refund, transfer, etc.; protocol version: identifying the protocol version used to ensure data format consistency; and other metadata: such as the transaction's origin, destination, and transmission status.

[0179] Transaction object data refers to information related to a specific transaction, which may include: Transaction Object ID: recording the identifier of the object participating in the transaction; Transaction Object Account Information: recording the account information of the transaction object, such as account balance and account status; Transaction Object Behavior Data: recording the transaction object's operational behavior, such as login time and operation type; and other transaction object-related data, such as device information and IP address. Among these, Transaction Object ID and Transaction Object Account Information are short fields, while Transaction Object Behavior Data and Device Information are long fields. For example, multiple actions require a longer field to fully record the details of the transaction. Device information may include device model, device ID, IP address, and geographical location. The length and format of this information vary depending on the device type and network environment, therefore, longer fields are needed to ensure that various possible device identifiers can be accommodated.

[0180] Since protocol data and transaction object data may contain sensitive information, such as transaction object account information, transaction amount, and operational behavior, they are usually encrypted during transmission. Therefore, after obtaining the target transaction log data, the protocol data and transaction object data can be decrypted first, and then merged into a complete dataset to obtain the merged data. In some embodiments, the protocol data can be merged with two longer fields (transaction object behavior data and device information) to reduce the number of compression steps.

[0181] Step 605: Compress the merged data according to the compression dictionary, and encapsulate the version number into the compressed data to obtain compressed data.

[0182] The compression dictionary contains a mapping between repeating templates or strings and shortcodes. It can iterate through the merged data, finding parts that match the patterns or strings defined in the compression dictionary, and replacing the matching parts with the mapped shortcodes. The version number is a unique identifier for the compression dictionary, ensuring the correct dictionary version is used during decompression. To avoid data parsing errors due to version mismatches, the version number can be encapsulated as metadata in the header or footer of the compressed data. This way, during decompression, the version number is read first, and then the appropriate compression dictionary is selected for decompression based on that version number.

[0183] Various compression algorithms can be used to compress merged data, such as Zlib or ZSTD. Zlib uses the default compression method, which, simply put, first replaces repeated strings in the data with (distance, length) pairs; then, it uses Huffman coding on the replaced data to re-encode the characters according to their frequency of occurrence. Simultaneously, during the replacement process, it ensures that the memory usage of the algorithm remains controllable regardless of the data size. Zlib compression is highly effective for large volumes of data with high repetition. ZSTD compression uses the parameter `fast` to balance the compression ratio with compression and decompression speeds. Higher compression and decompression speeds result in a lower compression ratio. At its highest compression ratio, ZSTD's compression speed is 5.56 times that of Zlib, and its decompression speed is 4.15 times faster.

[0184] It should be noted that before compressing the merged data according to the compression dictionary, the target compression level can be determined. For example, after obtaining historical transaction log data, the historical transaction log data can be randomly divided into two groups: a dictionary training data group and a compression performance test data group. The dictionary training data is used to construct the compression dictionary, and then the compression performance test data is compressed at different compression levels based on this compression dictionary. The compression ratio and compression time under different compression levels can be tested. Then, the compression level that balances the compression ratio and compression time can be selected as the target compression level. The weighted compression ratio and average compression time under different compression levels are shown in Table 1. Taking 10 compression levels as an example, compression level 3 provides a lower compression ratio with a lower compression time, that is, a lower amount of compressed data. Therefore, compression level 3 can be selected as the target compression level.

[0185] Table 1 Compression Effect Table

[0186] Compression level Weighted compression ratio Average compression time 1 25.3382％ 7.27871 2 25.4935％ 9.01881 3 24.3661％ 12.8936 4 25.6832％ 13.5975 5 25.5424％ 19.745 6 24.5262％ 31.9337 7 24.0323％ 38.15 8 23.8195％ 42.2228 9 23.6473％ 47.7109 10 23.5224％ 57.0347

[0187] Step 606: Encrypt the compressed data to obtain encrypted data.

[0188] You can choose the required encryption algorithm and generate a corresponding key based on the selected algorithm. You can choose a suitable algorithm from a wide range of options based on the specific application scenario and security requirements. Different encryption algorithms have different characteristics and applicable ranges; when choosing, you should comprehensively consider factors such as data sensitivity, encryption and decryption efficiency, and key management difficulty. After determining the encryption algorithm, you can generate a corresponding key and use the encryption algorithm and key to generate encrypted data from the compressed data. The key can be generated using a secure and reliable random number generation method to ensure its randomness and unpredictability. The encryption process can involve using the selected encryption algorithm and the generated key to perform a series of specific mathematical operations on the compressed data. These mathematical operations can scramble the original structure and order of the data, transforming it into a form that is completely unreadable before decryption.

[0189] The encryption algorithm used can be a symmetric encryption algorithm, such as the Advanced Encryption Standard (AES). The AES algorithm divides the data to be encrypted into fixed-size blocks, each 128 bits in size. If the last block of data is less than 128 bits, the AES algorithm uses padding techniques to supplement the data to reach the required block size. Next, each 128-bit data block is combined with a key through a process called key scheduling. This process generates a series of round keys, which are used in subsequent encryption rounds. The AES encryption process consists of multiple rounds, each including four basic steps: byte substitution (SubBytes), row shifting (ShiftRows), column mixing (MixColumns), and round key addition (AddRoundKey). The byte substitution step uses a fixed lookup table (S-box) to replace each byte. The row shifting step rearranges the position of each byte in the row. The column mixing step mixes the columns by multiplying a fixed matrix. Finally, the round key addition step XORs the round key with the data block. After multiple rounds of this processing, the final round excludes column obfuscation steps, consisting only of byte substitution, row shifting, round key addition, and, in some cases, additional steps. This combination of rounds ensures data diffusion and obfuscation, so that each bit of the original data affects multiple output bits, making the relationship between output and input unpredictable. The AES encryption formula is C = E(K, P), where K is the key, P is the plaintext, and C is the ciphertext. The plaintext can be divided into blocks of 128 bits each, and then encrypted group by group until all plaintext is encrypted. The key length can be 128, 192, or 256 bits.

[0190] Step 607: Encode the encrypted data to obtain encoded data.

[0191] In the process of encoding encrypted data to obtain encoded data, the encrypted data in binary form can be converted into a text string that is easy to transmit and store. This conversion is achieved through a selected encoding method. The basic principle is to map binary data into a printable character sequence according to specific rules, thereby solving problems such as unreadable data, transmission errors, or format incompatibility that may occur with binary data in text protocols or storage environments. The encoding process may include dividing the encrypted binary data into fixed-length segments, then converting each segment into corresponding text characters according to the encoding rules, and padding or formatting adjustments as necessary to ensure the standardization and integrity of the output.

[0192] Understandably, the data storage cluster only supports UTF-8 character encoding. The compressed transaction log data is in binary format and cannot be saved to the data storage cluster. Therefore, the compressed target transaction log data can be specially encoded to convert the binary data into text characters. Available encoding algorithms are as follows:

[0193] The Base64 encoding algorithm uses 64 characters (AZ, az, 0-9, +, / ) as its encoding table. During encoding, 6 bits of the encrypted data are encoded using 8-bit encoding. The binary data is grouped into groups of 6 bits, each corresponding to a character in the Base64 encoding table. If there are fewer than 6 bits, an equals sign (=) is used. Decoding involves restoring the Base64 characters back to their original 6-bit values ​​to obtain the original binary data. The encoding efficiency is 75%, and the total length increases by approximately 33% after encoding.

[0194] Base122 removes six characters from the 128 characters of UTF-8: quotation marks "", newline character \n, carriage return character \r, escape characters \ and &, and the null character. The remaining 122 characters form the encoding table. During Base122 encoding, 7 bits of the data are encrypted using 8-bit encoding. The binary data is grouped into groups of 7 bits, each corresponding to a character in the Base122 encoding table. If there are fewer than 7 bits, trailing zeros are added. During Base122 decoding, the Base122 encoded characters are restored to their original 7-bit bits to obtain the original binary data. Base122 encoded strings are always valid UTF-8 encodings. The efficiency of Base122 encoding is 87.5%, and the data length increases by approximately 14%.

[0195] Step 608: Add protocol format to the encoded data to obtain verification data.

[0196] The protocol format can be a version of the protocol format and structure that is identifiable during data transmission and storage. It can include compression type, encryption type, and encoding type, that is, it contains information about how the data is compressed, encoded, and verified. The data header can guide the receiver on how to correctly decompress and decode the data. In addition to encoded data and protocol format, verification data can also include verification information, which verifies whether the data has been tampered with or corrupted during transmission.

[0197] This protocol implements version control and data verification by adding two bytes to the data header, thereby supporting compatibility between new and old versions and subsequent continuous optimization. Specifically, these two bytes are used to identify the data verification identifier and the compression encoding version number, respectively.

[0198] To ensure compatibility with different compression, encryption, and encoding types for the receiver, a 2-byte (16-bit) header can be added to the new compressed encoding data header as a data verification identifier and a type version number. This allows the receiver to verify the data based on the data verification identifier and determine the appropriate decoding, decryption, and decompression methods based on the type version number. Taking compatibility with different compression and encoding types as an example, as shown in Table 2 (compression encoding version table), the data verification identifier is located at the first position of the data header. It ensures data integrity by adding a Cyclic Redundancy Check (CRC). CRC check is a commonly used error detection method that can detect errors that may occur during data transmission. In this protocol, an 8-bit Hex is carried after the data header, which is the CRC-32 check result of the plaintext string for data verification. The protocol defines two different verification methods: SYN (0x16) indicates a normal protocol without CRC check; DC1 (0x11) indicates a verification protocol with CRC check. The protocol can flexibly adapt to different application scenarios, using the normal protocol in scenarios where data integrity requirements are not high, and the verification protocol in scenarios where data integrity needs to be ensured. The compression encoding version number is located as the second bit in the data header, and its value can range from 0x20 to 0x7F. This version number identifies the data compression encoding method, enabling the receiver to correctly parse the data based on the version number. The provided compression encoding version table lists three different data header types and their corresponding version numbers, encoding types, and compression types. If neither the data header nor the compression encoding version number is present, it indicates that the data is compressed and encoded using other methods, based on Base64 encoding and compressed using Zlib. A SYN data header indicates that no checksum is required, and a 0x20 compression encoding version number indicates that the data is encoded using Base122 and compressed using ZSTD-xx. A DC1 data header indicates that checksum is required, and a 0x20 compression encoding version number indicates that the data is encoded using Base122-CRC and compressed using ZSTD-xx. This not only supports multiple compression and encoding methods but also distinguishes different processing methods through version numbers, thus ensuring data compatibility and scalability.

[0199] Table 2 Compression Encoding Version Table

[0200] Data Header Compression encoding version number Encoding type Compression type none none Base64 Zlib SYN 0x20 Base122 ZSTD-xx DC1 0x20 Base122-CRC ZSTD-xx

[0201] Step 609: Categorize and store the verification data in the data storage cluster.

[0202] A data storage cluster can be a collection of data storage systems with shared resources and a shared management interface. By distributing data across multiple nodes, it achieves high reliability, high scalability, and high performance. Verification data can be categorized and stored based on version information, encoding type, compression type, and verification information contained in the data header. For example, data can be grouped into different groups or directories based on version number, distinguishing different versions of data and facilitating subsequent version management and traceability. When a specific version of data needs to be viewed, the corresponding storage location can be quickly located.

[0203] Another schematic diagram of the data processing in this disclosure embodiment can be referred to. Figure 7 As shown, transaction logs are compressed, encrypted, and encoded before being stored in the data storage cluster. Specifically, historical transaction logs 701 can be pre-acquired, and a ZSTD compression dictionary 702 can be generated or maintained based on these historical transaction logs 701 as training data. Next, transaction logs 703 to be processed can be obtained. These transaction logs typically contain sensitive information, which may pose security risks in an unencrypted state. Therefore, the encrypted data is restored to its original format 704. The decrypted data is then combined with the protocol field and two excessively long fields to obtain combined data 705. Subsequently, the combined data is compressed using the ZSTD compression algorithm 706. The ZSTD compression algorithm 706, combined with the ZSTD compression dictionary 702, effectively reduces the size of the combined data. The compressed data is then encrypted using the AES encryption algorithm 707, using the same key for both encryption and decryption to provide data protection. The encrypted data can then be encoded using the Base122 encoding algorithm 708. 122 different characters are used to convert the binary encrypted data into text format data. Then, an encoding and compression version number is added to the text data to obtain the final processed data 709. The encoding and compression version number allows tracing the encoding and compression algorithms used in the final processed data 709, ensuring that the corresponding algorithm can be used for data parsing. The final processed data 709 can be stored in the data storage cluster 710, where it can be organized and stored according to certain rules or strategies for easy retrieval. It is understandable that historical data related to transaction logs can also be extracted from the data storage cluster 710, restored to its original state, and then used as training data to generate or maintain the ZSTD compressed dictionary 702.

[0204] Description of apparatus and devices according to embodiments of this disclosure

[0205] It is understood that although the steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this embodiment, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0206] It should be noted that in the various specific embodiments of this disclosure, when processing is required based on data related to the characteristics of the target object, such as target object attribute information or a set of attribute information, the permission or consent of the target object will be obtained first. Furthermore, the collection, use, and processing of this data will comply with the relevant laws, regulations, and standards of the relevant regions. In addition, when this application embodiment needs to obtain target object attribute information, separate permission or consent from the target object will be obtained through pop-up windows or redirection to a confirmation page. Only after obtaining the target object's separate permission or consent will the necessary target object-related data for the normal operation of this application embodiment be obtained.

[0207] Figure 8 A schematic diagram of the structure of a data processing apparatus 800 provided in an embodiment of this disclosure. The apparatus 800 includes:

[0208] The first acquisition unit 801 is used to acquire resource transfer log data to be processed and a first compressed dictionary. The first compressed dictionary has a first version number and is stored in a first storage device.

[0209] The second acquisition unit 802 is used to acquire the first historical resource transfer log data corresponding to the resource transfer log data to be processed, and to evaluate and determine the target compression level based on the first historical resource transfer log data and the first compression dictionary.

[0210] Compression unit 803 is used to compress the resource transfer log data to be processed based on the first compression dictionary and the target compression level, and to generate first compressed data based on the first version number and the compressed data;

[0211] The sending unit 804 is used to encrypt the first compressed data and send the encrypted data to a second storage device for storage. The second storage device is different from the first storage device.

[0212] Optionally, in one feasible implementation, the transmitting unit 804 includes:

[0213] The encryption subunit is used to encrypt the first compressed data to obtain encrypted data;

[0214] The encoding subunit is used to encode the encrypted data to obtain encoded data;

[0215] The transmitting subunit is used to send the encoded data to the second storage device for storage.

[0216] Optionally, in one feasible implementation, the transmitting subunit includes:

[0217] Add a module to add version protocol data in a preset format to the encoded data to obtain format verification data;

[0218] The sending module is used to send the format verification data to the second storage device for storage.

[0219] Optionally, in one feasible implementation, the compression unit 803 includes:

[0220] The merging sub-unit is used to merge the protocol data and transfer object data in the resource transfer log data to be processed into combined data;

[0221] The first compression subunit is used to compress the combined data based on the first compression dictionary and the target compression level, and to generate the first compressed data based on the first version number and the compressed data.

[0222] Optionally, in one feasible implementation, the second acquisition unit 802 includes:

[0223] The first evaluation subunit is used to evaluate the compression effect of different compression levels based on the first historical resource transfer log data and the first compression dictionary, and obtain compression effect evaluation indicators corresponding to multiple compression levels.

[0224] The first determining subunit is used to determine the target compression level among multiple compression levels based on compression effect evaluation indicators.

[0225] Optionally, in one feasible implementation, the compression unit 803 includes:

[0226] The second compression subunit is used to compress the resource transfer log data to be processed based on the first compression dictionary and the target compression level, and to generate second compressed data based on the first version number and the compressed data.

[0227] The second evaluation subunit is used to evaluate the compression ratio of the second compressed data to obtain the target compression ratio.

[0228] The adjustment subunit is used to adjust the first compression dictionary when the target compression ratio is less than a preset threshold, so as to obtain the second compression dictionary and the second version number corresponding to the second compression dictionary;

[0229] The third compression subunit is used to compress the resource transfer log data to be processed based on the second compression dictionary and the target compression level, and to generate the first compressed data based on the second version number and the compressed data.

[0230] Optionally, in one feasible implementation, the adjustment subunit includes:

[0231] The determination module is used to identify duplicate data in the resource transfer log data to be processed.

[0232] The update module is used to update the first compressed dictionary based on duplicate data to obtain the second compressed dictionary and the second version number corresponding to the second compressed dictionary.

[0233] This disclosure also provides an apparatus for constructing a first compressed dictionary, including:

[0234] The third acquisition unit is used to acquire multiple second historical resource transfer log data within a preset time range;

[0235] The building unit is used to construct a first compression dictionary for compressing resource transfer log data based on multiple second historical resource transfer log data.

[0236] Optionally, in one feasible implementation, the building unit includes:

[0237] The second determining subunit is used to determine the protocol data format of multiple second historical resource transfer log data;

[0238] A sub-unit is constructed to build a first compression dictionary for compressing resource transfer log data based on multiple second historical resource transfer log data and protocol data formats.

[0239] This disclosure also provides an apparatus for processing encrypted data stored in a second storage device, including:

[0240] The decryption unit is used to decrypt the encrypted data to obtain the first version number and the first compressed data;

[0241] The query unit is used to initiate a compressed dictionary query request to the first storage device based on the first version number, and to receive the first compressed dictionary returned by the first storage device;

[0242] The decompression unit is used to decompress the first compressed data based on the first compression dictionary to obtain resource transfer log data.

[0243] This disclosure also provides an apparatus for processing a compressed dictionary query request, including: the compressed dictionary query request includes first permission information;

[0244] The permission verification unit is used to verify permissions based on the first permission information and obtain the permission verification result.

[0245] The dictionary feedback unit is used to send the first compressed dictionary to the second storage device based on the first version number when the permission verification result is successful.

[0246] Reference Figure 9 , Figure 9 To implement the structural block diagram of a portion of the terminal 140 of the data processing method according to an embodiment of this disclosure, the terminal 140 includes: a radio frequency (RF) circuit 910, a memory 915, an input unit 930, a display unit 940, a sensor 950, an audio circuit 960, a wireless fidelity (WiFi) module 970, a processor 980, and a power supply 990, etc. Those skilled in the art will understand that... Figure 9 The terminal 140 structure shown does not constitute a limitation on a mobile phone or computer, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0247] The RF circuit 910 can be used to receive and transmit signals during information transmission or calls. In particular, it receives downlink information from the base station and processes it with the processor 980; in addition, it transmits uplink data to the base station.

[0248] The memory 915 can be used to store software programs and modules. The processor 980 executes various terminal functions and document editing by running the software programs and modules stored in the memory 915.

[0249] The input unit 930 can be used to receive input numeric or character information, and to generate key signal inputs related to the terminal's settings and function control. Specifically, the input unit 930 may include a touch panel 931 and other input devices 932.

[0250] The display unit 940 can be used to display input or provided information, as well as various menus of the terminal. The display unit 940 may include a display panel 941.

[0251] Audio circuitry 960, speaker 961, and microphone 962 provide an audio interface.

[0252] In this embodiment, the processor 980 included in the terminal 140 can execute the data processing method of the previous embodiment.

[0253] The terminal 140 in this disclosure includes, but is not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc.

[0254] Figure 10 This is a partial structural block diagram of a server 110 for implementing the data processing method of this disclosure embodiment. The server 110 can vary significantly due to different configurations or performance, and may include one or more central processing units (CPUs) 1022 (e.g., one or more processors) and storage devices 1032, and one or more storage media 1030 (e.g., one or more mass storage devices) for storing application programs 1042 or data 1044. The storage devices 1032 and storage media 1030 can be temporary or persistent storage. The program stored in the storage media 1030 may include one or more modules (not shown in the figure), each module including a series of instruction operations on the server 110. Furthermore, the CPU 1022 may be configured to communicate with the storage media 1030 and execute the series of instruction operations in the storage media 1030 on the server 110.

[0255] Server 110 may also include one or more power supplies 1026, one or more wired or wireless network interfaces 1050, one or more input / output interfaces 1058, and / or one or more operating systems 1041, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0256] The central processing unit 1022 in server 110 can be used to execute the data processing method of the embodiments of this disclosure.

[0257] This disclosure also provides a storage medium for storing program code for executing the data processing methods of the foregoing embodiments.

[0258] This disclosure also provides a computer program product comprising a computer program. The processor of an electronic device reads and executes the computer program, causing the electronic device to perform the data processing method described above.

[0259] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in this disclosure and the foregoing drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “including,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatuses.

[0260] It should be understood that in this disclosure, "at least one item" means one or more, and "more than one" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0261] It should be understood that in the description of the embodiments disclosed herein, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0262] In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0263] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0264] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0265] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0266] It should also be understood that the various implementation methods provided in this disclosure can be combined arbitrarily to achieve different technical effects.

[0267] The above is a detailed description of the embodiments of this disclosure. However, this disclosure is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this disclosure. All such equivalent modifications or substitutions are included within the scope defined by the claims of this disclosure.

Claims

1. A data processing method, characterized in that, include: Obtain the resource transfer log data to be processed and the first compressed dictionary, the first compressed dictionary having a first version number, and the first compressed dictionary being stored in the first storage device; Obtain the first historical resource transfer log data corresponding to the resource transfer log data to be processed, and evaluate and determine the target compression level based on the first historical resource transfer log data and the first compression dictionary; The resource transfer log data to be processed is compressed based on the first compression dictionary and the target compression level, and first compressed data is generated based on the first version number and the compressed data. The first compressed data is encrypted, and the encrypted data is sent to a second storage device for storage. The second storage device is different from the first storage device.

2. The method according to claim 1, characterized in that, The construction process of the first compressed dictionary includes: Acquire multiple second historical resource transfer log data within a preset time range; A first compression dictionary is constructed based on the multiple second historical resource transfer log data to compress the resource transfer log data.

3. The method according to claim 2, characterized in that, The first compression dictionary for compressing the resource transfer log data based on the plurality of second historical resource transfer log data includes: Determine the protocol data format of the plurality of second historical resource transfer log data; A first compression dictionary for compressing resource transfer log data is constructed based on the plurality of second historical resource transfer log data and the protocol data format.

4. The method according to claim 1, characterized in that, The process of processing the encrypted data stored in the second storage device includes: The encrypted data is decrypted to obtain the first version number and the first compressed data; Based on the first version number, a compression dictionary query request is initiated to the first storage device, and the first compression dictionary returned by the first storage device is received; The first compressed data is decompressed based on the first compression dictionary to obtain the resource transfer log data.

5. The method according to claim 4, characterized in that, The compressed dictionary query request includes first permission information, and the processing procedure of the compressed dictionary query request by the first storage device includes: Based on the first permission information, perform permission verification to obtain the permission verification result; When the permission verification result is successful, the first compression dictionary is sent to the second storage device based on the first version number.

6. The method according to claim 1, characterized in that, The step of encrypting the first compressed data and sending the encrypted data to the second storage device for storage includes: The first compressed data is encrypted to obtain encrypted data; The encrypted data is encoded to obtain encoded data; The encoded data is sent to a second storage device for storage.

7. The method according to claim 6, characterized in that, The step of sending the encoded data to the second storage device for storage includes: Add version protocol data in a preset format to the encoded data to obtain format verification data; The format verification data is sent to a second storage device for storage.

8. The method according to claim 1, characterized in that, The step of compressing the resource transfer log data to be processed based on the first compression dictionary and the target compression level, and generating first compressed data based on the first version number and the compressed data, includes: The protocol data and transfer object data in the resource transfer log data to be processed are merged into combined data; The combined data is compressed based on the first compression dictionary and the target compression level, and first compressed data is generated based on the first version number and the compressed data.

9. The method according to claim 1, characterized in that, The step of determining the target compression level based on the first historical resource transfer log data and the first compression dictionary includes: Based on the first historical resource transfer log data and the first compression dictionary, the compression effect of different compression levels is evaluated to obtain compression effect evaluation indicators corresponding to multiple compression levels. The target compression level is determined from among the multiple compression levels based on the compression effect evaluation index.

10. The method according to claim 1, characterized in that, The step of compressing the resource transfer log data to be processed based on the first compression dictionary and the target compression level, and generating first compressed data based on the first version number and the compressed data, includes: The resource transfer log data to be processed is compressed based on the first compression dictionary and the target compression level, and second compressed data is generated based on the first version number and the compressed data. The compression ratio of the second compressed data is evaluated to obtain the target compression ratio; When the target compression ratio is less than a preset threshold, the first compression dictionary is adjusted to obtain a second compression dictionary and a second version number corresponding to the second compression dictionary; The resource transfer log data to be processed is compressed based on the second compression dictionary and the target compression level, and first compressed data is generated based on the second version number and the compressed data.

11. The method according to claim 10, characterized in that, The step of adjusting the first compressed dictionary to obtain a second compressed dictionary and a second version number corresponding to the second compressed dictionary includes: Identify duplicate data in the resource transfer log data to be processed; The first compressed dictionary is updated based on the duplicate data to obtain a second compressed dictionary and a second version number corresponding to the second compressed dictionary.

12. A data processing apparatus, characterized in that, include: The first acquisition unit is used to acquire resource transfer log data to be processed and a first compressed dictionary. The first compressed dictionary has a first version number and is stored in a first storage device. The second acquisition unit is used to acquire the first historical resource transfer log data corresponding to the resource transfer log data to be processed, and to evaluate and determine the target compression level based on the first historical resource transfer log data and the first compression dictionary. A compression unit is used to compress the resource transfer log data to be processed based on the first compression dictionary and the target compression level, and to generate first compressed data based on the first version number and the compressed data. The sending unit is used to encrypt the first compressed data and send the encrypted data to a second storage device for storage, the second storage device being different from the first storage device.

13. A storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the data processing method according to any one of claims 1 to 11.

14. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the data processing method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program that is read and executed by a processor of an electronic device, causing the electronic device to perform the data processing method according to any one of claims 1 to 11.

Images