A data processing method, related device and storage medium

Abstract

Embodiments of the present application disclose a data processing method, related devices and a storage medium. The present application comprises: determining a target compression algorithm corresponding to to-be-compressed data; if a compression algorithm currently applied by a first processing module does not belong to the target compression algorithm, using a virtual machine to write the target compression algorithm to the first processing module; locking the target compression algorithm as a compression algorithm applied by the first processing module; using the virtual machine to input the to-be-compressed data to the first processing module; and using the target compression algorithm to compress the to-be-compressed data in the first processing module. Since the virtual machine shields the difference between different operation platforms and provides a unified communication interface, the virtual machine is used to write the compression algorithm and transmit the to-be-compressed data to the first processing module, so that it is not necessary to write the compression algorithm of the first processing module for each application and it is not necessary to write related code for transmission of the to-be-compressed data, thereby improving the efficiency of compressing data by using the first processing module and bringing convenience for file storage.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, related apparatus, and storage medium. Background Technology

[0002] With the rapid development of big data, massive amounts of data are growing explosively, posing enormous challenges to data storage and management. To improve storage space utilization, data compression technology is an essential and crucial technology in storage systems.

[0003] To reduce the computational burden on the central processing unit (CPU), FPGAs can be used for data compression. However, when a Java application uses an FPGA to compress data, the user needs to write the compression algorithm for the FPGA and then install it into the FPGA hardware. Simultaneously, related code needs to be written to transmit the data to be compressed to the FPGA for compression. The entire process of using FPGAs for data compression is cumbersome and inefficient.

[0004] Therefore, an efficient and convenient data compression solution is urgently needed. Summary of the Invention

[0005] In view of this, this application provides a data processing method, related apparatus, and storage medium for improving the efficiency of data compression.

[0006] This application provides a data processing method, including:

[0007] Obtain the data to be compressed;

[0008] Determine the target compression algorithm required for the data to be compressed;

[0009] Determine whether the compression algorithm currently applied by the first processing module is the target compression algorithm;

[0010] If not, the virtual machine writes the target compression algorithm to the first processing module;

[0011] Determine whether the compression algorithm used by the first processing module has been successfully switched to the target compression algorithm;

[0012] If so, the target compression algorithm will be locked to the compression algorithm applied by the first processing module;

[0013] The virtual machine is used to input the data to be compressed into the first processing module;

[0014] In the first processing module, the target compression algorithm is used to compress the data to be compressed.

[0015] Another aspect of this application provides a data processing apparatus, comprising:

[0016] The acquisition unit is used to acquire the data to be compressed;

[0017] The determining unit is used to determine the target compression algorithm required for the data to be compressed.

[0018] The judgment unit is used to determine whether the compression algorithm currently applied by the first processing module is the target compression algorithm;

[0019] The writing unit is used to write the target compression algorithm to the first processing module using a virtual machine when the compression algorithm currently applied by the first processing module is the target compression algorithm.

[0020] The judgment unit is also used to determine whether the compression algorithm applied by the first processing module has been successfully switched to the target compression algorithm;

[0021] A locking unit is used to lock the target compression algorithm to the compression algorithm applied by the first processing module;

[0022] An input unit is used to input data to be compressed into the first processing module using a virtual machine;

[0023] The compression unit is used in the first processing module to compress the data to be compressed using the target compression algorithm.

[0024] In one possible design, in another implementation of the embodiments of this application,

[0025] The input unit is also used to input the data to be compressed to the first processing module using a virtual machine when the compression algorithm currently applied by the first processing module is the target compression algorithm;

[0026] The compression unit is also used in the first processing module to compress the data to be compressed using the target compression algorithm.

[0027] In one possible design, in another implementation of the embodiments of this application, the data processing apparatus further includes a configuration unit.

[0028] A configuration unit is used to configure N compression algorithms for a virtual machine, where N is an integer greater than or equal to 1;

[0029] The determining unit is specifically used to receive compression instructions for the data to be compressed, which specify the target compression algorithm required for the data to be compressed.

[0030] The target compression algorithm is determined from N compression algorithms available in the virtual machine.

[0031] In one possible design, in another implementation of the embodiments of this application,

[0032] The configuration unit is also used to configure N compression algorithms for the virtual machine, where N is an integer greater than or equal to 1;

[0033] The determining unit is also used to determine the default compression algorithm from N compression algorithms;

[0034] The determining unit is specifically used to determine the default compression algorithm as the target compression algorithm required for the data to be compressed.

[0035] In one possible design, in another implementation of the embodiments of this application,

[0036] The determining unit is also used to determine the amount of data to be compressed;

[0037] The judgment unit is also used to determine whether the amount of data exceeds the first preset threshold.

[0038] The triggering unit is also used to trigger the determining unit when the amount of data reaches a first preset threshold;

[0039] The input unit is also used to transmit the data to be compressed to the second processing module when the amount of data does not exceed the first preset threshold.

[0040] The compression unit is also used to compress the data to be compressed using the second processing module.

[0041] In one possible design, in another implementation of the embodiments of this application, the first processing module includes multiple threads.

[0042] The compression unit is specifically used to write the data to be compressed into the first processing queue of the first processing module.

[0043] Determine if there is an idle thread among multiple threads;

[0044] When there is an idle thread among multiple threads, the data to be compressed is obtained from the first processing queue;

[0045] In the idle thread, the target compression algorithm is used to compress the data to be compressed;

[0046] The compression unit is also used to compress the data to be compressed by the second processing module when there are no idle threads among the multiple threads.

[0047] In one possible design, in another implementation of the embodiments of this application,

[0048] The determining unit is also used to determine the number of tasks in the waiting queue of multiple threads when there are no idle threads among the multiple threads, where the waiting queue represents the queue of tasks to be processed in multiple threads.

[0049] The judgment unit is also used to determine whether the number of tasks exceeds a second preset threshold;

[0050] The compression unit is also used to compress the data to be compressed by the second processing module when the number of tasks exceeds the second preset threshold.

[0051] In one possible design, in another implementation of the embodiments of this application, the data processing device further includes a preemption unit.

[0052] The determining unit is also used to determine the first priority corresponding to the first processing queue when there are no idle threads among multiple threads;

[0053] The acquisition unit is also used to acquire the second processing queue of multiple threads, wherein the second processing queue is the processing queue that is currently being executed;

[0054] The determining unit is also used to determine the second priority corresponding to the second processing queue;

[0055] The preemption unit is used to preempt the target thread occupied by the second processing queue when the first priority is higher than the second priority;

[0056] The compression unit is also used to compress the data to be compressed in the first processing queue using the target compression algorithm in the target thread.

[0057] In one possible design, in another implementation of the embodiments of this application, the data processing device further includes an output unit.

[0058] The output unit is used to output the compressed data to be compressed, thus obtaining the target data.

[0059] The determination unit is used to determine, based on the target data, whether there is a new idle thread in the first processing module.

[0060] In one possible design, in another implementation of the embodiments of this application, the data processing apparatus further includes a parsing unit.

[0061] The acquisition unit is also used to acquire the file to be compressed;

[0062] The parsing unit is used to parse the file to be compressed into a byte array;

[0063] The unit of determination is also used to determine if a byte array is data to be compressed.

[0064] This application also provides a computer device, including: a memory, a processor, and a bus system; the memory is used to store program code; the processor is used to execute the data processing method described in any of the preceding aspects according to instructions in the program code.

[0065] In another aspect, this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the data processing method described in any of the preceding aspects.

[0066] According to another aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data processing method described in any of the preceding aspects.

[0067] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0068] This application provides a data processing method. After obtaining the data to be compressed, a target compression algorithm is determined. If the compression algorithm currently applied by the first processing module is not the target compression algorithm, a virtual machine writes the target compression algorithm to the first processing module and locks it as the compression algorithm applied by the first processing module. This ensures that the compression algorithm applied by the first processing module matches the target compression algorithm required by the data to be compressed. The virtual machine inputs the data to be compressed to the first processing module, and the first processing module compresses the data using the target compression algorithm. Through this method, during the compression process, the virtual machine shields the differences between different operating platforms, providing a unified communication interface for different operating platforms. Therefore, by writing the compression algorithm and transmitting the data to be compressed to the first processing module through the virtual machine, it is unnecessary to write the compression algorithm for each application or write related code for the first processing module. This improves the efficiency of data compression using the first processing module and facilitates file storage. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0070] Figure 1 A network architecture diagram for the operation of a data compression system;

[0071] Figure 2 A schematic diagram illustrating an application scenario of the data processing method provided in this application;

[0072] Figure 3 This is a flowchart of a data processing method in an embodiment of this application;

[0073] Figure 4 This is another flowchart of the data processing method in the embodiments of this application;

[0074] Figure 5A This is another flowchart of the data processing method in the embodiments of this application;

[0075] Figure 5B This is another flowchart of the data processing method in the embodiments of this application;

[0076] Figure 6 This is a flowchart illustrating data compression based on the first priority of the first processing queue in an embodiment of this application.

[0077] Figure 7 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0078] Figure 8 This is a schematic diagram of the structure of a computer device in an embodiment of this application. Detailed Implementation

[0079] This application provides a data processing method, related apparatus, and storage medium to improve the efficiency of data compression using FPGA.

[0080] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “corresponding to,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0081] First, some terms that may appear in the embodiments of this application will be explained.

[0082] Data compression: In computer science and information theory, data compression is the process of representing information using fewer bits (or other information-related units) than unencoded data according to a specific encoding mechanism. It aims to reduce the amount of data to decrease storage space and improve transmission, storage, and processing efficiency without losing useful information, or to reorganize data according to certain algorithms to reduce redundancy and storage space.

[0083] Field programmable gate array (FPGA): It emerged as a semi-custom circuit in the field of application-specific integrated circuit (ASIC), which not only solves the shortcomings of custom circuits, but also overcomes the limitation of the limited number of gate circuits in the original programmable devices.

[0084] Virtual machine (VM): A virtual machine is a complete computer system simulated by software, possessing full hardware system functionality and running in a completely isolated environment. Any task that can be performed on a physical computer can also be performed in a VM. When creating a VM, a portion of the physical machine's hard drive and memory capacity is used as the VM's hard drive and memory capacity. Each VM has its own independent complementary metal-oxide-semiconductor (CMOS) hard drive and operating system, and can be operated just like a physical machine.

[0085] It should be understood that the data processing method provided in this application can be applied to systems or programs in terminal devices that include data compression functionality, such as video compression software or audio compression software. Specifically, the data compression system can run on systems such as... Figure 1 In the network architecture shown, such as Figure 1 The diagram shown is a network architecture diagram of the data compression system. It can be understood that... Figure 1 The diagram shows various terminal devices, which can be computer devices. In real-world scenarios, more or fewer types of terminal devices may participate in the data compression process. The specific number and types depend on the actual scenario and are not limited here. Figure 1 The example shows one server, but in real-world scenarios, multiple servers can be involved, especially in scenarios involving multi-model training and interaction. The specific number of servers depends on the actual scenario.

[0086] In this embodiment, the server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication. The terminal and server can be connected to form a blockchain network; this application does not impose any restrictions.

[0087] Specifically, cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to realize the computing, storage, processing, and sharing of data.

[0088] Cloud technology is a collective term for network technology, information technology, integration technology, management platform technology, and application technology applied to the cloud computing business model. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will all require robust system support, which can only be achieved through cloud computing.

[0089] It should be understood that the data processing method provided in the embodiments of this application can be applied to... Figure 1 The server shown can also be applied to Figure 1 The terminal device shown. After receiving the data to be compressed from the terminal device, the server can compress the data using the data processing method provided in the embodiments of this application; alternatively, the terminal device can compress the data using the data processing method provided in the embodiments of this application after receiving the data to be compressed from the server; or the server or terminal device can compress the local data stored on its own device according to the user's needs, thereby reducing the occupation of storage resources, and the specifics are not limited here.

[0090] The data processing method provided in this application can improve the efficiency of data compression using the first processing module. The first processing module can be a programmable processor, such as an FPGA, an ARM (Advanced RISC Machines) processor, or a digital signal processor; specific limitations are not specified here. On the other hand, the second processing module can include various possible processor types, including but not limited to a central processing unit (CPU), microprocessor, microcontroller, artificial intelligence processor, or heterogeneous processor. For ease of understanding, this application uses an FPGA as the first processing module and a CPU as the second processing module as an example for illustration.

[0091] For easier understanding, please refer to Figure 2 , Figure 2 This is a schematic diagram illustrating an application scenario of the data processing method provided in this application. Figure 2 As shown, with the rapid development of big data, massive amounts of data are growing explosively, posing enormous challenges to data storage and management. To improve storage space utilization and conserve storage resources on computer devices, data compression is often necessary. Traditional data compression typically uses the CPU. However, CPU-based compression methods often consume significant computing power and are slow. With the rapid development of FPGAs, heterogeneous computing combining CPUs and FPGAs is currently one of the main methods to improve computing speed.

[0092] In existing FPGA-based data compression solutions, for each different application (e.g., a Java application), the user needs to write the FPGA compression algorithm, then install it into the FPGA hardware, and also write related code to transmit the data to be compressed to the FPGA for compression. This FPGA-based data compression process is cumbersome and inefficient. Figure 2 As shown, in computer devices, Java applications can use the Java Virtual Machine to transmit data to be compressed to the FPGA or CPU for compression. Figure 2The compression API module shown in this application allows users to start the virtual machine without modifying the code, simply by calling the relevant parameters. Therefore, it provides FPGA compression capabilities for existing compression methods in Java applications while retaining the original CPU compression capabilities by default. The scheduling module can classify received compression tasks according to data size, thus determining whether to use FPGA or CPU compression; it can also determine the choice based on the waiting queue status within the FPGA.

[0093] Since virtual machines shield the differences between different operating platforms and provide a unified communication interface, writing compression algorithms and transmitting data to be compressed to the FPGA through virtual machines eliminates the need to write compression algorithms for each application or write related code. This allows the data to be compressed to be transmitted to the FPGA for compression, improving the efficiency of data compression using FPGA.

[0094] On the other hand, in existing FPGA-based data compression solutions, even if users write code and compression algorithms for transmitting data to the FPGA, these codes and algorithms often become inapplicable when ported to different operating systems (such as Windows or Linux). In such cases, users need to re-maintain the aforementioned code and algorithms. However, virtual machines, by shielding the differences between operating platforms and providing a unified communication interface, eliminate the need to modify the relevant transmission code and compression algorithms for different operating systems, greatly simplifying the FPGA-based data compression process.

[0095] It should be understood that the data processing method provided in the embodiments of this application is not limited to the JAVA programming language, but can also be applied to other types of managed languages, such as programs written in C#, which can use the Common Language Runtime (CLR) as a virtual machine to write compression algorithms and transmit data to be compressed to the FPGA. The specific application is not limited here.

[0096] Based on the above introduction, the data processing methods in this application will be described below. Please refer to [link / reference]. Figure 3 , Figure 3 A flowchart of a data processing method provided in an embodiment of this application is shown below. Figure 3 As shown, one embodiment of the data processing method in this application includes:

[0097] 101. Obtain the data to be compressed;

[0098] In this embodiment, the acquired data to be compressed should be input to the virtual machine in a byte array format. Specifically, in computer programming, a continuous sequence of variables of data type bytes is called a byte array. Arrays are one of the most basic data structures, and bytes are the smallest standard scalar type in most programming languages. Byte arrays are invaluable when reading files stored in unknown or arbitrary binary formats, or when it is necessary to efficiently store large amounts of data to save memory. There are also instances where byte arrays can be used to store string data to help reduce memory usage. Using byte arrays allows for several optimizations, making accessing and modifying information in the array faster than using other types.

[0099] The data processing method provided in this application embodiment can be applied to the compression of files of various data types, including but not limited to video files, audio files, text, or images, etc., without specific limitations here. Taking a JAVA application as an example, after obtaining a file of the above data types, the JAVA application will parse it into a byte array format as the data to be compressed. The data to be compressed will then be input into the virtual machine, and step 102 will be executed.

[0100] 102. Determine the target compression algorithm required for the data to be compressed;

[0101] Compression algorithm: A compression algorithm is an algorithm for compressing data. In the fields of electronics and communications, it is also often referred to as signal encoding. It includes two steps: compression and decompression (decoding and encoding).

[0102] To adapt to the needs of different operating systems, the virtual machine comes pre-loaded with multiple compression algorithms, such as various types of entropy coding or source coding. Generally, the virtual machine selects one of these compression algorithms as the default and uses that default algorithm to compress the data to be compressed. In other words, the current virtual machine's default compression algorithm is directly used as the target compression algorithm required for the data to be compressed in this application.

[0103] In practical applications, users can also select the compression algorithm (i.e., the target compression algorithm) to be used for the data to be compressed based on their own needs (such as the user's requirements for compression ratio or compression format).

[0104] 103. Determine whether the compression algorithm currently applied by the first processing module is the target compression algorithm;

[0105] In the process of compressing data using the first processing module, the corresponding compression algorithm (i.e., the target compression algorithm) needs to be written into the first processing module first, and then the first processing module can use the compression algorithm to compress the data.

[0106] In practical applications, the compression algorithm used by the first processing module may vary. Taking an FPGA as the first processing module as an example, the FPGA often has a compression algorithm pre-written into it for other tasks. Therefore, it's necessary to first determine whether the target compression algorithm used to compress the data to be compressed is consistent with the algorithm currently used by the FPGA. If they are inconsistent, the compression algorithm used by the FPGA needs to be rewritten to the target compression algorithm. Alternatively, if the first processing module has just undergone initialization and does not contain any compression algorithm, then it is also considered that the compression algorithm currently used by the first processing module is inconsistent with the required target compression algorithm. If yes, then step 107 is executed: input the data to be compressed into the first processing module using a virtual machine; if no, then step 104 is executed: write the target compression algorithm into the first processing module using a virtual machine.

[0107] 104. If not, then the virtual machine is used to write the target compression algorithm to the first processing module;

[0108] As mentioned in step 103, in practical applications, the first processing module often already has a compression algorithm written in during the task processing process, due to the need to execute other tasks previously. Therefore, after step 103, which determines whether the compression algorithm currently applied by the first processing module is the target compression algorithm, if the compression algorithm currently applied by the first processing module happens to be the target compression algorithm, in order to improve processing efficiency and reduce processing steps, it is not necessary to write the target compression algorithm again; the target compression algorithm currently applied by the first processing module can be used directly.

[0109] If the compression algorithm currently used by the first processing module is not the target compression algorithm, that is, the compression algorithm currently used by the first processing module is inconsistent with the required target compression algorithm, then the target compression algorithm needs to be written into the first processing module to update the algorithm state of the first processing module so that the algorithm used by the first processing module is the target compression algorithm.

[0110] For virtual machines, their standard library often has multiple pre-built compression algorithms, so users do not need to manually implement compression algorithms. They can simply use the virtual machine and write the target compression algorithm from its standard library into the first processing module.

[0111] 105. Determine whether the compression algorithm used by the first processing module has been successfully switched to the target compression algorithm;

[0112] In practical applications, the writing of the target compression algorithm cannot be guaranteed to be 100% successful. For example, the first processing module may not be compatible with the target compression algorithm being written, or other high-priority tasks may forcibly tamper with the compression algorithm applied by the first processing module again, which may cause the writing of the target compression algorithm to fail. In addition to the two situations mentioned above, there may be other situations that may cause the writing of the target compression algorithm to fail, which will not be elaborated here.

[0113] Therefore, before inputting the data to be compressed into the first processing module, it is first determined whether the compression algorithm used by the first processing module has been successfully switched to the target compression algorithm. If yes, it means that the target compression algorithm has been successfully written, and step 106 is executed: the target compression algorithm is locked to the compression algorithm used by the first processing module, thereby preventing the data to be compressed from continuing to be compressed even if the target compression algorithm fails to be written, thus improving the stability and reliability of the scheme. If no, it means that the target compression algorithm has failed to be written, and step 109 is executed: the second processing module is used to compress the data to be compressed.

[0114] 106. If so, then the target compression algorithm will be locked to the compression algorithm applied by the first processing module;

[0115] If the compression algorithm used by the first processing module has been successfully switched to the target compression algorithm, then the writing of the target compression algorithm can be considered successful. Furthermore, to prevent other tasks of the first processing module from forcibly tampering with the compression algorithm during data compression, this embodiment can utilize a state lock to lock the target compression algorithm to the one used by the first processing module, thereby ensuring the stability of the compression algorithm. Then, a virtual machine can be used to input the data to be compressed into the first processing module, thereby executing the subsequent data compression process.

[0116] 107. Input the data to be compressed into the first processing module using a virtual machine;

[0117] After the above steps, the algorithm currently applied by the first processing module has been determined to be the target compression algorithm required for the data to be compressed. Therefore, the first processing module can be used to compress the data, and it is necessary to transmit the data to be compressed to the first processing module. Since the virtual machine shields the differences between different operating platforms and provides a unified communication interface, in this embodiment, the virtual machine is used to directly transmit the data to be compressed to the first processing module, thus eliminating the need to write code to transmit the data to the first processing module.

[0118] 108. In the first processing module, the target compression algorithm is used to compress the data to be compressed;

[0119] After the above steps, the first processing module has obtained the data to be compressed and the target compression algorithm to be used. At this point, the first processing module can use the target compression algorithm to compress the data to be compressed.

[0120] 109. The second processing module is used to compress the data to be compressed;

[0121] In this embodiment of the application, if the target compression algorithm fails to be written, that is, the compression algorithm applied by the first processing module cannot be rewritten to the target compression algorithm, in order to continue to compress the data to be compressed, the task of using the first processing module to perform data compression can be abandoned, and the second processing module can be used to compress the data to be compressed instead.

[0122] This application provides a data processing method. After obtaining the data to be compressed, a target compression algorithm is determined. If the compression algorithm currently applied by the first processing module is not the target compression algorithm, a virtual machine writes the target compression algorithm to the first processing module and locks it as the compression algorithm applied by the first processing module. This ensures that the compression algorithm applied by the first processing module matches the target compression algorithm required by the data to be compressed. The virtual machine inputs the data to be compressed to the first processing module, and the first processing module compresses the data using the target compression algorithm. Through this method, during the compression process, the virtual machine shields the differences between different operating platforms, providing a unified communication interface for different operating platforms. Therefore, by writing the compression algorithm and transmitting the data to be compressed to the first processing module through the virtual machine, it is unnecessary to write the compression algorithm for each application or write related code for the first processing module. This improves the efficiency of data compression using the first processing module and facilitates file storage.

[0123] On the other hand, taking an FPGA as the first processing module and a CPU as the second processing module as an example, the data processing method provided in this application, in a compression test of a 1GB file, compared with the traditional scheme using a CPU for data compression, increases the speed by 8 times and reduces CPU utilization by 2 times when compressing a 1GB file. This invention can greatly improve the throughput of the application. Similarly, the decrease in CPU utilization also leads to a decrease in power consumption. Since the power consumption of an FPGA is much lower than that of a CPU, the decrease in CPU utilization also leads to a decrease in the overall power consumption of the device.

[0124] On the other hand, in this embodiment, the compression algorithm is written to the FPGA and the data to be compressed is transmitted through a virtual machine, so that the parameters of the original application (e.g., JAVA application) do not need to be modified, thus ensuring the compatibility and stability of the solution.

[0125] It should be understood that the data processing method provided in this application can be applied to cloud servers configured with a first processing module and a second processing module (e.g., FPGA and CPU). When a cloud server configured with a first processing module and a second processing module needs to compress local data or other received data, the data processing method provided in this application can be used. Cloud servers can possess powerful computing capabilities, enabling them to perform computationally demanding tasks for terminal devices linked to them. For example, ... Figure 1 The server shown can be a cloud service that receives data to be compressed from multiple terminal devices. In this case, the amount of data to be compressed from multiple terminal devices is often quite large. The cloud server can then utilize its powerful computing capabilities to use the data to be compressed to execute the data processing method provided in this application.

[0126] Cloud computing is a computing model that distributes computing tasks across a large pool of computers, enabling various application systems to access computing power, storage space, and information services as needed. The network providing these resources is called the "cloud." From the user's perspective, resources in the "cloud" appear infinitely scalable, readily available, on-demand, and expandable, with payment based on usage.

[0127] As a provider of fundamental cloud computing capabilities, a cloud resource pool (referred to as a cloud platform, generally called an IaaS (Infrastructure as a Service) platform) is established. Various types of virtual resources are deployed in the resource pool for external customers to choose from. The cloud resource pool mainly includes: computing devices (virtualized machines containing operating systems), storage devices, and network devices.

[0128] Based on logical function, a PaaS (Platform as a Service) layer can be deployed on top of the IaaS (Infrastructure as a Service) layer, and a SaaS (Software as a Service) layer can be deployed on top of the PaaS layer. Alternatively, SaaS can be deployed directly on top of IaaS. PaaS is a platform for running software, such as databases and web containers. SaaS refers to various types of business software, such as web portals and bulk SMS senders. Generally speaking, SaaS and PaaS are upper layers compared to IaaS.

[0129] Optionally, in the above Figure 3 Based on the corresponding embodiments, in another optional embodiment of the data processing method provided in this application, the following steps may also be included:

[0130] If the compression algorithm currently applied by the first processing module is the target compression algorithm, then the virtual machine is used to input the data to be compressed into the first processing module;

[0131] In the first processing module, the target compression algorithm is used to compress the data to be compressed.

[0132] In this application, during the data compression process, it is necessary to ensure that the compression algorithm applied by the first processing module is consistent with the target compression algorithm required by the data to be compressed. As can be seen from step 103 above, if the compression algorithm currently applied by the first processing module is inconsistent with the required target compression algorithm, the target compression algorithm needs to be written into the first processing module to update the algorithm state of the first processing module, so that the algorithm applied by the first processing module is the target compression algorithm.

[0133] For easier understanding, please refer to Figure 4 , Figure 4 This is another flowchart of the data processing method in an embodiment of this application. For example... Figure 4 As shown, in this embodiment, after steps 201 to 202 (see the description of steps 101 to 102 above), if the compression algorithm currently applied by the first processing module is consistent with the required target compression algorithm, then there is no need to write the target compression algorithm to the first processing module again; that is, step 203 is not required, and step 204 is executed directly. On the other hand, after... Figure 3 Steps 101 to 105, as shown, represent the data compression process of the data to be compressed. If the first processing module is needed to compress other data in the future, and the target compression algorithm required for the subsequent data to be compressed has not changed, i.e. it is the same as the previous target compression algorithm, then there is no need to rewrite the algorithm state of the first processing module. That is, there is no need to execute step 203, and step 204 can be executed directly.

[0134] For example, if the first processing module has already used the target compression algorithm to perform other data compression tasks before processing the compression task of the data to be compressed, then when the first processing module needs to compress the data, it does not need to update the algorithm state of the first processing module. It can directly use the target compression algorithm currently applied by the first processing module to compress the data. That is, when the compression algorithm currently applied by the first processing module is the target compression algorithm, there is no need to execute the step of "writing the target compression algorithm to the first processing module using a virtual machine". Step 204 can be executed directly, thereby improving the execution efficiency of the data compression process.

[0135] For example, if the default compression algorithm in the factory settings of the first processing module is exactly the target compression algorithm required by the data to be compressed, and the algorithm status of the first processing module has not been modified after leaving the factory, then there is no need to update the algorithm status of the first processing module. The target compression algorithm currently applied by the first processing module can be used directly to compress the data to be compressed.

[0136] Optionally, in the above Figure 3 Based on the corresponding embodiments, in another optional embodiment of the data processing method provided in this application, before obtaining the data to be compressed, the following steps may be included:

[0137] Configure N compression algorithms for the virtual machine, where N is an integer greater than or equal to 1;

[0138] The target compression algorithm needed to determine the data to be compressed includes:

[0139] Receive compression instructions for the data to be compressed, which specify the target compression algorithm required for the data to be compressed;

[0140] The target compression algorithm is determined from N compression algorithms available in the virtual machine.

[0141] In this embodiment, to adapt to the needs of different operating systems and users' different requirements for data compression encoding methods, N compression algorithms can be pre-configured in the virtual machine, where N is an integer greater than or equal to 1. For example, the pre-configured N compression algorithms may include run-length encoding (RLE), lempel-ziv-welch encoding (LZW), Shannon coding, Huffman coding, or rrithmetic coding in entropy coding; they may also include fast fourier transform (FFT) coding or subband coding (SBC) in source coding; or other compression algorithms besides those described above. Specific limitations are not specified here.

[0142] After obtaining the data to be compressed, the target compression algorithm to be used must first be determined. Users can also select the desired compression algorithm (i.e., the target compression algorithm) from a set of N preset compression algorithms based on their specific needs (such as requirements for compression ratio or compression format). Specifically, upon receiving a compression command specifying the target compression algorithm, the target compression algorithm can be determined from the virtual machine's N compression algorithms.

[0143] In this embodiment, by pre-setting multiple compression algorithms in the virtual machine, the solution can adapt to the needs of different operating systems and the different encoding requirements of users for data compression, thereby improving the flexibility of the solution.

[0144] Optionally, in the above Figure 3 Based on the corresponding embodiments, in another optional embodiment of the data processing method provided in this application, before obtaining the data to be compressed, the following steps may be included:

[0145] Configure N compression algorithms for the virtual machine, where N is an integer greater than or equal to 1;

[0146] Determine the default compression algorithm from N compression algorithms;

[0147] The target compression algorithm needed to determine the data to be compressed includes:

[0148] The default compression algorithm is determined to be the target compression algorithm required for the data to be compressed.

[0149] In this embodiment, to adapt to the needs of different operating systems and users' different requirements for data compression encoding methods, N compression algorithms can be pre-configured in the virtual machine, and one of them can be selected as the default compression algorithm. Therefore, when users do not have specific requirements for compression algorithms, the default compression algorithm can be used as the target compression algorithm for subsequent data compression processes.

[0150] In this embodiment, by pre-setting multiple compression algorithms in the virtual machine, the solution can adapt to the needs of different operating systems and the different encoding requirements of users for data compression, thereby improving the flexibility of the solution.

[0151] Optionally, in the above Figure 3 Based on the corresponding embodiments, in another optional embodiment of the data processing method provided in this application, after obtaining the data to be compressed, the following steps may be included:

[0152] Determine the amount of data to be compressed;

[0153] Determine whether the data volume exceeds the first preset threshold;

[0154] If the amount of data exceeds the first preset threshold, the step of determining the target compression algorithm required for the data to be compressed is triggered.

[0155] Depend on Figure 3 As can be seen from the embodiments shown, in the process of data compression using the first processing module, before performing the data compression task, corresponding preparatory work needs to be performed (such as copying and transmitting the data to be compressed and rewriting the compression algorithm of the first processing module). Therefore, when the amount of data to be compressed is small, the above-mentioned preparatory work cannot be avoided when using the first processing module for data compression, which brings unnecessary system overhead.

[0156] In this embodiment, a first preset threshold can be configured based on the actual needs and the amount of data to be compressed. After acquiring the data to be compressed, the amount of data to be compressed is first determined. It is then determined whether the amount of data exceeds the first preset threshold. If the amount of data exceeds the first preset threshold, the step of determining the target compression algorithm required for the data to be compressed is triggered, thereby executing the subsequent data compression process. If the amount of data does not exceed the first preset threshold, the first processing module is not used to compress the data to be compressed.

[0157] For example, taking an FPGA as the first processing module and a CPU as the second processing module, assume that the data size of the data to be compressed, as defined by the first preset threshold, is 2 gigabytes (GB). If, after obtaining the data to be compressed, it is determined that the data size is 1.5GB, since 1.5GB is less than 2GB (the first preset threshold), the data size does not exceed the first preset threshold, and the FPGA is not used to compress the data. If, after obtaining the data to be compressed, it is determined that the data size is 2.5GB, since 2.5GB is greater than 2GB (the first preset threshold), the data size exceeds the first preset threshold, and the FPGA can be used to compress the data.

[0158] In this embodiment, the amount of data to be compressed is limited by the above method, which avoids the situation where the data to be compressed is too small and is still compressed by the first processing module, thereby saving system overhead.

[0159] Optionally, in the above Figure 3 Based on the corresponding embodiments, in another optional embodiment of the data processing method provided in this application, the following steps may also be included:

[0160] If the amount of data does not exceed the first preset threshold, the data to be compressed is transmitted to the second processing module.

[0161] The second processing module is used to compress the data to be compressed.

[0162] When using the first processing module for data compression, it is inevitable to perform operations such as judgment or rewriting of the compression algorithm of the first processing module. If the amount of data to be compressed is too small, the efficiency obtained by using the first processing module for compression is not high. Therefore, in this embodiment, if the amount of data to be compressed does not exceed a first preset threshold, the data to be compressed can be transmitted to the second processing module, which will then compress the data. This avoids the situation where the first processing module is still used to compress data with an excessively small amount of data, thereby saving system overhead.

[0163] Optionally, in the above Figure 3 Based on the corresponding embodiments, in another optional embodiment of the data processing method provided in this application, the first processing module includes multiple threads. In the first processing module, a target compression algorithm is used to compress the data to be compressed, specifically including the following steps:

[0164] Write the data to be compressed into the first processing queue of the first processing module;

[0165] Determine if there is an idle thread among multiple threads;

[0166] If it exists, retrieve the data to be compressed from the first processing queue;

[0167] In the idle thread, the target compression algorithm is used to compress the data to be compressed.

[0168] In this embodiment, taking advantage of the fact that the first processing module includes multiple threads, after the data to be compressed is input into the first processing module using a virtual machine, the data to be compressed is written into the first processing queue of the first processing module. It is then determined whether there is an idle thread among the multiple threads. If an idle thread exists, the data to be compressed can be retrieved from the first processing queue, and the target compression algorithm is used to compress the data in that idle thread.

[0169] Optionally, in the above Figure 3 Based on the corresponding embodiments, in another optional embodiment of the data processing method provided in this application, the following steps may also be included:

[0170] If no idle thread exists among the multiple threads, the second processing module is used to compress the data to be compressed.

[0171] In this embodiment, if all threads of the first processing module have ongoing task flows, meaning there are no idle threads among the threads of the first processing module, then the first processing module can be abandoned for data compression, and the second processing module can be used to compress the data to be compressed. This allows for timely processing of the data compression task, avoiding excessively long task waiting times and improving the efficiency of the solution.

[0172] Optionally, in the above Figure 3 Based on the corresponding embodiments, in another optional embodiment of the data processing method provided in this application, after compressing the data to be compressed using the target compression algorithm, the method may further include the following steps:

[0173] If there are no idle threads among the multiple threads, then determine the number of tasks in the waiting queue of the multiple threads, where the waiting queue represents the queue of tasks to be processed in the multiple threads;

[0174] Determine whether the number of tasks exceeds the second preset threshold;

[0175] If the limit is exceeded, the second processing module will be used to compress the data to be compressed.

[0176] In this embodiment, if all threads of the first processing module have ongoing task flows (i.e., no idle threads), in addition to assigning the data compression task to the second processing module as mentioned above, the first processing queue can also be added to the waiting queue. However, to avoid a large number of queued tasks leading to excessively long waiting times for data compression tasks, this embodiment sets a second preset threshold for the number of tasks in the waiting queue across multiple threads. The waiting queue refers to the queue of tasks awaiting processing across multiple threads. A larger number of tasks in the waiting queue indicates more tasks awaiting processing by the first processing module, resulting in a larger backlog of tasks. Consequently, the task of compressing the data will be further down the queue, leading to a longer waiting time.

[0177] Therefore, when there are no idle threads among the multiple threads of the first processing module, the number of tasks waiting in the queue of the current multiple threads can be determined. It is then determined whether the number of tasks exceeds a second preset threshold. If the number of tasks reaches the second preset threshold, it indicates that the first processing module has many tasks waiting to be processed, resulting in a large backlog of tasks and potentially long queuing times, affecting the efficiency of the data compression process. In this case, the second processing module can be selected to process the data to be compressed. However, if the number of tasks is less than the second preset threshold, it indicates that the first processing module does not have many tasks waiting to be processed, and the first processing module can continue to be used for the data compression process, with the data to be compressed remaining in the first processing queue.

[0178] For example, taking an FPGA as the first processing module and a CPU as the second processing module, assume that the number of tasks in the waiting queue limited by the second preset threshold is 3. After writing the data to be compressed into the first processing queue of the FPGA, if the number of tasks in the waiting queue of the FPGA is 4 (greater than the second preset threshold), then the CPU can be used to process the data to be compressed. If the number of tasks in the waiting queue of the FPGA is 2 (less than the second preset threshold), then the FPGA can continue to be used for the data compression process, and the first processing queue of the data to be compressed will continue to be queued.

[0179] In this way, when there are no idle threads among the multiple threads of the first processing module, it can be determined whether to continue using the first processing module for data compression based on the number of tasks in the waiting queue of the multiple threads. This avoids the situation where the queuing time of the data compression task process is too long and improves the execution efficiency of data compression.

[0180] For a better understanding, please refer to the above embodiments. Figure 5A , Figure 5A This is another flowchart of the data processing method in an embodiment of this application. For example... Figure 5A As shown, taking an FPGA as the first processing module and a CPU as the second processing module as an example, the data processing method can specifically include the following steps:

[0181] 301. Obtain the data to be compressed. If the data to be compressed is less than the first preset threshold, then proceed to step 302. If the data to be compressed reaches the first preset threshold, then determine whether the compression algorithm currently applied by the FPGA is the target compression algorithm required by the data to be compressed. If yes, then proceed to step 304. If no, then proceed to step 303.

[0182] 302. Compression is performed using the CPU;

[0183] 303. Use a virtual machine to write the target compression algorithm to the FPGA. If the target compression algorithm is successfully written, proceed to step 304. If the target compression algorithm fails to be written, proceed to step 302.

[0184] 304. Input the data to be compressed into the FPGA using a virtual machine. If the queue in the FPGA is full, proceed to step 302. If the queue in the FPGA is not full, proceed to step 305.

[0185] 305. Use FPGA to compress the data to be compressed;

[0186] 306. Compression complete.

[0187] Optionally, in the above Figure 3 Based on the corresponding embodiments, in another optional embodiment of the data processing method provided in this application, the following steps may also be included:

[0188] If there are no idle threads among the multiple threads, then determine the first priority corresponding to the first processing queue;

[0189] Get the second processing queue of multiple threads, where the second processing queue is the processing queue that is currently being executed;

[0190] Determine the second priority corresponding to the second processing queue;

[0191] If the first priority is higher than the second priority, then the first processing queue will preempt the target thread occupied by the second processing queue.

[0192] In the target thread, the target compression algorithm is used to compress the data to be compressed in the first processing queue.

[0193] In this embodiment, all threads of the first processing module have ongoing task flows, meaning there are no idle threads among the multiple threads of the first processing module. If the first processing module is still needed to compress the data to be compressed, the priority of the first processing queue where the data to be compressed is located can be compared with the priority of the tasks being processed in the multiple threads to determine whether the first processing module should be used to compress the data to be compressed first.

[0194] Please see Figure 5B , Figure 5B This is another flowchart of the data processing method in an embodiment of this application. For example... Figure 5B As shown, taking an FPGA as the first processing module and a CPU as the second processing module as an example, the data processing method can specifically include the following steps:

[0195] 401. Prepare the byte array to be compressed;

[0196] Step 401 and Figure 3 Step 101 shown is similar, and will not be described in detail here;

[0197] 402. Determine if the data to be compressed is too small;

[0198] Configure a first preset threshold. If the amount of data to be compressed exceeds the first preset threshold, it means that the size of the data to be compressed is sufficient, and proceed to step 403; if the amount of data to be compressed does not exceed the first preset threshold, it means that the data to be compressed is too small, and proceed to step 404.

[0199] 403. Determine the algorithm status of the FPGA;

[0200] By checking the FPGA's algorithm status, determine whether the compression algorithm currently applied by the FPGA is consistent with the target algorithm required by the data to be compressed. If they are consistent, proceed to step 408; otherwise, proceed to step 405.

[0201] 404. Compression is performed using the CPU;

[0202] If compression is complete, proceed to step 415.

[0203] 405. Write the required FPGA algorithm into the FPGA;

[0204] Step 405 and Figure 3 Step 103 is similar to the one shown, and will not be described in detail here;

[0205] 406. Determine if the write operation was successful;

[0206] In practical applications, writing compression algorithms cannot be guaranteed to be 100% successful. For example, incompatibility between the FPGA and the compression algorithm being written may cause the writing process to fail. Therefore, after writing the compression algorithm, it is necessary to determine whether the writing was successful. If successful, step 407 is executed; if the writing is unsuccessful, step 404 is executed.

[0207] 407. Update the FPGA status;

[0208] If the compression algorithm is successfully written, the written compression algorithm will be updated to the currently applied compression algorithm.

[0209] 408. Submit the compression task to the work queue;

[0210] 409. Determine if the queue is full;

[0211] If the queue is full, the CPU can be used for compression, i.e., step 404 is executed; if the queue is not full, the compression task is written into the FPGA's waiting queue, i.e., step 410 is executed.

[0212] 410. Write the compression task into the FPGA's wait queue;

[0213] 411. Retrieve the byte array to be compressed from the queue;

[0214] If the current task execution order is the waiting queue where the data to be compressed is located, retrieve the byte array to be compressed from the queue and execute the compression process.

[0215] 412. Compression is performed using FPGA compression algorithms;

[0216] The appropriate compression algorithm is used to compress the data to be compressed. Step 412 and... Figure 3 Step 105 is similar to the one shown, and will not be described in detail here;

[0217] 413. Return the compressed byte group;

[0218] If compression is complete, the compressed byte array is returned.

[0219] 414. Notify the thread that submitted the compression task that compression is complete;

[0220] Once compression is complete, a notification will be sent indicating that the compression task has been completed, allowing subsequent compression tasks to continue.

[0221] 415. Compression complete.

[0222] For easier understanding, please refer to Figure 6 , Figure 6This is a flowchart illustrating data compression based on the first priority of the first processing queue in an embodiment of this application. For example... Figure 6 As shown, this embodiment may specifically include the following steps:

[0223] 501. Determine the first priority corresponding to the first processing queue;

[0224] Since there are no idle threads among the multiple threads of the first processing module at this time, it can be determined whether the first processing queue needs to be prioritized based on its first priority. Therefore, it is necessary to determine the first priority corresponding to the first processing queue.

[0225] 502. Determine the second priority corresponding to the second processing queue;

[0226] Specifically, in this embodiment, the second processing queue refers to the processing queue that the first processing module is currently executing. It should be noted that in this embodiment, the execution order of steps 501 and 502 is not limited. For example, step 501 can be executed first, followed by step 502; step 502 can be executed first, followed by step 501; or steps 501 and 502 can be executed simultaneously. No specific limitation is made here.

[0227] 503. Compare the first priority and the second priority;

[0228] After determining the first priority and the second priority, the two priorities need to be compared. If the first priority is higher than the second priority, then step 404 is executed; if the first priority is lower than or equal to the second priority, then the first processing queue is placed into the waiting queue for queuing and processing.

[0229] 504. Preempt the target thread occupied by the second processing queue;

[0230] If the first priority is higher than the second priority, it means that the first processing queue corresponding to the data to be compressed should take precedence over the second processing queue. In this case, the first processing queue needs to preempt the target thread occupied by the second processing queue. That is, the target thread stops executing the second processing queue with lower priority and starts executing the first processing queue with higher priority.

[0231] 505. In the target thread, compress the data to be compressed in the first processing queue.

[0232] In this embodiment, when there are no idle threads among the multiple threads of the first processing module, the priority of the first processing queue corresponding to the data to be compressed is combined to determine whether to compress the data to be compressed first, thereby reducing the queue waiting time of the data to be compressed and improving the execution efficiency of the scheme.

[0233] Optionally, in the above Figure 3 Based on the corresponding embodiments, in another optional embodiment of the data processing method provided in this application, after compressing the data to be compressed using the target compression algorithm, the method may further include the following steps:

[0234] Output the compressed data to be compressed, and obtain the target data;

[0235] Based on the target data, it is determined that there is a new idle thread in the first processing module.

[0236] In this embodiment, after compressing the data to be compressed using the target compression algorithm, the compressed data to be compressed, i.e., the target data, is output. After obtaining the target data, it indicates that the compression process for that data is complete. The thread previously used to compress that data in the first processing module should then be in an idle state, i.e., a new idle thread. This new idle thread can be used to execute other task queues, including subsequent data compression tasks. Through this method, after the compression process is completed, it can be determined that a new idle thread exists that can be used to execute other task queues, improving the flexibility of the solution.

[0237] It should be understood that the data processing method provided in the embodiments of this application can be applied to the field of cloud storage technology. For example, when Figure 2 The server or terminal shown executes the data processing method provided in this application on the data to be compressed, and after outputting the target data, the target data can be stored in the cloud server, thereby saving the storage resources of the local device and reducing the transmission cost of transmitting the target data from the local device to the cloud server.

[0238] Cloud storage is a new concept that extends and develops from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as a storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of storage devices of various types (storage devices are also called storage nodes) in the network to work together through application software or application interfaces to provide data storage and business access functions to the outside world.

[0239] Currently, the storage method of storage systems is as follows: Logical volumes are created. During the creation of a logical volume, physical storage space is allocated to each logical volume. This physical storage space may consist of a single storage device or the disks of several storage devices. Clients store data on a logical volume, which means storing the data on the file system. The file system divides the data into many parts, each part being an object. Each object contains not only the data but also additional information such as a data identifier (ID, ID entity). The file system writes each object to the physical storage space of that logical volume and records the storage location information of each object. Therefore, when a client requests access to data, the file system can allow the client to access the data based on the storage location information of each object.

[0240] The process by which a storage system allocates physical storage space to a logical volume is as follows: the physical storage space is pre-divided into strips according to the capacity estimate of the objects stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the grouping of Redundant Array of Independent Disks (RAID). A logical volume can be understood as a strip, thus allocating physical storage space to the logical volume.

[0241] Optionally, in the above Figure 3 Based on the corresponding embodiments, in another optional embodiment of the data processing method provided in this application, before obtaining the data to be compressed, the following steps may be included:

[0242] Get the file to be compressed;

[0243] Parse the file to be compressed into a byte array;

[0244] The byte array is identified as the data to be compressed.

[0245] The data processing method provided in this application is applicable to the compression of files of various data types, including but not limited to video files, audio files, text, and images, etc., without specific limitations here. After obtaining a file of any data type to be compressed, it can be parsed into a byte array format and used as the data to be compressed. Through the above method, compression of files of various data types can be achieved, improving the feasibility of the solution.

[0246] To better implement the above-described solutions of the embodiments of this application, related apparatus for implementing the above solutions is also provided below. Please refer to... Figure 7 , Figure 7 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. The data processing device includes:

[0247] Acquisition unit 601 is used to acquire data to be compressed;

[0248] The determining unit 602 is used to determine the target compression algorithm required for the data to be compressed.

[0249] The judgment unit 603 is used to determine whether the compression algorithm currently applied by the first processing module is the target compression algorithm;

[0250] The writing unit 604 is used to write the target compression algorithm to the first processing module using a virtual machine when the compression algorithm currently applied by the first processing module is the target compression algorithm.

[0251] The judgment unit 603 is also used to determine whether the compression algorithm applied by the first processing module has been successfully switched to the target compression algorithm;

[0252] Locking unit 605 is used to lock the target compression algorithm to the compression algorithm applied by the first processing module;

[0253] Input unit 606 is used to input data to be compressed into the first processing module using a virtual machine;

[0254] Compression unit 607 is used in the first processing module to compress the data to be compressed using a target compression algorithm.

[0255] Optionally, in the above Figure 7 Based on the corresponding embodiments, in one embodiment of the data processing apparatus provided in this application, the data processing apparatus further includes a configuration unit 608.

[0256] Configuration unit 608 is used to configure N compression algorithms to the virtual machine, where N is an integer greater than or equal to 1;

[0257] The determining unit 602 is specifically used to receive a compression instruction for the data to be compressed, the compression instruction being used to specify the target compression algorithm required for the data to be compressed;

[0258] The target compression algorithm is determined from N compression algorithms available in the virtual machine.

[0259] Optionally, in the above Figure 7 Based on the corresponding embodiments, in one embodiment of the data processing apparatus provided in this application,

[0260] Configuration unit 608 is also used to configure N compression algorithms to the virtual machine, where N is an integer greater than or equal to 1;

[0261] The determining unit 602 is also used to determine the default compression algorithm from N compression algorithms;

[0262] The determining unit 602 is specifically used to determine the default compression algorithm as the target compression algorithm required for the data to be compressed.

[0263] Optionally, in the above Figure 7 Based on the corresponding embodiments, in one embodiment of the data processing apparatus provided in this application, the data processing apparatus further includes a trigger unit 609.

[0264] The determining unit 602 is also used to determine the amount of data to be compressed;

[0265] The judgment unit 603 is also used to determine whether the amount of data exceeds the first preset threshold.

[0266] The triggering unit 609 is also used to trigger the determining unit when the amount of data reaches the first preset threshold.

[0267] The input unit 606 is also used to transmit the data to be compressed to the second processing module when the amount of data does not exceed the first preset threshold.

[0268] The compression unit 607 is also used to compress the data to be compressed using the second processing module.

[0269] Optionally, in the above Figure 7 Based on the corresponding embodiments, in one embodiment of the data processing apparatus provided in this application, the first processing module includes multiple threads.

[0270] The compression unit 607 is specifically used to write the data to be compressed into the first processing queue of the first processing module.

[0271] Determine if there is an idle thread among multiple threads;

[0272] When there is an idle thread among multiple threads, the data to be compressed is obtained from the first processing queue;

[0273] In the idle thread, the target compression algorithm is used to compress the data to be compressed;

[0274] Compression unit 607 is also used to compress the data to be compressed by a second processing module when there are no idle threads among the multiple threads.

[0275] Optionally, in the above Figure 7 Based on the corresponding embodiments, in one embodiment of the data processing apparatus provided in this application,

[0276] The determining unit 602 is also used to determine the number of tasks in the waiting queue of multiple threads when there are no idle threads among the multiple threads, wherein the waiting queue represents the queue of tasks to be processed in multiple threads.

[0277] The judgment unit 603 is also used to determine whether the number of tasks exceeds the second preset threshold.

[0278] The compression unit 607 is also used to compress the data to be compressed by the second processing module when the number of tasks exceeds the second preset threshold.

[0279] In one possible design, in another implementation of the embodiments of this application, the data processing device further includes a preemption unit 610.

[0280] The determining unit 602 is also used to determine the first priority corresponding to the first processing queue when there are no idle threads among the multiple threads;

[0281] The acquisition unit 601 is also used to acquire the second processing queue of multiple threads, wherein the second processing queue is the processing queue that is being executed;

[0282] The determining unit 602 is also used to determine the second priority corresponding to the second processing queue;

[0283] The preemption unit 610 is used to preempt the target thread occupied by the second processing queue for the first processing queue when the first priority is higher than the second priority;

[0284] Compression unit 607 is also used to compress the data to be compressed in the first processing queue using a target compression algorithm in the target thread.

[0285] In one possible design, in another implementation of the embodiments of this application, the data processing device further includes an output unit 611.

[0286] Output unit 611 is used to output the compressed data to be compressed, thus obtaining the target data;

[0287] The determining unit 602 is used to determine, based on the target data, whether there is a new idle thread in the first processing module.

[0288] In this embodiment, the data processing device can perform the aforementioned... Figures 3 to 6 The specific operation of any of the illustrated embodiments will not be described in detail here.

[0289] This application also provides a computer device for performing... Figures 3 to 6 Operation of any of the illustrated embodiments. Please refer to... Figure 8 , Figure 8This is a schematic diagram of a computer device 700 according to an embodiment of this application. As shown, the computer device 700 can vary significantly due to different configurations or performance, and may include one or more central processing units (CPUs) 722 (e.g., one or more processors) and a memory 732, and one or more storage media 730 (e.g., one or more mass storage devices) for storing application programs 742 or data 744. The memory 732 and storage media 730 can be temporary or persistent storage. The program stored in the storage media 730 may include one or more modules (not shown in the figure), each module may include a series of instruction operations on the computer device. Furthermore, the CPU 722 may be configured to communicate with the storage media 730 and execute the series of instruction operations in the storage media 730 on the computer device 700.

[0290] Computer device 700 may also include one or more power supplies 726, one or more wired or wireless network interfaces 750, one or more input / output interfaces 758, and / or one or more operating systems 741, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0291] The steps performed in the above embodiments can be based on this Figure 8 The structure of the computer device shown.

[0292] This application also provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the methods described in the foregoing embodiments.

[0293] This application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned actions. Figures 3 to 6 A data processing method for any of the embodiments shown.

[0294] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0295] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0296] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0297] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0298] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, an interactive video management device, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0299] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized in that, include: Obtain the data to be compressed; Determine the target compression algorithm required for the data to be compressed; Determine whether the compression algorithm currently applied by the first processing module is the target compression algorithm; If not, the target compression algorithm is written to the first processing module using a virtual machine; Determine whether the compression algorithm applied by the first processing module has been successfully switched to the target compression algorithm; If so, a state lock is used to lock the target compression algorithm to the compression algorithm applied by the first processing module, so as to prevent the compression algorithm applied by the first processing module during the execution of the compression task from being tampered with by other tasks. The virtual machine is used to input the data to be compressed into the first processing module; In the first processing module, the target compression algorithm is used to compress the data to be compressed; If the target compression algorithm fails to write, the compression of the data to be compressed by the first processing module is abandoned, and the data to be compressed is sent to the second processing module so that the second processing module can compress the data to be compressed, thereby ensuring that the compression task can continue to be executed. The second processing module is different from the first processing module.

2. The method according to claim 1, characterized in that, The method further includes: If the compression algorithm currently applied by the first processing module is the target compression algorithm, then the virtual machine is used to input the data to be compressed to the first processing module; In the first processing module, the target compression algorithm is used to compress the data to be compressed.

3. The method according to claim 1, characterized in that, Before acquiring the data to be compressed, the method further includes: Configure N compression algorithms for the virtual machine, where N is an integer greater than or equal to 1; The target compression algorithm required to determine the data to be compressed includes: Receive a compression instruction for the data to be compressed, the compression instruction being used to specify the target compression algorithm required for the data to be compressed; The target compression algorithm is determined from the N compression algorithms of the virtual machine.

4. The method according to claim 1, characterized in that, Before acquiring the data to be compressed, the method further includes: Configure N compression algorithms for the virtual machine, where N is an integer greater than or equal to 1; A default compression algorithm is determined from the N compression algorithms; The target compression algorithm required to determine the data to be compressed includes: The default compression algorithm is determined to be the target compression algorithm required for the data to be compressed.

5. The method according to claim 1, 2, 3 or 4, characterized in that, After obtaining the data to be compressed, the method further includes: Determine the amount of data to be compressed; Determine whether the amount of data exceeds a first preset threshold; If the amount of data exceeds the first preset threshold, then the step of determining the target compression algorithm required for the data to be compressed is triggered; If the amount of data does not exceed the first preset threshold, the data to be compressed is transmitted to the second processing module. The second processing module is used to compress the data to be compressed.

6. The method according to claim 1, characterized in that, The first processing module includes multiple threads, and the step of compressing the data to be compressed using the target compression algorithm in the first processing module includes: The data to be compressed is written into the first processing queue of the first processing module; Determine if any of the multiple threads are idle; If there is an idle thread among the plurality of threads, the data to be compressed is obtained from the first processing queue; In the idle thread, the target compression algorithm is used to compress the data to be compressed; If there are no idle threads among the multiple threads, the second processing module is used to compress the data to be compressed.

7. The method according to claim 6, characterized in that, After compressing the data to be compressed using the target compression algorithm, the method further includes: If there are no idle threads among the plurality of threads, then the number of tasks in the waiting queue of the plurality of threads is determined, wherein the waiting queue represents the queue of tasks to be processed in the plurality of threads; Determine whether the number of tasks exceeds a second preset threshold; If the value exceeds the limit, the second processing module will be used to compress the data to be compressed.

8. The method according to claim 6, characterized in that, The method further includes: If there are no idle threads among the plurality of threads, then the first priority corresponding to the first processing queue is determined; Obtain the second processing queue of the plurality of threads, wherein the second processing queue is the processing queue that is currently being executed; Determine the second priority corresponding to the second processing queue; If the first priority is higher than the second priority, then the first processing queue will preempt the target thread occupied by the second processing queue. In the target thread, the target compression algorithm is used to compress the data to be compressed in the first processing queue.

9. The method according to any one of claims 6 to 8, characterized in that, After compressing the data to be compressed using the target compression algorithm, the method further includes: Output the compressed data to be compressed, and obtain the target data; Based on the target data, it is determined that a new idle thread exists in the first processing module.

10. A data processing apparatus, characterized in that, include: The acquisition unit is used to acquire the data to be compressed; A determining unit is used to determine the target compression algorithm required for the data to be compressed. The judgment unit is used to determine whether the compression algorithm currently applied by the first processing module is the target compression algorithm; The writing unit is used to write the target compression algorithm to the first processing module using a virtual machine when the compression algorithm currently applied by the first processing module is not the target compression algorithm. The judgment unit is also used to determine whether the compression algorithm applied by the first processing module has been successfully switched to the target compression algorithm. The locking unit is used to lock the target compression algorithm to the compression algorithm applied by the first processing module using a state lock, so as to prevent the compression algorithm applied by the first processing module during the execution of the compression task from being tampered with by other tasks. An input unit is used to input the data to be compressed into the first processing module using the virtual machine. A compression unit is used in the first processing module to compress the data to be compressed using the target compression algorithm. The device is further configured to, if the target compression algorithm fails to write, abandon the compression of the data to be compressed by the first processing module and send the data to be compressed to the second processing module so that the second processing module can compress the data to be compressed, thereby ensuring that the compression task can continue to be executed. The second processing module is different from the first processing module.

11. The apparatus according to claim 10, characterized in that, The device also includes a configuration unit. The configuration unit is used to configure N compression algorithms for the virtual machine, where N is an integer greater than or equal to 1; The determining unit is specifically used to receive a compression instruction for the data to be compressed, wherein the compression instruction is used to specify the target compression algorithm required for the data to be compressed. The target compression algorithm is determined from the N compression algorithms of the virtual machine.

12. The apparatus according to claim 10, characterized in that, The device also includes a configuration unit. The configuration unit is further configured to configure N compression algorithms for the virtual machine, where N is an integer greater than or equal to 1; The determining unit is further configured to determine a default compression algorithm from the N compression algorithms; The determining unit is specifically used to determine the default compression algorithm as the target compression algorithm required for the data to be compressed.

13. The apparatus according to claim 10, characterized in that, The device also includes a triggering unit. The determining unit is further configured to determine the amount of data to be compressed; The judgment unit is also used to determine whether the amount of data exceeds a first preset threshold. The triggering unit is further configured to trigger the determining unit if the amount of data exceeds the first preset threshold. The input unit is further configured to transmit the data to be compressed to the second processing module if the amount of data does not exceed the first preset threshold. The compression unit is further configured to compress the data to be compressed using the second processing module.

14. A computer device, characterized in that, The computer device includes a processor and memory: The memory is used to store program code; the processor is used to execute the data processing method according to any one of claims 1 to 9 according to the instructions in the program code.

15. A computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the data processing method according to any one of claims 1 to 9.

16. A computer program product, characterized in that, The computer program product includes instructions that, when executed on a computer device, cause the computer device to perform the method as described in any one of claims 1 to 9.

Images