Processing method of compressed data, electronic device, storage medium and program product

Abstract

This application provides a method for processing compressed data, an electronic device, a storage medium, and a program product, relating to the field of data processing technology. The method includes: in response to a decompression command for first compressed data, decompressing a first data fragment to obtain first decompressed data, wherein the first data fragment is the currently undecompressed data fragment among a plurality of data fragments included in the first compressed data; generating a first index of the first decompressed data in the first compressed data based on first offset information corresponding to the first decompressed data, wherein the first offset information includes a first offset, which indicates the position of the first data fragment among the plurality of data fragments. The technical solution of this application improves the efficiency of index generation.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method for processing compressed data, an electronic device, a storage medium, and a program product, which can be applied to the field of data decompression technology. Background Technology

[0002] Compressed data is familiar to most people. Indexes of compressed data can be used to quickly access the original data within the compressed data while it is still in an uncompressed state. Currently, the process of building an index for compressed data involves first obtaining the complete compressed data, then decompressing it, and finally building the index based on the decompressed data. This index building process is time-consuming and inefficient. Summary of the Invention

[0003] This application provides a method for processing compressed data, an electronic device, a computer-readable storage medium, and a computer program product to alleviate or solve one or more technical problems existing in the prior art.

[0004] In a first aspect, embodiments of this application provide a method for processing compressed data, applied to a decompression device, comprising:

[0005] In response to a decompression command for the first compressed data, the first data fragment is decompressed to obtain the first decompressed data, wherein the first data fragment is the data fragment to be decompressed among the plurality of data fragments included in the first compressed data; based on the first offset information corresponding to the first decompressed data, a first index of the first decompressed data in the first compressed data is generated, wherein the first offset information includes a first offset, the first offset being used to indicate the position of the first data fragment among the plurality of data fragments.

[0006] Secondly, embodiments of this application provide a method for processing compressed data, applied to a sender, comprising: in response to satisfying the decompression conditions of the first compressed data, sending a decompression command for the first compressed data to a decompression device; receiving first decompressed data and a first index sent by the decompression device, wherein the first decompressed data is obtained by decompressing the currently undecompressed data segment among the plurality of data segments included in the first compressed data, and the first index is generated based on the first offset information corresponding to the first decompressed data.

[0007] Thirdly, embodiments of this application provide a compressed data processing system, including: a client for sending a decompression command for first compressed data; and a decompression device for receiving the decompression command and implementing the method provided in the first aspect of this application.

[0008] Fourthly, embodiments of this application provide a compressed data processing system, comprising: a client for transmitting first compressed data to a server; the server for receiving the first compressed data and sending a decompression command for the first compressed data; and a decompression device for receiving the decompression command and implementing the method provided in the first aspect of this application.

[0009] Fifthly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the methods of embodiments of this application when executing the computer program.

[0010] Sixthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of any one of the embodiments of this application.

[0011] In a seventh aspect, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method of any one of the embodiments of this application.

[0012] According to the data decompression processing method provided in the embodiments of this application, after decompressing the current data fragments to be decompressed to obtain the first decompressed data, a first index of the first decompressed data in the first compressed data is generated based on the first offset information corresponding to the first decompressed data. Therefore, the index is generated during the decompression process, rather than generating the index after obtaining the complete first compressed data, which incurs additional time consumption. This shortens the overall time spent building the index and improves the efficiency of index building.

[0013] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0014] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.

[0015] Figure 1A and Figure 1B This illustration shows an application scenario diagram of the compressed data processing method according to an embodiment of this application;

[0016] Figure 2 A flowchart of a compressed data processing method 200 according to an embodiment of this application is shown;

[0017] Figure 3 A first schematic diagram of a method for processing compressed data according to an embodiment of this application is shown;

[0018] Figure 4 A second schematic diagram of a method for processing compressed data according to an embodiment of this application is shown;

[0019] Figure 5 A flowchart of a compressed data processing method 500 according to an embodiment of this application is shown;

[0020] Figure 6 A block diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0021] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0022] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application.

[0023] The following terms will be used in the following text:

[0024] A container image is a collection of container image layers that contains everything needed to run an application, such as code, runtime, libraries, environment variables, and configuration files.

[0025] Container image layer: A compressed package containing the parts of an application needed to run, also known as an image layer.

[0026] Image description file: describes which image layers make up the container image and configuration information, including startup commands, entry points, etc.

[0027] Container image repository: A server or service used to store and distribute container images; it is a centralized storage location for container images.

[0028] Image Index: An index for container images. Random access to the contents of a container image can be achieved by searching the image index.

[0029] To address the slow cold start speed of container images, related technologies involve a container image repository receiving a complete container image pushed by a client, then starting compute nodes to decompress the image layers within the container and generating an image index based on the decompression results. This method of building the image index not only incurs the overhead of additional compute nodes but also takes a considerable amount of time, often exceeding minutes, and for large models, it can take tens of minutes. Clearly, building an image index is time-consuming and inefficient.

[0030] Based on this, this application provides a method for processing compressed data, an electronic device, a computer-readable storage medium, and a computer program product to improve the efficiency of index generation. Figure 1A This is a schematic diagram illustrating an application scenario of a data compression processing method provided in an embodiment of this application, such as... Figure 1A As shown, this scenario provides a data compression processing system, which includes a client 110 and a decompression device 120.

[0031] The client 110 may be configured with a data transfer application. The client 110 is used to send a decompression command for the first compressed data to the decompression device 120 in response to a data transfer command input by the user through the data transfer application. For example, the data transfer application may be a standalone application (App), a small program embedded in other applications, or a web application, etc.

[0032] The decompression device 120 maintains a decompression strategy and an index generation strategy. The decompression strategy instructs the decompression of a first data fragment in response to a decompression command for the first compressed data, resulting in first decompressed data. The first data fragment is the currently undecompressed data fragment among multiple data fragments included in the first compressed data. The index generation strategy instructs the generation of a first index of the first decompressed data within the first compressed data based on first offset information corresponding to the first decompressed data. This first index is used to access the first decompressed data when it is in an undecompressed state. The first offset information includes a first offset, which indicates the position of the first data fragment among the multiple data fragments. The undecompressed state can be a completely undecompressed state, meaning no data fragments in the first compressed data have been decompressed; or it can be a partially undecompressed state, meaning some data fragments in the first compressed data have been decompressed, while others remain undecompressed.

[0033] The client 110 and the decompression device 120 can interact via a network, which can be a wired network or a wireless network. Both the client 110 and the decompression device 120 can be terminal devices or servers. For example, the terminal device can be a mobile phone, desktop computer, laptop, tablet computer, vehicle terminal, smart wearable device, etc., and the server can be a physical server, cloud server, etc.

[0034] Furthermore, such as Figure 1B As shown, in one embodiment, the decompression system in the above application scenario may further include a server 130. Accordingly, the client 110 is used to transmit first compressed data to the server 130 in response to a data transmission command input by a user through a data transmission application. The server 130 is used to receive the first compressed data transmitted by the client 110 and send a decompression command for the first compressed data to the decompression device 120. The server 130 may be a physical server, a cloud server, or a server cluster composed of multiple servers.

[0035] exist Figure 1A and Figure 1B The example shown uses a portable laptop as the client (110), a physical server as the decompression device (120), and a physical server as the server (130). It should be understood that... Figure 1A and Figure 1B The illustration is merely a representative representation of an application scenario for the compressed data processing method involved in this application and does not constitute a limitation on the technical solution of this application. In other embodiments, the application scenario of the compressed data processing method involved in this application may include more or fewer components.

[0036] It should be noted that the application scenarios or examples provided in the embodiments of this application are for ease of understanding, and the embodiments of this application do not specifically limit the application of the technical solutions. In addition, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0037] The technical solution of this application and how it solves the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0038] Figure 2 A flowchart of a decompression method 200 for compressed data according to an embodiment of this application is shown. This method can be applied to a decompression device, such as the decompression device 120 in FIG1 (hereinafter referred to as the decompression device), and is executed thereon. Figure 2 As shown, the method may include steps S201 and S202.

[0039] Step S201: In response to the decompression command for the first compressed data, the first data fragment is decompressed to obtain the first decompressed data, wherein the first data fragment is the data fragment to be decompressed among the multiple data fragments included in the first compressed data.

[0040] In one implementation, the arrangement order of the multiple data fragments included in the first compressed data is used to indicate the transmission order of the multiple data fragments. That is, the sender of the decompression command transmits each data fragment to the decompression device one by one according to the transmission order. Accordingly, after receiving the decompression command for the first compressed data sent by the sender, the decompression device can first determine whether the first compressed data is being decompressed for the first time according to the decompression command. If the first compressed data is being decompressed for the first time, and if the data reception conditions are met, the device receives the data fragments transmitted by the sender, determines the first data fragment to be decompressed according to the transmission order, and decompresses the first data fragment according to the decompression algorithm contained in the decompression command to obtain the first decompressed data. In other words, the transmission order is used to indicate the decompression order. The specific implementation process of determining whether the first compressed data is being decompressed for the first time according to the decompression command can be found in the relevant description below. The first compressed data can be any image layer in the container image, or it can be compressed data of other content, such as compressed data of images, audio, video, text, etc. The first compressed data can be any compression format among RAR (Roshal ARchive), ZIP (Zone Information Protocol), GZIP (GNU ZIP), ZSTD (Zstandard), etc.

[0041] In one implementation, the above-mentioned decompression of the first data fragment to obtain the first decompressed data may include: if the first data fragment is the first data fragment to be decompressed, determining the data to be decompressed in the first data fragment that meets the decompression conditions of the decompression algorithm, and decompressing the data to be decompressed to obtain the corresponding first decompressed data. If the first data fragment is not the first data fragment to be decompressed, if there is remaining compressed data in the currently decompressed last data fragment, determining the remaining compressed data and the data to be decompressed in the first data fragment that meets the decompression conditions of the decompression algorithm, and decompressing the data to be decompressed to obtain the corresponding first decompressed data; if there is no remaining compressed data in the currently decompressed last data fragment, determining the data to be decompressed in the first data fragment that meets the decompression conditions of the decompression algorithm, and decompressing the data to be decompressed to obtain the corresponding first decompressed data. The decompression conditions can vary depending on the decompression algorithm. For example, some decompression algorithms require complete data to be decompressed, while others require decompressing compressed data of a fixed size.

[0042] Considering that in practical applications, multiple senders may send decompression commands to the decompression device, in order to improve the processing efficiency of decompression commands, in one implementation, the decompression device can process each decompression command through a corresponding process. Specifically, when the decompression device receives a decompression command for the first compressed data, it can allocate a processing process for the first compressed data and send the access address of the processing process to the sender of the decompression command. The sender can allocate an upload process for the first compressed data and send a connection request to the processing process through the upload process according to the access address of the processing process. When the processing process receives the connection request, it establishes a connection with the corresponding upload process and receives the data fragments transmitted by the upload process through the connection. Allocating a processing process for the first compressed data can be done by selecting an idle processing process from pre-created processing processes and allocating it to the first compressed data, or by creating a new processing process and allocating it to the first compressed data.

[0043] Specifically, when the sender is a client, the aforementioned processing process may include a first receiving process and a first decompression process. The first receiving process establishes a first connection with the uploading process, receives data fragments transmitted by the uploading process through the first connection, and sends them to the corresponding first decompression process. The first decompression process decompresses the determined first data fragments to obtain first decompressed data and executes step S202. Accordingly, the aforementioned determination that the data receiving conditions are met may include: if the first connection is successfully established, determining that the data receiving conditions are met.

[0044] When the sender is a server, the aforementioned processing process can be a decompression process. Specifically, when the server receives an upload request for the first compressed data from the client, it allocates a second receiving process for the first compressed data and sends a decompression command to the decompression device based on information such as the first data identifier and decompression algorithm of the first compressed data contained in the upload request. The decompression device responds to the decompression command by allocating a second decompression process for the first compressed data and sends the communication address of the second decompression process to the server. The server, through its second receiving thread, sends a second connection request to the second decompression process based on the communication address of the second decompression process. The second decompression process responds to the second connection request and establishes a second connection with the second receiving process. If the server confirms that the second connection is successfully established, it sends the communication address of the second receiving process to the client. The client allocates an upload process for the first compressed data and sends a third connection request to the second receiving process through the upload process based on the communication address of the second receiving process. The second receiving process responds to the third connection request and establishes a third connection with the upload process. The upload process transmits data fragments of the first compressed data one by one to the second receiving process through the third connection. The second receiving process sends the received data fragments to the second decompression process through the second connection. The second decompression process determines the first data fragment to be decompressed, decompresses the first data fragment to obtain the first decompressed data, and executes step S202. Correspondingly, the aforementioned determination that the data reception conditions are met may include: if the second connection is successfully established, determining that the data reception conditions are met.

[0045] Step S202: Generate a first index of the first decompressed data in the first compressed data according to the first offset information corresponding to the first decompressed data, wherein the first offset information includes a first offset, which is used to indicate the position of the first data fragment in multiple data fragments.

[0046] To facilitate convenient access to the first decompressed data when the first compressed data is in an undeployed state, the decompression device, after obtaining the first decompressed data, generates a first index of the first decompressed data within the first compressed data based on the first offset information corresponding to the first decompressed data. This first index is then used to locate the first decompressed data within the first compressed data. In other words, the first index is used to access the first decompressed data when it is in an undeployed state. By using the first offset as one of the first offset information, the first data fragment can be quickly located based on the first offset, allowing access to the first decompressed data based on the first data fragment.

[0047] To facilitate access to the decompressed data by the sender of the decompression command, in one embodiment, step S202 may further include sending the first decompressed data and the first index to the sender.

[0048] Understandably, in practical applications, the decompression speed may be faster than the data fragment receiving speed. That is, after the decompression device finishes decompressing the first data fragment to be decompressed, it needs to wait to receive the next data fragment. During this waiting period, the index of the first decompressed data can be generated. This not only avoids the time spent waiting to receive subsequent data fragments, but also eliminates the need to generate the index after all the first compressed data has been received and decompressed, thus shortening the overall index generation time (from receiving the first compressed data to generating the index) and improving index generation efficiency. Furthermore, even if the decompression speed is slower than the data fragment receiving speed, because the index is generated while decompressing during data fragment transmission, rather than waiting until all the first compressed data has been decompressed, the overall index generation time is also shortened, improving index generation efficiency.

[0049] As can be seen, according to the data decompression processing method provided in this application embodiment, after decompressing the current data fragments to be decompressed to obtain the first decompressed data, a first index of the first decompressed data in the first compressed data is generated based on the first offset information corresponding to the first decompressed data. Therefore, the index of the decompressed data is generated during the decompression process of the first compressed data, rather than generating the index after obtaining the complete first compressed data, which consumes additional time. This shortens the overall time spent generating the index and improves the efficiency of index generation.

[0050] Considering that in practical applications, the transmission speed of data fragments may be faster than the decompression speed, to ensure the orderly decompression, in one implementation, after receiving the data fragments and offsets transmitted by the sender, the decompression device, upon determining that the storage conditions are met, stores the received data fragments and offsets in a first storage area, and retrieves the first data fragment to be decompressed and its first offset from the first storage area. In other words, the method may further include:

[0051] In response to receiving the first data fragment and the first offset, the system verifies whether the first offset is correct based on the first size, where the first size is the total size of the currently received data fragments; if the first offset is correct, the system associates and saves the first data fragment and the first offset to the first storage area.

[0052] Specifically, to ensure index validity, the decompression device, in response to the decompression command, can initialize a first size and update it based on the size of the received data fragments, provided that the first compressed data is being decompressed for the first time. To avoid indexing errors caused by data fragment loss during transmission, the sender also transmits a first offset of the data fragment when transmitting it to the decompression device. Correspondingly, upon receiving the first data fragment and the first offset, the decompression device verifies the correctness of the first offset based on the first size. If the first offset is correct, it determines that the storage conditions are met and associates and saves the first data fragment and the first offset to the first storage area.

[0053] In one implementation, the first offset may indicate the starting position of the first data fragment among multiple data fragments. Accordingly, the aforementioned verification of the correctness of the first offset based on the first size may include: determining whether the current first size is equal to the first offset; if the first size is equal to the first offset, the first offset is determined to be correct. For example, the initial value of the first size is zero, meaning the starting position of the first data fragment in the first compressed data is 0, the starting position of the second data fragment in the first compressed data is after the end position of the first data fragment, and so on. In kilobytes (KB), the current first size is 64K, and the first offset indicates that the starting position of the first data fragment among multiple data fragments is 64K (i.e., the first data fragment begins after 64K). If the first size is the same as the starting position, then the first offset is determined to be correct.

[0054] In another implementation, the first offset may indicate the position of the first data fragment among multiple data fragments, where all data fragments are of the same size. Accordingly, the aforementioned verification of the first offset based on the first size may include: determining the target size of each data fragment preceding the first data fragment based on the arrangement position and the fragment size; if the target size is the same as the first size, the first offset is determined to be correct. For example, in kilobytes, if the first size is 64K, the size of each data fragment is 32K, and the first offset indicates that the first data fragment is positioned 3 among multiple data fragments, then the target size can be determined as (3-1)*32 = 64K. Since the target size is consistent with the first size, the first offset is determined to be correct.

[0055] Therefore, upon receiving the first data fragment and the first offset, verifying the first offset ensures the integrity of the received first data fragment, thereby ensuring the accuracy of the index generated based on the first offset. By saving the first data fragment and the first offset to the first storage area, the currently undecompressed data fragment can be obtained from the first storage area, ensuring the orderly decompression of each data fragment.

[0056] To facilitate convenient access to the first decompressed data when the first compressed data is in an undecompressed state, in one embodiment, the first offset information may further include a second offset, which indicates the position of the first decompressed data in the currently decompressed data set of the first compressed data. Correspondingly, step S202 may further include:

[0057] The second offset is determined based on the second size, where the second size is the total size of the currently decompressed dataset assuming that the currently decompressed dataset does not contain the first decompressed data.

[0058] Specifically, to ensure index validity, the decompression device, in response to the decompression command, can initialize a second size and update it based on the size of the decompressed data obtained, provided that the first compressed data is being decompressed for the first time. Correspondingly, after obtaining the first decompressed data, the decompression device can determine a second offset based on the current second size.

[0059] In one implementation, the second offset indicates the starting position of the first decompressed data within the current decompressed dataset of the first compressed data. Accordingly, determining the second offset based on the second size may include: determining the second size as the second offset. For example, if the second size is 100K, then the second offset is determined to be 100k.

[0060] In another implementation, the second offset indicates the start and end positions of the first decompressed data within the current decompressed dataset of the first compressed data. Accordingly, determining the second offset based on the second size may include: determining the second size as the start position of the first decompressed data within the current decompressed dataset of the first compressed data; determining the end position of the first decompressed data within the current decompressed dataset of the first compressed data by summing the second size and the size of the first decompressed data; and determining the determined start and end positions as the second offset. For example, if the second size is 100K and the size of the first decompressed data is 56K, then the start position of the first decompressed data within the current decompressed dataset of the first compressed data is determined to be 100K and the end position to be 156K, i.e., the second offset is the start position 100K and the end position 156K.

[0061] Therefore, by determining the second offset of the first decompressed data and using the second offset as one of the first offset information corresponding to the first decompressed data, the first decompressed data can be accurately located based on the second offset when the first compressed data is in an undecompressed state, thereby enabling quick access to the first decompressed data.

[0062] It should be noted that the first offset information corresponding to the first decompressed data is not limited to the aforementioned first offset and second offset, but may also include the size of the first data fragment, the end position of the first data fragment in the first compressed data, etc., which can be set according to actual needs in practical applications.

[0063] After obtaining the first offset information corresponding to the first decompressed data, a first index of the first decompressed data in the first compressed data can be generated based on the first offset information. In one embodiment, generating the first index of the first decompressed data in the first compressed data based on the first offset information corresponding to the first decompressed data may include: determining the target offset of the decompressed data corresponding to the first data fragment based on a second offset, establishing a correlation between the first offset and the target offset, and using this correlation as the first index. The target offset is used to indicate the position of the decompressed data corresponding to the first data fragment in the currently decompressed data set of the first compressed data.

[0064] As described above, when decompressing a complete data fragment, the first decompressed data corresponds exactly to the first data fragment. Accordingly, determining the target offset of the decompressed data corresponding to the first data fragment based on the second offset can include: determining the second offset as the target offset of the decompressed data corresponding to the first data fragment.

[0065] For example, the first offset is used to indicate the starting position of the first data fragment among multiple data fragments, and the second offset is used to indicate the starting position of the first decompressed data in the current decompressed dataset of the first compressed data. The first offset is 64K and the second offset is 156K. The corresponding first index can be (64K, 156K). The first index indicates that the decompressed data corresponding to the data fragment with the starting position of 64K has the starting position of 156K in the decompressed dataset.

[0066] Furthermore, when the decompression condition indicates that any data fragment may contain remaining compressed data, the first decompressed data and the first data fragment may or may not correspond completely. In this case, to ensure the accuracy of the generated index, in one implementation, the aforementioned process of determining the remaining compressed data and the data to be decompressed in the first data fragment that meets the decompression algorithm's decompression conditions, and decompressing the data to be decompressed to obtain the corresponding first decompressed data, may include: concatenating the remaining compressed data and the first data fragment using a preset concatenation identifier to obtain concatenated data, determining the data to be decompressed in the concatenated data that meets the decompression algorithm's decompression conditions, and decompressing the data to be decompressed to obtain the first decompressed data. The first decompressed data includes the preset concatenation identifier; the decompressed data before the preset concatenation identifier is the decompressed data corresponding to the remaining compressed data; and the decompressed data after the preset concatenation identifier is the decompressed data corresponding to all or part of the compressed data in the first data fragment.

[0067] The second offset is used to indicate the starting position of the first decompressed data in the current decompressed dataset of the first compressed data. Accordingly, the aforementioned determination of the target offset of the decompressed data corresponding to the first data fragment based on the second offset may include: if it is determined that the first decompressed data contains a preset splicing identifier, determining the size of the decompressed data corresponding to the remaining compressed data based on the preset splicing identifier in the first decompressed data, adding the second offset to the size of the decompressed data corresponding to the remaining compressed data, and determining the sum as the target starting position of the decompressed data corresponding to the first data fragment in the current decompressed dataset of the first compressed data, and determining the target starting position as the target offset.

[0068] For example, the first offset is used to indicate the starting position of the first data fragment among multiple data fragments, and the second offset is used to indicate the starting position of the first decompressed data in the current decompressed dataset of the first compressed data. The first offset is 96K, the second offset is 202K, and the preset splicing identifier in the first decompressed data is located at the end position of 10K. Then the target offset is determined to be 202+10=212K. Therefore, the generated first index can be (96k, 212K). This first index indicates that the decompressed data applied to the data fragment with the starting position of 96K has the starting position of 212K in the decompressed dataset.

[0069] It should be noted that the first index in the above example is for illustration only and not for limitation. The specific form of the first index can be set as needed in actual application. The first index may also include the end position of the decompressed data in the decompressed dataset, etc., which will not be listed and explained in this application.

[0070] To ensure the accuracy of the data received by the decompression device, in one embodiment, step S202 may further include: determining the total amount of currently decompressed compressed data, determining the first verification data of the currently decompressed compressed data according to a preset verification algorithm, and sending the total amount of currently decompressed compressed data and the first verification data to the sender. Specifically, when the sender receives the total amount of currently decompressed compressed data and the first verification data, it can determine the decompressed compressed data based on the received total amount, generate target verification data for the determined decompressed compressed data according to the preset verification algorithm, and continue transmitting data fragments if the target verification data matches the first verification data; if the target verification data does not match the first verification data, it sends an error message to the decompression device and retransmits each data fragment of the first compressed data to the decompression device.

[0071] To ensure the accuracy of the first index, in one implementation, the method may further include: updating the first size according to the size of the first data fragment; and updating the second size according to the size of the first decompressed data.

[0072] Specifically, the first size is added to the size of the first data fragment to obtain a first sum result, and the first size is updated to the first sum result. The second size is added to the size of the first decompressed data to obtain a second sum result, and the second size is updated to the second sum result.

[0073] In one implementation, after verifying the first offset is correct, the first size can be updated according to the size of the first data fragment; and after obtaining the first decompressed data, the second size can be updated according to the size of the first decompressed data. In another implementation, if the decompression speed is greater than the receiving speed, after obtaining the first decompressed data, the first size can be updated according to the size of the first data fragment, and the second size can be updated according to the size of the first decompressed data.

[0074] In other words, the first size can be updated after receiving the first data fragment and before receiving the next data fragment. Similarly, the second size can be updated after receiving the first decompressed data and before decompressing the next data fragment. Therefore, by updating the first and second sizes, their accuracy is ensured. This allows for effective verification of each received first offset based on the first size, and accurate determination of the first offset information corresponding to each obtained first decompressed data based on the second size, thus guaranteeing the generation of an accurate index.

[0075] To effectively maintain the indexes of different compressed data, in one implementation, step S202 may further include:

[0076] If the first data shard is not the last data shard among multiple data shards, the index is saved to the first storage area; or, if the first data shard is the last data shard among multiple data shards, the currently obtained first index and the multiple indexes stored in the first storage area are associated with the first data identifier of the first compressed data and saved to the second storage area, and the multiple indexes in the first storage area are deleted.

[0077] Specifically, during the transmission of data fragments to the decompression device, if the sender determines that the data fragment to be transmitted is not the last data fragment in the first compressed data, it can transmit the data fragment, its first offset, and the first data identifier of the first compressed data to the decompression device. Correspondingly, after verifying the received first offset, the decompression device associates and saves the received data fragment, first offset, and first data identifier to the first storage area. If the sender determines that the data fragment to be transmitted is the last data fragment in the first compressed data, it can transmit the data fragment, its first offset, the first data identifier of the first compressed data, and a transmission end identifier to the decompression device. Correspondingly, after verifying the received first offset, the decompression device associates and saves the received data fragment, first offset, first data identifier, and transmission end identifier to the first storage area.

[0078] Furthermore, when the decompression device obtains the first data fragment and the first offset from the first storage area, it determines whether the first storage area contains a transmission end identifier associated with the first data fragment and the first offset. If no transmission end identifier is found, the generated first index is saved to the first storage area. If a transmission end identifier is found, it is determined that the first data fragment is the last data fragment in the first compressed data. The decompression device then associates the first index of the currently generated first decompressed data, along with multiple indexes stored in the first storage area, with the first data identifier and saves them to the second storage area, and deletes the multiple indexes in the first storage area. For example, the first storage area is local memory, and the second storage area is cloud storage, etc. It is understood that multiple indexes, the received data fragments, and the first offset can be stored in different storage locations within the first storage area.

[0079] To facilitate access to the decompressed data of the first compressed data, in one embodiment, saving the generated first index to the first storage area may include: determining the metadata corresponding to the generated first index and the first decompressed data as the first data fragment, and saving the metadata to the first storage area. Correspondingly, associating the first index of the currently generated first decompressed data, and multiple indexes stored in the first storage area with the first data identifier and saving them to the second storage area, and deleting multiple indexes in the first storage area, may include: determining the first index of the currently generated first decompressed data and the metadata corresponding to the first data fragment of the first decompressed data, associating this metadata and multiple metadata stored in the first storage area with the first data identifier and saving them to the second storage area, and deleting multiple metadata in the first storage area.

[0080] Therefore, after obtaining the first index of the last data fragment of the first compressed data, by associating each index of the first compressed data with the first data identifier of the first compressed data and saving it to the second storage area, it is convenient for the user of the index (which could be the sender of the decompression command or other users, etc.) to retrieve it from the second storage area, avoiding the access pressure on the decompression device caused by each user retrieving the index from the decompression device. By deleting each index in the first storage area, storage space for the index can be provided for processing the next compressed data, and the maintenance difficulty caused by storing indexes of different compressed data in the first storage area can be avoided.

[0081] Furthermore, when the server is the sender of the decompression command, the decompression device, after determining that the first data fragment is the last data fragment of the first compressed data, may further include: determining the total size of the received data fragments, determining the second verification data of the received data fragments according to a preset verification algorithm, and sending the total size of the received data fragments and the second verification data to the server. Wherein, after receiving the total size of the data fragments and the second verification data, the server generates third verification data for the data fragments transmitted by the client according to the preset verification algorithm. If the second verification data and the third verification data are consistent, and the total size of the data fragments transmitted by the client is consistent with the total size of the data fragments sent by the decompression device, then the received data fragments are saved to the second storage area. When the decompression command is sent by the client, the decompression process can send the total size of the received data fragments and the second check data to the corresponding receiving process. The receiving process generates the third check data of each data fragment transmitted by the client according to the preset check algorithm. If the second check data and the third check data are consistent, and the total size of each data fragment transmitted by the client is consistent with the total size of each data fragment sent by the decompression process, then the received data fragments are saved to the second storage area.

[0082] Therefore, by sending the total size of each received data fragment and the second verification data to the server or receiving process for verification, the accuracy of each received data fragment is ensured, thereby ensuring the validity of the generated first index.

[0083] In one implementation, the first compressed data can be any one of multiple compressed data sets included in the compressed data set; correspondingly, the method may further include:

[0084] In response to receiving the description information of the compressed data set, the index associated with multiple data identifiers in the description information is obtained from each index stored in the second storage area, wherein the multiple data identifiers correspond one-to-one with multiple compressed data, and each data identifier is associated with multiple indexes; the indexes associated with the multiple data identifiers are merged to obtain the index of the compressed data set.

[0085] Specifically, after confirming that all compressed data in the compressed data set has been transmitted, the sender transmits the description information of the compressed data set to the decompression device. Upon receiving the description information, the decompression device retrieves multiple data identifiers from the description information, obtains multiple associated indexes from the second storage area based on each data identifier, and merges the retrieved indexes according to a preset merging method to obtain the index of the compressed data set. The description information may also include the decompression order of the multiple compressed data in the compressed data set. The preset merging method may be intersection-merge, union-merge, etc. For the specific processing of intersection-merge and union-merge, please refer to relevant technologies; these will not be detailed here. In particular, when the compressed data set is a container image, the first compressed data is any image layer in the container image, and the description information may be an image description file.

[0086] It should be noted that this application does not impose specific limitations on the content and form of the descriptive information, as well as the merging method; these can be set as needed in practical applications.

[0087] Therefore, when the first compressed data is any one of the multiple compressed data included in the compressed data set, by receiving the description information of the compressed data set and retrieving the indexes associated with multiple data identifiers in the description information from the indexes stored in the second storage area, the accuracy of the retrieved indexes is ensured, thereby ensuring the accuracy of the generated index of the compressed data set. Furthermore, since each compressed data in the compressed data set generates its corresponding index during the upload and decompression process, rather than decompressing and generating indexes after obtaining the complete compressed data set, the index generation cycle of the compressed data set is greatly shortened, improving the efficiency of index generation.

[0088] To improve the processing efficiency of decompressed data, in one implementation, when the sender has multiple compressed data sets to transmit to the decompression device for decompression and corresponding index generation, the sender can also transmit multiple compressed data sets to the decompression device in parallel. That is, the decompression command also targets a second compressed data set, and the decompression command may also include a second data identifier for the second compressed data set. Correspondingly, the method further includes:

[0089] The second data fragment is decompressed in parallel to obtain the second decompressed data, wherein the second data fragment is the data fragment to be decompressed among the multiple data fragments included in the second compressed data; according to the second offset information corresponding to the second decompressed data, a second index of the second decompressed data in the second compressed data is generated, wherein the second index is used to access the second decompressed data when the second compressed data is in an undecompressed state, and the second offset information includes a third offset, which is used to indicate the position of the second data fragment in the second compressed data.

[0090] In other words, in response to decompression commands for the first compressed data and the second compressed data, the decompression device decompresses the first data fragment and the second data fragment in parallel to obtain the corresponding first decompressed data and the second decompressed data; and, in parallel, generates a first index of the first decompressed data in the first compressed data and a second index of the second decompressed data in the second compressed data based on the first offset information corresponding to the first decompressed data and the second offset information corresponding to the second decompressed data. The process of decompressing the second data fragment to obtain the second decompressed data and generating the second index of the second decompressed data in the second compressed data is the same as the aforementioned process of decompressing the first data fragment to obtain the first decompressed data and generating the first index of the first decompressed data in the first compressed data, as described above; the repetition will not be repeated here.

[0091] The second compressed data can be one or more, meaning the sender can transmit at least two compressed data sets to the decompression device simultaneously. This is illustrated by the sender acting as the client and transmitting N compressed data sets in parallel, where N is an integer greater than or equal to 2. Figure 4As shown, after determining the compressed data 1, compressed data 2... compressed data N to be uploaded in parallel, the client sends a decompression command to the decompression device according to the data identifier of each compressed data. In response to the decompression command, the decompression device assigns receiving process 1 and decompression process 1 to compressed data 1, receiving process 2 and decompression process 2 to compressed data 2,... and receiving process N and decompression process N to compressed data N, and sends the communication addresses of each receiving process to the client. In response to the received communication addresses, the client assigns uploading process 1 to compressed data 1, uploading process 2 to compressed data 2,... and uploading process N to compressed data N. Any uploading process (e.g., uploading process 1) sends a connection request to the receiving process according to the communication address of the corresponding receiving process (e.g., receiving process 1). Any receiving process (e.g., receiving process 1) establishes a connection with the corresponding uploading process (e.g., uploading process 1) according to the received connection request, and after successful connection, sends a second data transmission request to the corresponding uploading process (e.g., uploading process 1) through the established connection according to the data identifier of the corresponding compressed data (e.g., compressed data 1). In response to the second data transmission request, any uploading process (e.g., receiving process 1) transmits a data fragment (e.g., a fragment of compressed data 1) to the receiving process (e.g., receiving process 1) via a corresponding connection. The receiving process (e.g., receiving process 1) sends the received data fragment to the corresponding decompression process (e.g., decompression process 1). Upon receiving the data fragment, any decompression process (e.g., decompression process 1) decompresses the data as described above and generates an index for the decompressed data. It saves the metadata generated based on the decompressed data and the index to the first storage area. After generating the metadata for the last data fragment (e.g., the last data fragment of compressed data 1), it associates the metadata of the last data fragment and each metadata fragment in the first storage area with the corresponding data identifier (e.g., the data identifier of compressed data 1) and saves it to the second storage area, then deletes all metadata fragments from the first storage area. Any decompression process (e.g., decompression process 1) can also determine the second verification data for each received data fragment according to a preset verification algorithm, and send the total size of each received data fragment and the second verification data to the corresponding receiving process (e.g., receiving process 1). After receiving the total size of each data fragment and the second verification data sent by the decompression process, any receiving process (e.g., receiving process 1) generates the third verification data of each data fragment transmitted by the uploading process (e.g., uploading process 1) according to a preset verification algorithm. If the second verification data and the third verification data are consistent, and the total size of each data fragment transmitted by the uploading process (e.g., uploading process 1) is consistent with the total size of each data fragment sent by the decompression process, the received data fragments are saved to the second storage area.

[0092] Furthermore, when the client transmits N compressed data items that are compressed data within a compressed data set, and it is necessary to generate an index for the compressed data set, such as... Figure 4 As shown, after all compressed data in the compressed data set has been transmitted, the client can use any uploading process (e.g., uploading process 2) to transmit the description information of the compressed data set to the corresponding receiving process (e.g., receiving process 2). The receiving process (e.g., receiving process 2) then sends the description information to the corresponding decompression process (e.g., decompression process 2). The decompression process (e.g., decompression process 2) retrieves the associated metadata from the second storage area based on the description information and creates an index for the compressed data set based on the retrieved metadata. Alternatively, the receiving process (e.g., receiving process 2) sends the description information to a preset index generation process (…). Figure 4 (Not shown in the image), the index generation process retrieves the associated metadata from the second storage area based on the description information, and generates an index for the compressed dataset based on the retrieved metadata.

[0093] Therefore, the sending direction sends decompression commands to the decompression device for the first compressed data and the second compressed data, and transmits the first compressed data and the second compressed data to the decompression device in parallel. The decompression device processes the first compressed data and the second compressed data in parallel, which greatly improves the data processing efficiency.

[0094] Considering that decompression may be interrupted during the decompression process in practical applications, to avoid re-executing previously performed decompression operations after an interruption, in one embodiment, the method may further include: when a data fragment is currently received but no transmission end marker is received, and no new data fragment is received within a preset time period, generating a decompression interruption record for the first compressed data based on a third size, a fourth size, the size of the decompressed compressed data in the last decompressed data fragment, and the undecompressed data in the last decompressed data fragment, and associating the decompression interruption record with the first data identifier of the first compressed data in a second storage area. Here, the third size is the total size of the compressed data received at the time of the decompression interruption, and the fourth size is the total size of the decompressed dataset at the time of the decompression interruption.

[0095] Furthermore, since decompression interruptions are often caused by factors such as abnormal communication links between the sender and the decompression device, or sender malfunctions, which prevent the decompression device from continuing to receive data fragments, decompression is interrupted. Also, the transmission speed of data fragments may exceed the decompression speed. Therefore, when a decompression interruption occurs, if the decompression device determines that there are received data fragments that are not yet decompressed, it can continue decompressing those fragments and generate a decompression interruption record based on the relevant information of the last decompressed data fragment.

[0096] Correspondingly, if the first data fragment is the first data fragment to be decompressed corresponding to the decompression instruction, before decompressing the first data fragment to obtain the first decompressed data, the process may further include:

[0097] If the second storage area contains a decompression interruption record corresponding to the first data identifier, the third size is obtained from the decompression interruption record, wherein the decompression interruption record is generated when a decompression interruption occurs during any historical decompression of the first compressed data; a first data transmission request is sent to the sender of the decompression instruction according to the third size, the first data transmission request is used to request the sender to transmit the data to be transmitted after the third size; the first data fragment is determined according to the data to be transmitted transmitted by the sender.

[0098] In this process, when the sender of the decompression command receives the first data transmission request, it can retrieve the third-sized compressed data from the first compressed data in a sequential order, and determine the data to be transmitted after the third size in the first compressed data, i.e., the data fragments to be transmitted after the third size. Based on the arrangement of these data fragments in the first compressed data, the sender then transmits each data fragment to the decompression device sequentially. The decompression device determines the first data fragment to be decompressed based on the receiving order of the data fragments.

[0099] In one implementation, determining the data fragments to be transmitted in the first compressed data that are located after the third size may include: if the target end position corresponding to the third size is determined to be the middle position of the last transmitted data fragment, the sender of the decompression command may divide the compressed data in the first compressed data that is located after the target end position according to the preset size of each data fragment to obtain the data fragments to be transmitted. The last data fragment to be transmitted may be smaller than the preset size; alternatively, if the last data fragment to be transmitted is smaller than the preset size, preset data may be added to the last data fragment to make it reach the first size.

[0100] In another implementation, determining the data fragment to be transmitted after the third size in the first compressed data may include: if the target end position corresponding to the third size is determined to be the middle position of the last transmitted data fragment, the sender of the decompression command may obtain the untransmitted portion of compressed data in the last data fragment and determine this portion of compressed data as the first data fragment to be transmitted, or fill this portion of compressed data with preset data to a preset size according to a preset rule and then determine it as the first data fragment to be transmitted; and, determining the data fragments located after the last data fragment as data fragments to be transmitted. When the preset data is filled with preset data according to the preset rule, when the decompression device receives the corresponding data fragment, it can parse the data fragment according to the preset rule to obtain the portion of compressed data. For example, the preset rule may be to fill the data fragment with preset data after the portion of compressed data. Accordingly, after receiving the data fragment, if the decompression device determines that the data fragment includes the preset data, it obtains the data located before the preset data from the data fragment to obtain the portion of compressed data.

[0101] Furthermore, the above-mentioned determination of the data fragments to be transmitted after the third size in the first compressed data may also include: when determining that the target end position corresponding to the third size is the end position of the last transmitted data fragment, determining each data fragment after the last data fragment as the data fragment to be transmitted.

[0102] It should be noted that when the application scenario of this application embodiment is... Figure 1BIn the illustrated application scenario, if decompression is interrupted due to factors such as an abnormal communication link between the server and the decompression device, while the communication link between the client and the server remains normal, the server can, upon determining that the first decompression recovery condition is met (e.g., the communication link with the decompression device has returned to normal), send a decompression command for the first compressed data to the decompression device based on the data identifier of the first compressed data. Upon receiving the first data transmission request from the decompression device, the server processes it in the aforementioned manner. If decompression is interrupted due to an abnormal communication link between the server and the client, or due to a client malfunction, the client can also record the transmission failure position of the first compressed data. Upon determining that the second decompression recovery condition is met (e.g., the communication link with the server has returned to normal, or the client has recovered), the client transmits each data fragment and the first data identifier of the first compressed data to the server, starting from the data fragment where the transmission failure position is located, according to the order of the data fragments in the first compressed data. When the server receives the data fragment and the first data identifier transmitted by the client, it sends a decompression instruction for the first compressed data to the decompression device according to the first data identifier. When it receives the first data transmission request sent by the decompression device, it determines at least one data fragment located after the third size in the first compressed data from the data fragments currently transmitted by the client in the manner described above, and transmits it to the decompression device.

[0103] Furthermore, when the decompression device determines that the decompression interruption record includes undecompressed data in the last decompressed data segment when the decompression interruption occurred, the method further includes: obtaining the undecompressed data from the decompression interruption record, determining the data to be decompressed based on the undecompressed data and the first data segment, and decompressing the data to be decompressed to obtain the corresponding first decompressed data.

[0104] Furthermore, after the decompression device sends the first data transmission request to the sender, in order to facilitate the subsequent generation of the index, the method also includes: obtaining the third size and the fourth size from the decompression interruption record, and determining the third size as the current first size and the fourth size as the current second size.

[0105] As described above, the decompression device runs at least one decompression process. When a decompression interruption occurs during the decompression of the first compressed data, a decompression interruption record of the first compressed data can be generated through the decompression process. The decompression interruption record is then associated with the first data identifier of the first compressed data and saved to the first storage area and the second storage area, respectively. Since the decompression process allocated to the first compressed data may differ before and after the decompression interruption, in order to ensure the accuracy of decompression recovery without increasing the access pressure on the second storage area, in one embodiment, when the second storage area contains a decompression interruption record corresponding to the first data identifier, obtaining the third size from the decompression interruption record may include: when the second storage area contains a decompression interruption record corresponding to the first data identifier, determining whether the decompression interruption record stored in the second storage area is the same as the decompression interruption record stored in the first storage area; if they are the same, then obtaining the third size from the interruption record stored in the first storage area; if they are different, then replacing the decompression interruption record stored in the first storage area with the decompression interruption record stored in the second storage area, and then obtaining the third size from the decompression interruption record stored in the first storage area.

[0106] Therefore, during the decompression of the first compressed data, when a decompression interruption occurs, a decompression interruption record is generated. Upon restarting the decompression of the first compressed data, a first data transmission request is sent to the sender based on the third size in the decompression interruption record. This allows the sender of the decompression command to transmit the data to be transmitted after the third size in the first compressed data. This allows decompression of the data to be decompressed after the interruption point to continue from the interruption point without starting decompression from the beginning, avoiding repeated execution of decompression operations, shortening decompression time, and improving decompression efficiency.

[0107] In short, such as Figure 5As shown, in response to a decompression command sent by the sender, the decompression device allocates a corresponding processing process for any compressed data in at least one of the compressed data to be processed and establishes a connection with the sender's upload process. The processing process determines whether a corresponding decompression interruption record exists in the second storage area based on the data identifier of the corresponding compressed data. If a corresponding decompression interruption record exists in the second storage area, it is determined that the corresponding compressed data is not being decompressed for the first time, and it checks whether the decompression interruption record exists in the first storage area. If the decompression interruption record exists in the first storage area, a first data transmission request is sent to the upload process based on the decompression interruption record in the first storage area. If the decompression interruption record does not exist in the first storage area, the decompression interruption record in the first storage area is updated based on the decompression interruption record in the second storage area, and then a first data transmission request is sent to the upload process based on the decompression interruption record in the first storage area. If no corresponding decompression interruption record exists in the second storage area, it is determined that the corresponding compressed data is being decompressed for the first time, and a second data transmission request is sent to the upload process based on the data identifier of the compressed data, and the device waits to receive data fragments transmitted by the upload process. The process determines whether transmission is interrupted. If interrupted, a decompression interruption record is generated and saved to the first and second storage areas respectively, and the process ends. If transmission is not interrupted, the process receives the data fragments and offsets transmitted by the uploading process, and after verifying the offsets are correct, saves the corresponding data fragments and offsets to the first storage area. The process retrieves the currently uncompressed data fragments from the first storage area and decompresses them to obtain decompressed data. An index is generated based on the offset information corresponding to the decompressed data, written into metadata, and saved to the first storage area. The process determines whether all data fragments of the corresponding compressed data have been processed. If so, each metadata element and its corresponding data identifier are saved to the second storage area. Furthermore, if at least one compressed data element to be processed belongs to a compressed data set, upon receiving the description information of the compressed data set transmitted by the uploading process, the associated metadata is retrieved from the second storage area based on the description information, and the indexes in the retrieved metadata are merged to obtain the index of at least one compressed data element. It is evident that the process generates the corresponding index of the decompressed data while transmitting compressed data, rather than decompressing and generating the index only after receiving at least one compressed data element, thus greatly improving the index generation efficiency. It should be noted that when the processing process includes a receiving process and a decompression process, the offset verification operation and the corresponding saving of data fragments and offsets to the first storage area can be performed by either the receiving process or the decompression process.

[0108] Figure 5 A flowchart of a compressed data processing method 500 according to an embodiment of this application is shown. This method can be applied to the sender, for example, by... Figure 1A Client 110 executes or Figure 1B The server 120 executes, such as Figure 5 As shown, the method may include steps S501 and S502.

[0109] Step S501: In response to the condition for decompressing the first compressed data being met, a decompression command for the first compressed data is sent to the decompression device.

[0110] Specifically, when the sender is a client, if the first compressed data is being decompressed for the first time, the client determines that the decompression conditions for the first compressed data are met upon receiving the data transmission instruction input by the user. If the first compressed data is not being decompressed for the first time, i.e., a decompression interruption has occurred, considering that such interruptions are often caused by factors such as abnormal communication links between the client and the decompression device or client malfunctions, the client can determine that the decompression conditions for the first compressed data are met when it detects that the communication link between the client and the decompression device has returned to normal or that the client is in a normal state. Based on information such as the data identifier of the first compressed data, the client then sends a decompression instruction to the decompression device for the first compressed data.

[0111] When the sender is the server, if the first compressed data is being decompressed for the first time, the server determines that the decompression conditions for the first compressed data are met upon receiving the data fragments transmitted by the client. If the first compressed data is not being decompressed for the first time, i.e., a decompression interruption occurred, considering that the decompression interruption may have been caused by factors such as an abnormal communication link between the server and the decompression device, but the communication link between the server and the client is normal, meaning the server can receive data transmitted by the client normally, the server, upon determining that the communication link with the decompression device has been restored, determines that the decompression conditions for the first compressed data are met and sends a decompression command for the first compressed data to the decompression device based on information such as the data identifier of the first compressed data. Decompression interruption may also be caused by factors such as an abnormal communication link between the server and the client, or by client failure. In this case, upon determining that the communication link with the server has been restored or that the client has recovered, the client can, based on the transmission failure position of the first compressed data recorded by the client, start transmitting each data fragment and the data identifier of the first compressed data from the data fragment at the transmission failure position to the server. Accordingly, when the server receives the data transmitted by the client, it determines that the decompression conditions of the first compressed data are met, and sends a decompression command for the first compressed data to the decompression device based on the data identifier and other information of the first compressed data.

[0112] Step S502: Receive the first decompression data and the first index sent by the decompression device, wherein the first decompression data is obtained by decompressing the data segment to be decompressed among the multiple data segments included in the first compressed data, and the first index is generated based on the first offset information corresponding to the first decompression data.

[0113] The first index is used to access the first decompressed data when the first compressed data is in an undecompressed state. The specific implementation processes of steps S501 and S502 can be found in the preceding descriptions; details that are repeated will not be repeated here.

[0114] According to the data decompression processing method provided in the embodiments of this application, after decompressing the current data fragments to be decompressed to obtain the first decompressed data, a first index of the first decompressed data in the first compressed data is generated based on the first offset information corresponding to the first decompressed data. Therefore, the index of the decompressed data is generated during the decompression process of the first compressed data, rather than generating the index after obtaining the complete first compressed data, which consumes additional time. This shortens the overall time spent generating the index and improves the efficiency of index generation.

[0115] Corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide a compressed data processing apparatus, including: a decompression module, configured to decompress a first data segment to obtain first decompressed data in response to a decompression command for first compressed data, wherein the first data segment is the data segment currently to be decompressed among a plurality of data segments included in the first compressed data; and a generation module, configured to generate a first index of the first decompressed data in the first compressed data according to first offset information corresponding to the first decompressed data, wherein the first index is used to access the first decompressed data when the first compressed data is in an undecompressed state, and the first offset information includes a first offset, the first offset being used to indicate the position of the first data segment among the plurality of data segments.

[0116] In one embodiment, the first data fragment and the first offset are obtained from a first storage area, and the apparatus further includes a verification module and a first storage module. The verification module is configured to, in response to receiving the first data fragment and the first offset, verify whether the first offset is correct based on a first size, wherein the first size is the total size of the currently received data fragments. The first storage module is configured to, if the first offset is correct, associate and save the first data fragment and the first offset to the first storage area.

[0117] In one embodiment, the first offset information further includes a second offset, the second offset being used to indicate the position of the first decompressed data in the current decompressed dataset of the first compressed data, and the apparatus further includes: a first determining module, used to determine the second offset according to a second size before the generating module generates a first index of the first decompressed data in the first compressed data, wherein the second size is the total size of the current decompressed dataset when the current decompressed dataset does not contain the first decompressed data.

[0118] In one embodiment, the apparatus further includes: an update module, configured to update the first size according to the size of the first data fragment; and update the second size according to the size of the first decompressed data.

[0119] In one embodiment, the apparatus further includes: a second storage module, configured to, after the generation module generates a first index of the first decompressed data in the first compressed data, save the first index to a first storage area if the first data fragment is not the last data fragment among the plurality of data fragments; or, if the first data fragment is the last data fragment among the plurality of data fragments, associate the first index and the plurality of indexes stored in the first storage area with a first data identifier of the first compressed data and save them to a second storage area, and delete the plurality of indexes in the first storage area.

[0120] In one embodiment, the first compressed data is any one of a plurality of compressed data included in the compressed data set, and the apparatus further includes: a first acquisition module and a merging module. The first acquisition module is configured to, in response to receiving description information of the compressed data set, acquire, from each index stored in the second storage area, an index associated with a plurality of data identifiers in the description information, wherein the plurality of data identifiers correspond one-to-one with the plurality of compressed data, and each data identifier is associated with a plurality of indexes. The merging module is configured to merge the indexes associated with the plurality of data identifiers to obtain an index of the compressed data set.

[0121] In one embodiment, the decompression command further targets the second compressed data, and the decompression module is further configured to decompress the second data fragment in parallel to obtain the second decompressed data, wherein the second data fragment is the data fragment currently to be decompressed among the multiple data fragments included in the second compressed data; the generation module is further configured to generate a second index of the second decompressed data in the second compressed data according to the second offset information corresponding to the second decompressed data, wherein the second index is used to access the second decompressed data when the second compressed data is in an undecompressed state, and the second offset information includes a third offset, the third offset being used to indicate the position of the second data fragment in the second compressed data.

[0122] In one embodiment, the decompression command includes a first data identifier of the first compressed data, and the device further includes a second acquisition module, a sending module, and a second determination module. The second acquisition module is configured to, when the first data fragment is the first data fragment to be decompressed corresponding to the decompression command, and when the second storage area contains a decompression interruption record corresponding to the first data identifier, acquire a third size from the decompression interruption record. The decompression interruption record is generated when a decompression interruption occurs during any historical decompression of the first compressed data, and the third size is the total size of the compressed data received at the time of the decompression interruption. The sending module is configured to send a first data transmission request to the sender of the decompression command based on the third size. The first data transmission request requests the sender to transmit data to be transmitted after the third size. The second determination module is configured to determine the first data fragment based on the data to be transmitted transmitted by the sender.

[0123] Corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide a compressed data processing apparatus, including: a sending module, used to send a decompression command for the first compressed data to a decompression device in response to satisfying the decompression conditions of the first compressed data; and a receiving module, used to receive the first decompressed data and a first index sent by the decompression device, wherein the first decompressed data is obtained by decompressing the data segment currently to be decompressed among the multiple data segments included in the first compressed data, and the first index is generated based on the first offset information corresponding to the first decompressed data, and the first index is used to access the first decompressed data when the first compressed data is in an undecompressed state.

[0124] The functions of each module in each device in the embodiments of this application can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.

[0125] Figure 6This is a block diagram of an electronic device used to implement embodiments of this application. For example... Figure 6 As shown, the electronic device includes a memory 601 and a processor 602. The memory 601 stores a computer program that can run on the processor 602. When the processor 602 executes the computer program, it implements the method described in the above embodiments. The number of memories 601 and processors 602 can be one or more. In a specific implementation, the electronic device may also include a communication interface 603 for communicating with external devices and exchanging data.

[0126] In practical implementation, if the memory 601, processor 602, and communication interface 603 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0127] Optionally, in a specific implementation, if the memory 601, processor 602 and communication interface 603 are integrated on a single chip, the memory 601, processor 602 and communication interface 603 can communicate with each other through an internal interface.

[0128] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0129] This application provides a computer program product, including a computer program that, when executed by a processor, implements the method provided in this application.

[0130] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0131] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0132] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0133] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0134] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0135] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0136] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0137] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0138] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0139] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0140] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0141] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A processing method of compressed data, applied to a decompression device, comprising: in response to a decompression instruction for first compressed data, decompressing a first data segment to obtain first decompressed data, wherein the first data segment is a data segment currently to be decompressed among a plurality of data segments included in the first compressed data; generating a first index of the first decompressed data in the first compressed data according to first offset information corresponding to the first decompressed data, wherein the first offset information comprises a first offset, and the first offset is used to indicate a position of the first data segment in the plurality of data segments.

2. The method of claim 1, wherein, The first data segment and the first offset are obtained from a first storage area, and the method further comprises: in response to receiving the first data segment and the first offset, verifying whether the first offset is correct according to a first size, wherein the first size is a total size of data segments currently received; in a case where the first offset is correct, associatively saving the first data segment and the first offset to the first storage area.

3. The method of claim 1, wherein the first offset information further comprises a second offset, and the second offset is used to indicate a position of the first decompressed data in a current decompressed data set of the first compressed data, and the method further comprises, before the generating the first index of the first decompressed data in the first compressed data according to the first offset information corresponding to the first decompressed data: determining the second offset according to a second size, wherein the second size is a total size of the current decompressed data set in a case where the current decompressed data set does not contain the first decompressed data.

4. The method of claim 1, and the method further comprises, after the generating the first index of the first decompressed data in the first compressed data: in a case where the first data segment is not a last data segment in the plurality of data segments, saving the first index to a first storage area; or, in a case where the first data segment is the last data segment in the plurality of data segments, associatively saving the first index and a plurality of indexes stored in the first storage area to a second storage area with a first data identifier of the first compressed data, and deleting the plurality of indexes in the first storage area.

5. The method of claim 4, wherein the first compressed data is any one of a plurality of compressed data included in a compressed data set, and the method further comprises: in response to receiving description information of the compressed data set, obtaining indexes associated with a plurality of data identifiers in the description information from the indexes stored in the second storage area, wherein the plurality of data identifiers correspond to the plurality of compressed data one by one, and each data identifier is associated with a plurality of indexes; merging the indexes associated with the plurality of data identifiers to obtain an index of the compressed data set.

6. The method of claim 1, wherein the decompression instruction is also for second compressed data, and the method further comprises: decompress the second data segment in parallel to obtain second decompressed data, wherein the second data segment is a data segment currently to be decompressed among a plurality of data segments included in the second compressed data; generate a second index of the second decompressed data in the second compressed data according to second offset information corresponding to the second decompressed data, wherein the second index is used to access the second decompressed data in a case that the second compressed data is in an uncompressed state, and the second offset information includes a third offset amount used to indicate a position of the second data segment in the second compressed data.

7. The method of any one of claims 1-6, wherein the decompression instruction includes a first data identifier of the first compressed data, and in a case that the first data segment is a first data segment to be decompressed corresponding to the decompression instruction, the decompressing the first data segment to obtain first decompressed data further includes: in a case that a decompression interruption record corresponding to the first data identifier is included in a second storage area, obtaining a third size from the decompression interruption record, wherein the decompression interruption record is generated in a process of decompressing the first compressed data in a historical time, and the third size is a total size of compressed data received at a time of the decompression interruption; sending a first data transmission request to a sender of the decompression instruction according to the third size, wherein the first data transmission request is used to request the sender to transmit to-be-transmitted data located after the third size; and determining the first data segment according to the to-be-transmitted data transmitted by the sender.

8. A processing method of compressed data, applied to a sender, and comprising: sending a decompression instruction of first compressed data to a decompression device in response to a decompression condition of the first compressed data being satisfied; and receiving first decompressed data and a first index sent by the decompression device, wherein the first decompressed data is obtained by decompressing a data segment currently to be decompressed among a plurality of data segments included in the first compressed data, and the first index is generated according to first offset information corresponding to the first decompressed data.

9. A processing system of compressed data, comprising: a client configured to send a decompression instruction for first compressed data; and a decompression device configured to receive the decompression instruction and implement the method of any one of claims 1-7.

10. A processing system of compressed data, comprising: a client configured to transmit first compressed data to a server; the server configured to receive the first compressed data and send a decompression instruction for the first compressed data; and a decompression device configured to receive the decompression instruction and implement the method of any one of claims 1-7.

11. An electronic device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor implements the method of any one of claims 1-8 when executing the computer program. ​ ​ ​ ​ ​ ​ ​ 12. A computer readable storage medium having stored therein a computer program which, when executed by a processor, implements the method of any one of claims 1 to 8.

13. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images