Data compression method and device

Abstract

The data compression method comprises the steps that a first file is obtained, the first file comprises at least one first section and at least one second section, the first section is a section containing holes or a plurality of continuous zeros in the first file, and the second section is a section for storing data blocks in the first file; and compressing the at least one first section in the first file to generate a second file, the second file comprising at least one third section and at least one fourth section, the third section comprising information for indicating to compress the at least one first section, and the fourth section comprising the data block. According to the method, the holes or the continuous zeros in the file are compressed, so that the file storage space is reduced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data compression method and device. Background Art

[0002] With the development of information technology, the amount of data transmitted between devices is increasing. Therefore, data transmission requires a large bandwidth and takes up a lot of space when stored. In file operations of some file systems, the file displacement can be greater than the current length of the file. If the displacement is greater than the current length of the file, the next write operation will "extend" the file and create a "hole" in the file. Such files with holes are called sparse files. Sparse files are basically the same as other ordinary files. The difference is that part of the content in the file is a hole, and the hole does not occupy disk space in a system that supports sparse files. Most modern file systems support the "sparse file" format, such as the Apple file system (APFS) and the fourth extended file system (ext4). When a sparse file is copied in a file system, it can be specified whether it is still stored as a sparse file. However, when it is copied across file systems or transmitted on the network, its existing sparse file information will be lost, so that the hole actually occupies storage space. This situation not only leads to a large amount of storage space being wasted, but also consumes a lot of bandwidth during network transmission. Summary of the invention

[0003] The present application provides a data compression method, which compresses holes or multiple consecutive zeros in a file, thereby reducing storage space and bandwidth consumption during data transmission.

[0004] In a first aspect, the present application provides a method for data compression, the method comprising: obtaining a first file, the first file comprising at least one first segment and at least one second segment, wherein the first segment is a segment in the first file that includes a hole or multiple consecutive zeros, and the second segment is a segment in the first file that stores a data block; compressing at least one first segment in the first file to generate a second file, the second file comprising at least one third segment and at least one fourth segment, the third segment comprising information for indicating that the at least one first segment is compressed, and the fourth segment comprises a data block.

[0005] In an embodiment of the present application, the first segment in the first file is compressed to generate a second file, thereby saving the storage space occupied by holes with no valid information or multiple consecutive zeros in the file, reducing the resource occupancy and power consumption on the terminal side, reducing the bandwidth required for data transmission, and improving transmission efficiency.

[0006] In one possible implementation, compressing at least one first segment in a first file to generate a second file includes: obtaining at least one first position of at least one first segment in the first file, the at least one first position corresponding one-to-one to the position of at least one first segment in the first file; writing the at least one first position into a linked list, the linked list being used to indicate the position of at least one first segment in the first file; writing the linked list into the second file; and writing a data block in at least one second segment into the second file.

[0007] In this implementation, the positions of holes or consecutive zeros in the first file before compression are saved in the compressed second file using a linked list. The linked list occupies a small storage overhead, reduces the resource occupation and power consumption on the terminal side, and reduces the compression rate. At the same time, it reduces the bandwidth required for data transmission and improves transmission efficiency. And the centralized storage in the linked list facilitates the second file after compression to recover the first file before compression according to the linked list information, which is convenient for managing and reading the position information of holes or consecutive zeros, and reduces the resource overhead of repeated reading and writing.

[0008] In one possible implementation, the first position includes a first offset and a first length, and obtaining at least one first position of at least one first segment in the file to be sent includes: opening the first file; obtaining at least one first offset and first length of at least one first segment in the first file, and the at least one first offset and first length correspond one-to-one to the at least one first segment.

[0009] In this implementation, the offset and size of the hole or continuous multiple zeros in the first file before compression are saved in the compressed second file by using a linked list, and the offset and size as the position information of the hole or continuous multiple zeros can be obtained by calling a function or directly obtained by a mark in the file, which is convenient to obtain and reduces the computational complexity. And according to the offset and size of each section of the hole or continuous multiple zeros, it is convenient to restore the compressed second file to the first file before compression according to the linked list information, and at the same time, the storage overhead occupied by the linked list is small, which saves storage resources and reduces the bandwidth required for data transmission.

[0010] In one possible implementation, writing at least one first position into a linked list includes: generating an i-th node of the linked list based on an i-th first offset and an i-th first length; writing the i-th node into the linked list; pointing a first pointer of the linked list to the i-th node; generating an i+1-th node of the linked list based on an i+1-th first offset and an i+1-th first length; writing the i+1-th node into the linked list; and linking the i-th node and the i+1-th node according to a second pointer, where i is a positive integer.

[0011] In this embodiment, the offset and size of each hole or multiple consecutive zeros are written into each node of the linked list in turn and linked with a second pointer, thereby reducing computational complexity, improving processing efficiency, and facilitating the restoration of the compressed second file to the first file before compression according to the information of each node in the linked list.

[0012] In a possible implementation, writing the linked list into the second file includes: writing the linked list into a third section of the second file, where the third section includes a file header of the second file.

[0013] In a possible implementation, writing the data in the at least one second section into the second file includes:

[0014] The data blocks in the at least one second section are written into a fourth section of the second file, where the fourth section includes a file body of the sent file.

[0015] In this implementation, a linked list storing the offsets and sizes of holes or consecutive zeros in the file before compression is written into the file header of the compressed file, and the data is written into the file body of the compressed file. This reduces storage overhead and reduces the bandwidth required for data transmission. It also facilitates file management and reading, reduces computational complexity, and facilitates recovery of the first file before compression.

[0016] In a possible implementation, the method further includes: sending the second file to the receiving device; or sending the encrypted second file to the receiving device.

[0017] In this implementation, the second file compressed with holes or multiple consecutive zeros is sent to the receiving device, saving bandwidth required for data transmission. Optionally, the encrypted second file can also be sent to the receiving device to improve the security of data transmission and protect the privacy of the user.

[0018] In a second aspect, the present application provides a method for decompressing data, the method comprising: obtaining a second file; the second file comprising at least one third section and at least one fourth section, the third section comprising information for indicating that at least one first section in the first file is compressed, and the fourth section comprising a data block in the first file. Decompressing the second file to generate a first file, the first file comprising at least one first section and at least one second section, wherein the first section is a section in the first file that includes a hole or multiple consecutive zeros, and the second section is a section in the first file that stores a data block.

[0019] In the embodiment of the present application, the compressed second file is restored to the first file before compression, so that the user can view and use it. The compressed second file saves the storage space occupied by holes or multiple consecutive zeros without valid information in the file, reduces the resource occupation and power consumption of the terminal side, reduces the bandwidth required for data transmission, and improves the transmission efficiency.

[0020] In one possible implementation, decompressing the second file to generate the first file includes: obtaining a linked list in the second file, the linked list being used to indicate the position of at least one first segment in the first file, the first segment being a segment in the first file containing a hole or multiple consecutive zeros; determining the position of at least one first segment in the first file according to at least one first position in the linked list; at least one first position having a one-to-one correspondence with the position of at least one first segment in the first file; writing the hole or multiple consecutive zeros into at least one first segment in the first file; obtaining at least one data block in the second file; and writing at least one data block into at least one second segment in the first file to generate the second file, the second segment being a segment storing data blocks in the first file.

[0021] In this implementation, the positions of holes or consecutive zeros in the first file before compression are saved in the compressed second file using a linked list. The linked list occupies a small storage overhead, reduces the resource occupation and power consumption on the terminal side, and reduces the compression rate. The bandwidth required for data transmission is reduced, and the transmission efficiency is improved. In addition, the concentrated storage in the linked list facilitates the recovery of the compressed second file according to the linked list information to obtain the first file before compression, which is convenient for managing and reading the position information of holes or consecutive zeros, and reduces the resource overhead of repeated reading and writing.

[0022] In one possible implementation, the first position includes a first offset and a first length, and determining the position of at least one first segment in the first file based on at least one first position in the linked list includes: obtaining at least one first offset and first length in the first linked list, at least one first offset and first length corresponding one-to-one to at least one first segment; determining the position of at least one first segment in the first file based on at least one first offset and first length in the linked list.

[0023] In this embodiment, the offset and size of the hole or continuous multiple zeros in the first file before compression are saved in the second file after compression by a linked list. The offset and size as the position information of the hole or continuous multiple zeros can be obtained by calling a function or directly obtained through a mark in the file, which is convenient for acquisition and reduces the complexity of calculation. The storage overhead occupied by the linked list is small, storage resources are saved, and the bandwidth required for data transmission is reduced. And according to the offset and size of each hole or continuous multiple zeros, it is convenient to restore the compressed second file to the first file before compression according to the linked list information. At the same time, the storage overhead occupied by the linked list is small, storage resources are saved, and the bandwidth required for data transmission is reduced.

[0024] In a possible implementation, obtaining the linked list in the second file includes: obtaining the first linked list from the third section of the second file, where the third section includes a file header of the compressed file.

[0025] In a possible implementation, obtaining at least one data block in the second file includes: obtaining a linked list from a fourth section of the second file, where the fourth section includes a file body of the second file.

[0026] In this implementation, a linked list storing the offsets and sizes of holes or consecutive zeros in the file before compression is written in the file header of the compressed file, and data blocks are written in the file body of the compressed file. This reduces storage overhead and reduces the bandwidth required for data transmission. It also facilitates file management and reading, and facilitates recovery of the first file before compression.

[0027] In a possible implementation, the method further includes: when a file system of a receiving device that receives the second file does not support sparse files, the receiving device stores the second file.

[0028] In this embodiment, when the file system of the receiving device that receives the second file does not support sparse files, the compressed second file is decompressed into the first file before compression, and the empty parts in the first file will occupy actual storage space. Therefore, when the user has no need to view and use it, the second file can be stored in the receiving device without decompressing it, thereby saving storage space and improving storage utilization.

[0029] In a possible implementation, the method further includes: receiving a second file from a sending device; or receiving an encrypted second file from the sending device, decrypting the encrypted second file, and obtaining a decrypted second file.

[0030] In this implementation, the compressed second file is received from the sending device, saving the bandwidth required for data transmission. Optionally, the encrypted second file can also be received from the receiving device, and the encrypted second file is decrypted to obtain the decrypted second file, thereby improving the security of data transmission and protecting the privacy of the user.

[0031] In a third aspect, the present application provides a data device, comprising a module for executing the method in the first aspect or any possible implementation scheme of the first aspect.

[0032] Exemplarily, the device may include: an acquisition module, configured to acquire a first file, the first file including at least one first segment and at least one second segment, wherein the first segment is a segment in the first file including a hole or multiple consecutive zeros, and the second segment is a segment in the first file storing a data block;

[0033] The processing module is used to compress at least one first segment in the first file to generate a second file, wherein the second file includes at least one third segment and at least one fourth segment, wherein the third segment contains information indicating that the at least one first segment is compressed, and the fourth segment contains a data block.

[0034] In one possible implementation, compressing at least one first segment in a first file to generate a second file includes: obtaining at least one first position of at least one first segment in the first file, the at least one first position corresponding one-to-one to the position of at least one first segment in the first file; writing the at least one first position into a linked list, the linked list being used to indicate the position of at least one first segment in the first file; writing the linked list into the second file; and writing a data block in at least one second segment into the second file.

[0035] In one possible implementation, the first position includes a first offset and a first length, and obtaining at least one first position of at least one first segment in the file to be sent includes: opening the first file; obtaining at least one first offset and first length of at least one first segment in the first file, and the at least one first offset and first length correspond one-to-one to the at least one first segment.

[0036] In one possible implementation, writing at least one first position into a linked list includes: generating an i-th node of the linked list based on the i-th first offset and the i-th first length; writing the i-th node into the linked list; pointing the first pointer of the linked list to the i-th node; generating an i+1-th node of the linked list based on the i+1-th first offset and the i+1-th first length; writing the i-th node into the linked list; and linking the i-th node and the i+1-th node according to a second pointer, where i is a positive integer.

[0037] In a possible implementation, it is characterized in that writing the linked list into the second file includes: writing the linked list into a third section of the second file, and the third section includes a file header of the second file.

[0038] In a possible implementation, it is characterized in that writing data in at least one second segment into the second file includes: writing a data block in at least one second segment into a fourth segment of the second file, and the fourth segment includes a file body of the sent file.

[0039] In a possible implementation, it is characterized in that the processing module is further used to: send the second file to the receiving device; or send the encrypted second file to the receiving device.

[0040] It should be understood that the third aspect corresponds to the technical solution of the first aspect, so the beneficial effects achieved by each possible implementation method are similar and will not be repeated here.

[0041] In a fourth aspect, an embodiment of the present application provides a device for data decompression, which includes a module for executing the method described in the second aspect or any possible implementation scheme of the second aspect.

[0042] Exemplarily, the device may include: an acquisition module for acquiring a second file; the second file includes at least one third segment and at least one fourth segment, the third segment contains information for indicating compression of at least one first segment in the first file, and the fourth segment contains a data block.

[0043] The processing module is used to decompress the second file to generate a first file, wherein the first file includes at least one first segment and at least one second segment, wherein the first segment is a segment in the first file that includes a hole or multiple consecutive zeros, and the second segment is a segment in the first file that stores data blocks.

[0044] In one possible implementation, decompressing the second file to generate the first file includes: obtaining a linked list in the second file, the linked list being used to indicate the position of at least one first segment in the first file, the first segment being a segment in the first file containing a hole or multiple consecutive zeros; determining the position of at least one first segment in the first file according to at least one first position in the linked list; at least one first position having a one-to-one correspondence with the position of at least one first segment in the first file; writing the hole or multiple consecutive zeros into at least one first segment in the first file; obtaining at least one data block in the second file; and writing at least one data block into at least one second segment in the first file to generate the second file, the second segment being a segment storing data blocks in the first file.

[0045] In one possible implementation, the first position includes a first offset and a first length, and the position of at least one first segment in the first file is determined based on at least one first position in the linked list: at least one first offset and first length in the first linked list are obtained, and at least one first offset and first length correspond one-to-one to at least one first segment; based on at least one first offset and first length in the linked list, the position of at least one first segment in the first file is determined.

[0046] In a possible implementation, obtaining the linked list in the second file includes: obtaining the first linked list from the third section of the second file, where the third section includes a file header of the compressed file.

[0047] In a possible implementation, obtaining at least one data block in the second file includes: obtaining a linked list from a fourth section of the second file, where the fourth section includes a file body of the second file.

[0048] In a possible implementation, the processing module is further configured to: when a file system of a receiving device that receives the second file does not support sparse files, the receiving device stores the second file.

[0049] In a possible implementation manner, the processing module is further configured to: receive the second file from the sending device; or receive the encrypted second file from the sending device, decrypt the encrypted second file, and obtain the decrypted second file.

[0050] It should be understood that the technical effects that can be achieved in the fourth aspect can be specifically described by referring to the technical effects brought about by the above-mentioned second aspect or any possible implementation method of the second aspect, and will not be repeated here.

[0051] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor coupled to each other, the processor calling a program code stored in the memory to execute a method as described in the first aspect or any possible implementation of the first aspect.

[0052] It should be understood that the technical effects that can be achieved in the fifth aspect can be specifically described by referring to the technical effects brought about by the above-mentioned first aspect or any possible implementation method of the first aspect, and will not be repeated here.

[0053] In a sixth aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor coupled to each other, the processor calling a program code stored in the memory to execute a method as described in the second aspect or any possible implementation of the second aspect.

[0054] It should be understood that the technical effects that can be achieved in the sixth aspect can be specifically described by referring to the technical effects brought about by the above-mentioned second aspect or any possible implementation scheme of the second aspect, and will not be repeated here.

[0055] In a seventh aspect, an embodiment of the present application provides a file transmission system, the system comprising one or more first ends and one or more second ends, wherein the first end is used to execute the method of any one of claims 1 to 7; the second end is used to execute the method of any one of claims 8 to 14. The first end comprises the data compression device described in the third aspect and any possible implementation of the third aspect, the second end comprises the data decompression device described in the fourth aspect and any possible implementation of the fourth aspect, or the first end comprises the electronic device described in the fifth aspect, and the second end comprises the electronic device described in the sixth aspect.

[0056] It should be understood that the technical effects that can be achieved in the seventh aspect can be specifically referred to the description of the technical effects brought about by the above-mentioned first aspect or any possible implementation method of the first aspect, and the description of the technical effects brought about by the above-mentioned second aspect or any possible implementation method of the second aspect, and will not be repeated here.

[0057] In an eighth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising a program or instructions, which, when executed on a computer, causes the method in the first aspect or any possible implementation of the first aspect to be executed, or causes the method in the second aspect or any possible implementation of the second aspect to be executed.

[0058] It should be understood that the technical effects that can be achieved in the eighth aspect can be specifically described by referring to the technical effects brought about by the first aspect or any possible implementation of the first aspect or the second aspect or any possible implementation of the second aspect, and will not be repeated here.

[0059] In the ninth aspect, an embodiment of the present application provides a computer program product, including a program or instructions, which, when executed on a computer, enables the method in the first aspect or any possible implementation of the first aspect to be executed, or enables the method in the second aspect or any possible implementation of the second aspect to be executed.

[0060] It should be understood that the technical effects that can be achieved in the eighth aspect can be specifically described by referring to the technical effects brought about by the first aspect or any possible implementation of the first aspect or the second aspect or any possible implementation of the second aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 Schematic diagram of two file transmission systems used in embodiments of the present application;

[0062] Figure 2A schematic diagram of another file transmission system used in an embodiment of the present application;

[0063] FIG3( a) is a schematic diagram of a terminal storage system used in an embodiment of the present application;

[0064] FIG3( b ) is a schematic diagram of a cloud storage system used in an embodiment of the present application;

[0065] Figure 4 A schematic diagram of the hardware architecture of an electronic device used in an embodiment of the present application;

[0066] Figure 5 A schematic diagram of the software architecture of an electronic device used in an embodiment of the present application;

[0067] Figure 6 A flowchart of a data compression method provided in an embodiment of the present application;

[0068] Figure 7 A schematic diagram of a first file composition provided in an embodiment of the present application;

[0069] Figure 8 A schematic diagram of a second file composition provided in an embodiment of the present application;

[0070] Fig. 9 A flowchart of a data compression method provided in an embodiment of the present application;

[0071] Fig.10 A flowchart of a data compression method provided in an embodiment of the present application;

[0072] Fig.11 A flowchart of a data compression method provided in an embodiment of the present application;

[0073] Fig.12 A flowchart of a data decompression method provided in an embodiment of the present application;

[0074] Fig.13 A flowchart of a data decompression method provided in an embodiment of the present application;

[0075] Fig.14 A block diagram of a data compression device provided in an embodiment of the present application;

[0076] Fig.15 A block diagram of a data decompression device provided in an embodiment of the present application;

[0077] Fig.16 A block diagram of an electronic device used in an embodiment of the present application; DETAILED DESCRIPTION

[0078] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0079] The terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations on the present application. As used in the specification and the appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" refer to one, two or more. The term "and / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist; for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0080] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear at different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0081] To facilitate understanding, the technical terms involved in the embodiments of the present application are first explained accordingly.

[0082] Sparse File: A sparse file is an abstract file provided by the underlying file system. Sparse files provide a larger virtual address space, free space management, non-contiguous use of address space, and maintenance of metadata, with reliable performance and scalability. Sparse files only utilize the allocated / initialized space within the file, not the entire address space of the file. For example, when a sparse file is created, it can include 1 terabyte (TB) of address space, but only 44 kilobytes (KB) of allocated / initialized data starting at address 0, and another 100KB of data starting at address 0xffff (or 64KB). Then, out of the entire 1TB space, a sparse file can only utilize 144KB, plus a small number of additional bytes for file metadata.

[0083] Typically, files provide a single contiguous address space. In file systems that can support files larger than 4TB, to represent objects that may extend to 1 gigabyte (GB) in size, the 1GB object is spaced out in the file, e.g., 10GB is pre-allocated into 10 segments. This approach wastes a lot of disk space. For objects that never reach 1GB in size, this allocation is also wasteful. Sparse files provide a single contiguous address space that initially includes regions of unallocated / uninitialized space. Modern file systems that support sparse files (e.g., Ext4, XFS, Btrfs, NTFS) provide a system interface that pre-allocates regions directly in the file without initializing the space (for actual data use). Such systems can also deallocate unused regions of files that have been previously written. These file systems provide multiple file states: unallocated, allocated but uninitialized, and allocated and initialized. In addition, some file systems provide "thin" provisioned sparse files. This means that these systems do not allocate disk space for the file until data is written. Any of the above systems can be used to provide sparse files.

[0084] With modern file systems such as Ext4, each object can be placed at a fixed logical address interval, and the unused portion between the objects is uninitialized. In this way, the continuous address space of each object can be extended in the logical address space without being blocked by other objects in the file, so that disk space is not wasted. The underlying file system transparently manages the available space in the disk, providing disk areas to back up objects when they are written. When the data in the object is no longer needed, the disk space can be returned to the file system available space through a system call, and the file system allocator can reuse the unneeded disk space to extend other objects.

[0085] Application scenarios of sparse files include but are not limited to the following scenarios: Block device file snapshots: The content stored in block devices often has a lot of blanks, so it is suitable to be stored in sparse files. Virtual machine disk files: Virtual machine disk files are usually very large files, but most of the content is empty, so sparse files can be used to save disk space Database files. Database files: They are usually very large files, but most of the content is empty, so sparse files can be used to save disk space. Log files: Log files are usually very large files, but most of the content is empty, so sparse files can be used to save disk space. Backup files: Backup files are usually very large files, but most of the content is empty, so sparse files can be used to save disk space. Pre-download files: Pre-allocated files for downloading a large file, which are basically empty at the beginning and gradually filled with content as the download progresses. For example, some of the videos downloaded in WeChat are sparse files.

[0086] File system, the computer file system is a method of storing and organizing computer data, which makes it easy to access and find it. The file system uses the abstract logical concepts of files and tree directories instead of the concept of data blocks used by physical devices such as hard disks and optical disks. Strictly speaking, the file system is a set of abstract data types that implement operations such as data storage, hierarchical organization, access, and retrieval. Most modern file systems support the "sparse file" format, such as the Apple file system (APFS) and the fourth extended file system (Ext4). On file systems that support the "sparse file" format, applications can choose to use the "sparse file" format for storage, storing only the content of the file that contains real data; on file systems that do not support the "sparse file" format, applications will store the entire content of the file.

[0087] Hole: File hole. In UNIX file operations, the file displacement can be greater than the current length of the file. In this case, the next write operation to the file will extend the file and form a hole in the file. Holes do not occupy storage space in file systems that support coefficient files. Sparse files contain hole information, such as the offset and size of the hole marked in metadata. Or holes can be regarded as the part of the sparse file that is not written but is set to 0 or occupied by the ASCII code NULL character.

[0088] Data compression: also known as source coding, is the process of representing information using fewer data bits (or other information-related units) than the uncoded data according to a specific coding mechanism. The compression ratio is an indicator used to evaluate the results of data compression, which refers to the ratio of the amount of data after compression to the amount of data before compression.

[0089] Sparseness ratio: refers to the ratio of the actual space occupied by a file to its logical size. The smaller the sparseness ratio, the higher the proportion of empty data blocks in the file, and the higher the efficiency of the sparse file.

[0090] With the development of information technology, the amount of data transmitted between devices is increasing. Therefore, data transmission requires a large bandwidth and takes up a lot of space when stored. In file operations of some file systems, the file displacement can be greater than the current length of the file. If the displacement is greater than the current length of the file, the next write operation will "extend" the file and create a "hole" in the file. Such files with holes are called sparse files. Sparse files are basically the same as other ordinary files. The difference is that part of the content in the file is a hole, and the hole does not occupy disk space in a system that supports sparse files. Most modern file systems support the "sparse file" format, such as the Apple file system (APFS) and the fourth extended file system (ext4). When a sparse file is copied in a file system, it can be specified whether it is still stored as a sparse file. However, when it is copied across file systems or transmitted on the network, its existing sparse file information will be lost, so that the hole actually occupies storage space. This situation not only leads to a large amount of storage space being wasted, but also consumes a lot of bandwidth during network transmission.

[0091] In view of the above problems, the present application provides a data compression method, which compresses holes or multiple consecutive zeros in a file, thereby reducing storage space and bandwidth consumption during data transmission.

[0092] The present application can be used in but not limited to the following file transfer systems: In some scenarios, users can save files on cloud-side devices, i.e., end-to-end cloud backup. In some scenarios, users can also download files from cloud-side devices locally and save them on local electronic devices, i.e., local backup. In some scenarios, users can transfer backup files from electronic device A to electronic device B, and then restore them on electronic device B. For example, data synchronization between new and old electronic devices is also called end-to-end cloning. In some scenarios, users use multiple terminal devices to transfer files to each other, so that file synchronization appears in multiple terminal devices, i.e., multi-device distributed collaborative file transmission scenarios. The above scenarios all involve data compression and / or decompression. Due to the large number of types and quantities of data to be transmitted, the process of data compression and / or decompression often takes a long time and consumes more storage resources and transmission bandwidth.

[0093] See also Figure 1 As shown, the present application provides a schematic diagram of a terminal side and cloud side collaboration scenario in which the terminal side performs compression and the other side performs decompression. Figure 1As shown in (a), the user can download the file from the cloud side device to the local and save it on the local electronic device, that is, local backup. Parameter settings are performed in advance, such as setting parameters related to the compression algorithm, etc., which are not particularly limited here. When the terminal side needs to download files from the cloud side, the terminal side can manually trigger the compression condition, that is, after selecting the compression option, select the file to be downloaded, then the terminal side can send a compression instruction to the cloud side, and the cloud side responds to the compression instruction to compress the file that the user needs to download, thereby saving running memory and storage space, while improving the efficiency of cloud side compression, and at the same time, it can also improve the image transmission rate of file transmission to the terminal side. When the terminal side needs to view the downloaded file, the compressed file can be decompressed on the terminal side, thereby saving running memory and improving the efficiency of decompression on the terminal side.

[0094] like Figure 1 As shown in (b), users can save files on cloud-side devices, i.e., end-to-end cloud backup. Parameter settings are performed in advance, such as setting decompression-related parameter settings, etc., which are not particularly limited here. When files on the terminal side need to be uploaded to the cloud side, the terminal side can manually trigger the preset compression conditions, that is, after the compression option is selected, the files to be uploaded are selected, then the terminal side can compress the files to be uploaded, thereby saving the terminal side's running memory and storage space, while improving the efficiency of terminal side compression, and also improving the image transmission rate of the files transmitted to the cloud side. When the cloud side needs to view the compressed file, the terminal side can send a decompression command to the cloud side, and the cloud side responds to the decompression command to decompress the compressed file, thereby saving the running memory and improving the efficiency of decompression on the cloud side. When the cloud side does not need to view the compressed file, the compressed file can be saved on the cloud side to save the storage resources of the cloud side device.

[0095] See also Figure 2 , is a schematic diagram of a terminal side and a terminal side cooperative scenario provided by the present application, in which one side performs compression and the other side performs decompression. Figure 2As shown, parameters are set in advance, such as setting compression algorithm related settings, etc., which are not particularly limited here. When the first terminal side needs to send a file to the second terminal side, for example, the first terminal side and the second terminal side correspond to different users. For example, user A sends the file required by user B to the second terminal side corresponding to user B through the first terminal side, or the first terminal side and the second terminal side correspond to the same user, and the same user wants to clone the file in the first terminal side to the second terminal side, and can manually trigger the compression function, that is, switch the "compression option" from the closed state to the open state, and then select the file to be sent, so that the first terminal side can re-compress the selected file, thereby saving the running memory and storage space of the first terminal side, while improving the compression efficiency of the first terminal side, and at the same time, it can also improve the image transmission rate of the file transferred to the second terminal side. When the second terminal side needs to view the compressed file, the compressed file can be decompressed on the second terminal side, thereby saving the running memory and improving the decompression efficiency of the second terminal side. It should be understood that the two terminal devices here are an example, and the number of terminal devices can also be more than two. When multiple terminal devices perform file synchronous transmission, that is, in a multi-device distributed collaborative file transmission scenario, the transmission between any two devices can refer to the transmission between the first terminal side and the second terminal side.

[0096] The present application can be applied to but not limited to the following storage on the terminal side or cloud side: Please refer to Figure 3(a), which is a schematic diagram of compression processing inside the terminal side provided by the present application. As shown in Figure 3(a), the file-related system application can be a gallery, and the image-related third-party application can be WeChat, etc., without special restrictions here. When the user has not viewed the file and the compression conditions are met, the operating system reads the file and compresses it, and then saves the compressed file to the memory. It should be understood that the compressed file can overwrite the original file. When the user needs to view the file, the operating system reads the file and decompresses it for display.

[0097] Please refer to Figure 3(b), which is a schematic diagram of compression and decompression on the cloud side in an embodiment of the present application. As shown in Figure 3(b), before compression / decompression, parameter settings are performed, such as setting compression algorithm related settings, etc., which are not particularly limited here. After the settings are completed, when the terminal side determines that the compression conditions are met, it can send an instruction to compress the file to the cloud side, and then the cloud side can implement the compression method, thereby saving running memory while improving the compression efficiency of the cloud side. Of course, it can also reduce the file's occupancy of the cloud side storage space, so that the cloud side can accommodate more storage resources. When the user needs to view the file on the cloud side, he can send a decompression command to the cloud side to control the cloud side to decompress the file, thereby saving running memory while improving the efficiency of decompression on the cloud side.

[0098] The method provided in the embodiments of the present application can be applied to electronic devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc. The above-mentioned terminal devices can be the electronic devices, and the embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

[0099] For example, Figure 4 The structure diagram of the electronic device 100 is shown. The terminal device may be the electronic device 100, which may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0100] It is to be understood that the structure illustrated in the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0101] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0102] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0103] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0104] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0105] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0106] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0107] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0108] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0109] The mobile communication module 150 may provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc.

[0110] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0111] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for image rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0112] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0113] The electronic device 100 can realize the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194 and the application processor. The ISP is used to process the data fed back by the camera 193. In some embodiments, the ISP can be set in the camera 193.

[0114] The camera 193 is used to capture still images or videos. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0115] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0116] NPU is a neural-network (NN) computing processor. It can quickly process input information by drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, and can also continuously self-learn.

[0117] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0118] The internal memory 121 may be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area.

[0119] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0120] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0121] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0122] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.

[0123] Figure 5 It is a software structure diagram of the electronic device 100 used in this application. The layered architecture divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, respectively, the application layer, the application framework layer, the Android runtime (Android runtime) and the system library, and the kernel layer. The application layer can include a series of application packages.

[0124] like Figure 5 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0125] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0126] like Figure 5 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0127] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0128] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0129] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0130] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.).

[0131] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0132] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0133] Android runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0134] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0135] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0136] The system library may include multiple functional modules, such as surface manager, media libraries, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0137] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0138] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0139] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0140] A 2D graphics engine is a drawing engine for 2D drawings.

[0141] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0142] It should be understood that the technical solutions in the embodiments of the present application can be used in Android, IOS, Hongmeng and other systems.

[0143] The following is an introduction to the data compression algorithm provided by this application in conjunction with the accompanying drawings. Figure 6 , which is a flow chart of a data compression method provided in an embodiment of the present application.

[0144] Step 201, obtaining a first file.

[0145] In a possible embodiment, the first file includes at least one first segment and at least one second segment, wherein the first segment is a segment in the first file containing a hole, and the second segment is a segment in the first file storing a data block; a hole refers to a portion of the first file where disk space has been allocated but not filled with data, and a data block refers to a portion of the first file filled with data.

[0146] In the embodiment of the present application, the first segment in the first file is compressed to generate a second file, which saves the storage space occupied by the holes in the file without valid information, reduces the resource occupancy and power consumption on the terminal side, reduces the bandwidth required for data transmission, and improves transmission efficiency.

[0147] In a possible embodiment, the first file may be a file containing a hole or a file not containing a hole. Optionally, the first file may only include data blocks.

[0148] Step 202: Determine whether the first file has a first section (hole).

[0149] In a possible embodiment, the first file may be a file known to have holes. For such a first file, the electronic device may directly compress the holes contained in the first file.

[0150] In a possible embodiment, the electronic device needs to determine whether the first file contains a hole.

[0151] In a possible embodiment, the electronic device may determine whether the file system of the electronic device supports a sparse file feature, and determine whether the first file is a sparse file.

[0152] The file system of the electronic device supports the sparse file feature, which can also be understood as the file system of the electronic device supports sparse files or supports sparse file formats. It can mean that the file system of the electronic device has all or part of the functions that rely on sparse files, including but not limited to: reading sparse files, creating sparse files, writing new data in sparse files, etc.

[0153] In a possible embodiment, the first application may determine whether the file system of the electronic device supports the sparse file feature by calling a system API.

[0154] Taking the Linux system as an example, the first application can use the fallocate function to allocate storage space without actually writing data. If the return value of the fallocate function is 0, it is determined that the file system of the electronic device supports the "sparse file" feature. If the return value of the fallocate function is non-zero, it is determined that the file system of the electronic device does not support the "sparse file" feature.

[0155] Optionally, when the file system of the electronic device supports a “sparse file” feature, it is determined whether the original file to be backed up is a sparse file.

[0156] In a possible embodiment, on a Linux system, the first application may obtain file information through an lseek function, determine whether the original file is a sparse file according to the file information, and find holes in the sparse file.

[0157] In a possible embodiment, on a WINDOWS operating system, a SetFilePointer system call may be used to detect and form holes, and CreateFile() and CloseFile() may be used to open and close files, thereby detecting and generating sparse files.

[0158] Exemplarily, the file information may include: the block size of the file (st_blksize), the number of disk blocks occupied by the file (st_blocks), and the actual size of the file (st_size). If the product of the block size of the file and the number of disk blocks occupied by the file is less than the actual size of the file, it means that the original file is a sparse file, otherwise, the original file is not a sparse file.

[0159] A sparse file can be considered as a file containing holes. Taking the Linux system as an example, the file header of a sparse file is defined by a file system such as ext4 or XFS. In a possible embodiment, the electronic device can determine whether the first file is a sparse file by reading information in the file header of the sparse file.

[0160] Optionally, during a subsequent compression process, the electronic device may compress the sparse file.

[0161] If the first file is not a sparse file, or the file system of the electronic device does not support the sparse file feature, the electronic device may not compress it and directly perform corresponding processing operations based on the original file. The electronic device may also determine whether there is a file hole in the first file by searching whether there is space in the first file that has been allocated disk space but not filled with actual data.

[0162] Optionally, in a subsequent compression process, the electronic device may compress the holes in the retrieved sparse file.

[0163] In the case where there is no hole in the first file, or in other words, the electronic device does not retrieve the hole in the sparse file, the electronic device can directly perform corresponding processing operations based on the original file.

[0164] In a possible embodiment, a hole may be regarded as all-zero data of a certain length. Generally, whether a file contains a file hole can be determined by searching whether there is all-zero data in the file.

[0165] Step 203: compress the first section (hole) in the first file to generate a second file.

[0166] In a possible embodiment, when there are holes in the first file, the electronic device may compress the holes in the first file.

[0167] In a possible embodiment, at least one first section (hole) in the first file is compressed to generate a second file. It should be understood that compressing the first section is compressing the hole in the first section. The second file includes at least one third section and at least one fourth section, the third section includes information for indicating that the at least one first section is compressed, and the fourth section includes a data block.

[0168] Taking the Linux system as an example, the file header of a sparse file is defined by a file system such as ext4 or XFS. In the file system, each file has an inode structure, which contains the file's metadata information, including the file's size, permissions, and creation time. For sparse files, the st_blocks field in the inode structure indicates the number of disk blocks occupied by the file, and the st_size field indicates the actual size of the file. When reading the data blocks and holes of a sparse file (i.e., the first file), the system call lseek() function can be used to obtain the location of the hole in the sparse file. All data blocks are written to the second file, the file header marks the location and size information of the hole, and the data blocks are written to the data area of ​​the file body. After the loop completes the writing of all valid data nodes, the file compression is completed.

[0169] Specifically, the structure of the original file can be as follows Figure 7 As shown, the structure of the compressed file can be as follows Figure 8 shown. Figure 7 A schematic diagram of the composition of a first file 300 is provided by way of example. The first file 300 may also be referred to as an original file 300 or a file 300 before compression. The shaded portion in the figure represents a data block 310 (or valid data 310) in the first file 300. The valid data 310 or data block 310 refers to meaningful data or real data in the first file 300. The blank portion represents a hole 320 in the first file 300.

[0170] The hole 320 refers to a portion of the first file 300 where disk space has been allocated but not filled with data, and the data block 310 refers to a portion of the first file 300 that is filled with data.

[0171] In some examples, the hole 320 may be generated when the electronic device downloads files in multiple threads at the same time. For example, the hole 320 may be generated when a chat software (such as WeChat, QQ, etc.) on the electronic device shares a large file (such as a video file).

[0172] In some examples, the first file 300 may be a sparse file, and thus the file block header of the first file 300 may be defined in the same manner as the file block header of a sparse file.

[0173] Specifically, on a Linux system, the file block header of the first file 300 may be defined by a file system such as ext4 or an extended file system (XFS), and the file block header of the first file 300 may include the location and size information of valid data nodes of the file block data.

[0174] Figure 8An exemplary schematic diagram of the composition of a second file 400 is provided. The second file 400 may be obtained by compressing the first file 300 using the data compression method provided in the present application.

[0175] The second file 400 may also be referred to as a target file 400 or a compressed file 400. The second file 400 may include a file header 410 and a file body 420, wherein the file header 410 may be used to indicate or describe one or more information of information related to the compression algorithm of the first file 300, the size of the first file 300, and the size of the real data in the first file 300. The file body 420 corresponds to the valid data in the first file 300, or in other words, the content of the file body data 420 is the same as the content of the valid data in the first file 300.

[0176] In a possible embodiment, the hole in the first file is compressed to generate the second file. Fig. 9 :

[0177] Optionally, a hole node data structure may be predefined including an offset value of the hole in the file (eg, offset) and a hole length (eg, len), and a hole linked list data structure may be predefined including a pointer to a hole node (eg, seg) and / or a link pointer to a hole linked list (eg, next).

[0178] Step 2031, open the first file.

[0179] In a possible embodiment, the first file may be opened by a system call open (applicable to LINUX operating system). Optionally, the inode structure information of the file may be obtained, including the file size and the number of disk blocks occupied.

[0180] Step 2032: Obtain at least one first position of at least one first segment in the first file.

[0181] In a possible embodiment, at least one first position corresponds one-to-one to the position of at least one first section in the first file, that is, corresponds one-to-one to the section where the hole is located.

[0182] In a possible embodiment, the first position includes a first offset and a first length. The obtaining of at least one first position of at least one first segment in the file to be sent includes: opening the first file; obtaining at least one first offset and a first length of at least one first segment in the first file, wherein the at least one first offset and the first length correspond to the at least one first segment one by one.

[0183] In this embodiment, the offset and size of the hole in the first file before compression are saved in the second file after compression by using a linked list. The offset and size as the location information of the hole can be obtained by calling a function or directly obtained by using a mark in the file, which is convenient for acquisition and reduces the computational complexity. And according to the offset and size of each hole, the second file after compression can be restored to the first file before compression according to the linked list information, while the storage overhead occupied by the linked list is reduced, storage resources are saved, and the bandwidth required for data transmission is reduced.

[0184] Exemplarily, the offset and length of the hole in the first file are obtained by calling the lseek() function (applicable to the LINUX operating system).

[0185] Step 2033, write at least one first position into the linked list.

[0186] In a possible embodiment, the linked list is used to indicate the position of the at least one first segment in the first file.

[0187] In this embodiment, the positions of the holes in the first file before compression are saved in the second file after compression using a linked list. The storage overhead occupied by the linked list is small, which reduces the resource occupation and power consumption on the terminal side and reduces the compression rate. At the same time, the bandwidth required for data transmission is reduced, and the transmission efficiency is improved. And the centralized storage in the linked list facilitates the second file after compression to restore the first file before compression according to the linked list information, which facilitates the management and reading of the position information of the holes and reduces the resource overhead of repeated reading and writing.

[0188] In a possible embodiment, the i-th node of a linked list is generated according to the i-th first position (i-th first offset and i-th first length); the i-th node is written into the linked list; the first pointer of the linked list points to the i-th node; the i+1-th node of the linked list is generated according to the i+1-th first offset and the i+1-th first length; the i+1-th node is written into the linked list; the i-th node and the i+1-th node are linked according to the second pointer, where i is a positive integer.

[0189] Exemplarily, a node (hole_seg) is generated according to a first position and then added to the record of the linked list (hole_seg_table_item); the seg pointer points to the node, and then lseek is called again to search for subsequent holes, and the subsequent hole nodes are linked together through the next pointer in the hole linked list, so that these holes are added to the hole linked list.

[0190] Step 2033 is looped, that is, data starting from the position where the file pointer is located is read until the number of bytes read reaches the file size, and a hole segment in the first file, that is, a linked list corresponding to the first segment, is generated.

[0191] In this embodiment, the offset and size of each hole are written into each node of the linked list in turn and linked with a second pointer, thereby reducing computational complexity, improving processing efficiency, and facilitating the restoration of the compressed second file to the first file before compression according to the information of each node in the linked list.

[0192] Step 2034, write the linked list into the second file.

[0193] In a possible embodiment, the linked list may be written into the third section of the second file, where the third section includes the file header of the second file. The file header definition is shown in the table, where the number of bytes of field #6 is variable:

[0194] Table 1 File header format

[0195] Fields Number of bytes definition describe Field #1 4 magic Compressed file header characteristic value Field #2 4 version Version Number Field #3 8 file_size Original file length Field #4 8 hole_seg_num Number of holes Field #5 4 hole_seg_nums[m] Compressed size Field #6 expasion_area_length m hole_seg_num length

[0196] Step 2035: Write the data blocks in at least one second section into the second file.

[0197] In a possible embodiment, a data block in at least one second section may be written into a fourth section of the second file, where the fourth section includes a file body of the sent file. Optionally, the writing order may be sequentially written according to the order of the first file.

[0198] Exemplarily, the read() function or pread() function can be used to read data blocks of the original file, that is, valid data, and the data blocks are sequentially written into the file body of the compressed file. Optionally, the data blocks can be written into the buffer once and then into the compressed file.

[0199] In this embodiment, a linked list storing the offset and size of the holes in the file before compression is written into the file header of the compressed file, and the data is written into the file body of the compressed file. This reduces the storage overhead and reduces the bandwidth required for data transmission. It also facilitates file management and reading, reduces computational complexity, and facilitates recovery of the first file before compression.

[0200] S204, processing the compressed first file, that is, the second file.

[0201] Taking the Linux system as an example, the file header of a sparse file is defined by a file system such as ext4 or XFS. In the file system, each file has an inode structure, which contains the file's metadata information, including the file's size, permissions, and creation time. For sparse files, the st_blocks field in the inode structure indicates the number of disk blocks occupied by the file, and the st_size field indicates the actual size of the file.

[0202] In a possible embodiment, after executing the data compression operation in step S203, the electronic device may perform a series of processing operations on the second file.

[0203] For example, in the end-to-end cloning scenario, the electronic device can send the second file to another electronic device to synchronize the data of the first file. Alternatively, the electronic device can also save the compressed file directly on the local storage medium of the electronic device. Since the number of bytes of the second file is reduced relative to the number of bytes of the first file after the hole is compressed, it takes less time to send the second file, and the storage resources occupied by storing the second file are less, the storage resources are more efficiently utilized, and the bandwidth occupied during transmission is reduced.

[0204] In a possible embodiment, the second file may be recompressed, and optionally, the data blocks in the second file may be compressed, so as to further reduce the storage space occupied by the file, reduce the bandwidth occupied by the transmission, and improve the transmission rate.

[0205] In a possible embodiment, the second file may be sent to the receiving device, or the encrypted second file may be sent to the receiving device. Optionally, an encryption algorithm such as AES may be used to protect user privacy and improve data security.

[0206] In a possible embodiment, the size of the hole and the size of the file can be recorded to obtain compression rate information and compression benefit. The compression rate or sparsity rate can be used to represent the proportion of real data in the file, and can be calculated by the ratio of the space occupied by the real data to the space occupied by the file.

[0207] S205: directly process the first file without compressing the first file.

[0208] When the first file does not include a hole, or when the first file does not include a hole that meets a preset condition, the electronic device may not compress the first file, but directly perform corresponding processing operations on the first file.

[0209] For example, in the end-to-end cloning scenario, the electronic device may directly send the first file to another electronic device and / or other storage device. Alternatively, the electronic device may also directly store the first file on a local storage medium of the electronic device.

[0210] In a possible embodiment, when the file system of the electronic device does not support the "sparse file" feature, or when the file system of the electronic device supports the "sparse file" feature but the original file to be backed up is not a sparse file, the original file is uploaded to the cloud side device.

[0211] In a possible embodiment, when the file system of the electronic device does not support the "sparse file" feature, the first application reads the original file and uploads it to the cloud-side device. In this case, the compression type of the original file can be marked as "no compression" in the first database, or the compression type of the original file can be marked as "0" to indicate that the metadata stores the original file.

[0212] In some examples, when the file system of the electronic device supports the "sparse file" feature, but the original file to be backed up is not a sparse file, the first application reads the original file and uploads it to the cloud-side device. In this case, the compression type of the original file can be marked as "no compression" in the first database, or the compression type of the original file can be marked as "0" to indicate that the original file is stored in the metadata.

[0213] refer to Fig.10 The present application also proposes a data compression method, when the first file is a non-sparse file including a segment of multiple consecutive zeros, compressing the segment of multiple consecutive zeros. Fig.10 A flowchart of another data compression method provided in an embodiment of the present application.

[0214] Step S501, obtaining a first file.

[0215] In a possible embodiment, the first file may be a file containing a segment of multiple consecutive zeros (hereinafter also referred to as a first segment), or may be a file not containing a segment of multiple consecutive zeros, wherein the segment of multiple consecutive zeros in the first file may be one or more. Optionally, the first file may only include valid data, or in other words, the first file may not include a segment of multiple consecutive zeros.

[0216] In a possible embodiment, the first file includes at least one first section and at least one second section, wherein the first section is a section of the first file including a plurality of consecutive zeros, and the second section is a section of the first file storing data blocks;

[0217] In an embodiment of the present application, the first segment in the first file is compressed to generate a second file, thereby saving the storage space occupied by multiple consecutive zeros without valid information in the file, reducing the resource occupancy and power consumption on the terminal side, reducing the bandwidth required for data transmission, and improving transmission efficiency.

[0218] Step S502, determining whether there is a segment (first segment) of multiple consecutive zeros in the first file.

[0219] In a possible embodiment, the first file may be a file that is known to have a segment of multiple consecutive zeros. For such a first file, the electronic device may compress the segment of multiple consecutive zeros contained therein.

[0220] In a possible embodiment, the electronic device needs to determine whether the first file contains a segment of multiple consecutive zeros. Optionally, in a subsequent compression process, the electronic device may compress the segment of multiple consecutive zeros in the first file.

[0221] In the case where the first file does not contain a segment of multiple consecutive zeros, the electronic device may not compress it and directly perform corresponding processing operations based on the original file.

[0222] Optionally, the electronic device may also determine whether there is a file hole in the first file by searching whether there is a space in the first file that has been allocated disk space but is not filled with actual data. Optionally, in the subsequent compression process, the electronic device may compress the hole in the retrieved first file. For the method of compressing the hole, see Figure 6 For the sake of brevity, the description of the related steps is not repeated here.

[0223] Step 503: compress the multiple consecutive zeros (first section) in the first file to generate a second file.

[0224] It should be understood that compressing the first section is to compress multiple consecutive zeros in the first section. In a possible embodiment, the second file includes at least one third section and at least one fourth section, the third section contains information indicating that the at least one first section is compressed, and the fourth section contains the data block.

[0225] In a possible embodiment, when the first file has multiple consecutive zero segments, the electronic device may compress the multiple consecutive zero segments in the first file.

[0226] In this embodiment, the positions of the continuous multiple zeros in the first file before compression are saved in the compressed second file using a linked list. The storage overhead occupied by the linked list is small, which reduces the resource occupation and power consumption of the terminal side and reduces the compression rate. At the same time, the bandwidth required for data transmission is reduced, and the transmission efficiency is improved. And the concentrated storage in the linked list facilitates the second file after compression to restore the first file before compression according to the linked list information, which is convenient for managing and reading the position information of the continuous multiple zeros, and reduces the resource overhead of repeated reading and writing.

[0227] A segment of multiple consecutive zeros is an all-zero data segment of a certain length. Generally, whether the file contains the segment can be determined by searching whether there is all-zero data in the file. In some examples, in order to improve detection efficiency, data with a length of consecutive "0" bits greater than or equal to a preset length is determined as the segment, and data with a length of consecutive "0" bits less than a preset length is determined not to be the segment of multiple consecutive zeros.

[0228] In a possible embodiment, the starting position and length of a segment of multiple consecutive zeros may be scanned to obtain nodes, and a linked list may be generated according to the nodes.

[0229] In a possible embodiment, multiple consecutive zeros in the first file are compressed to generate a second file, the steps are as follows: Fig.11 :

[0230] In a possible embodiment, a node data structure may be predefined to include an offset value of the first segment in the file (e.g., offset) and a length of the first segment (e.g., len). A hole linked list data structure may also be predefined to include a pointer to a hole node (e.g., seg) and a link pointer to the hole linked list (e.g., next).

[0231] Step 5031, open the first file.

[0232] In a possible embodiment, the first file may be opened by a system call open (applicable to LINUX operating system). Optionally, the inode structure information of the file may be obtained, including the file size and the number of disk blocks occupied.

[0233] Step 5032, obtaining at least one first position of at least one first segment in the first file.

[0234] In a possible embodiment, at least one first position corresponds to a position of at least one first section in the first file in a one-to-one manner, and the first section is a section in the first file that includes a plurality of consecutive zeros.

[0235] In a possible embodiment, the first position includes a first offset and a first length, and obtaining at least one first position of at least one first segment in the file to be sent includes: opening the first file; obtaining at least one first offset and first length of at least one first segment in the first file, and at least one first offset and first length correspond to at least one first segment one by one. Optionally, a sliding window can be used to obtain the first offset and first length of the first segment in the first file. Exemplarily, the first file content is read in a loop, and a full 0 value match is performed with 4096 bytes (the size of an ordinary file block) as a sliding window. If the match is successful, it is recorded as a segment of multiple consecutive zeros (also called the first segment), and the first offset of the first segment is the starting position of the sliding window.

[0236] If the sliding window +1 can still match all 0s successfully, the matching continues until it is not all 0s. At this time, the "end position of the first segment" is found, and the first length of the first segment is obtained by subtracting it. The loop is executed until the end of the first file to detect all the first segments.

[0237] In this embodiment, the offset and size of the continuous multiple zeros in the first file before compression are saved in the compressed second file by using a linked list. The offset and size as the position information of the continuous multiple zeros can be obtained by calling a function or directly obtained by using a mark in the file, which is convenient for acquisition and reduces the computational complexity. And according to the offset and size of each segment of the continuous multiple zeros, the compressed second file is restored to the first file before compression according to the linked list information, and at the same time, the storage overhead occupied by the linked list is reduced, storage resources are saved, and the bandwidth required for data transmission is reduced.

[0238] Step 5033, write at least one first position into the linked list.

[0239] In a possible embodiment, the linked list is used to indicate the position of at least one first segment in the first file.

[0240] In a possible embodiment, the i-th node of a linked list is generated according to the i-th first position (i-th first offset and i-th first length); the i-th node is written into the linked list; the first pointer of the linked list points to the i-th node; the i+1-th node of the linked list is generated according to the i+1-th first offset and the i+1-th first length; the i-th node is written into the linked list; the i-th node and the i+1-th node are linked according to the second pointer, where i is a positive integer.

[0241] Exemplarily, a node (hole_seg) is generated according to a first position, and then added to the record of the linked list (hole_seg_table_item); wherein the seg pointer points to the node, and then lseek is called again to search for subsequent holes, and the subsequent first segment nodes are linked together through the next pointer in the hole linked list, so that these first segment nodes are added to the linked list. Step 5033 is looped, that is, the data starting from the position where the file pointer is located is read until the number of bytes read reaches the file size, and a linked list corresponding to the first segment in the first file is generated.

[0242] In this embodiment, the offset and size of each segment of multiple consecutive zeros are written into each node of the linked list in turn and linked with a second pointer, thereby reducing computational complexity, improving processing efficiency, and facilitating the restoration of the compressed second file to the first file before compression according to the information of each node in the linked list.

[0243] Step 5034, write the linked list into the second file.

[0244] In a possible embodiment, the linked list is written into the third section of the second file, and the third section includes the file header of the second file. The contents related to the file header definition are similar to the related description in step 2034, and are not repeated here for brevity.

[0245] Step 5035: write the data blocks in at least one second segment into the second file.

[0246] In a possible embodiment, at least one data block in the second section is written into the fourth section of the second file, the fourth section includes the file body of the sent file, and optionally, the writing order may be written in sequence according to the order of the first file.

[0247] Exemplarily, the read() function or pread() function is used to read the data blocks of the original file, ie, valid data, and the data blocks are sequentially written into the file body of the compressed file. Optionally, the data blocks may be written into the buffer once and then into the compressed file.

[0248] In this embodiment, a linked list storing the offsets and sizes of multiple consecutive zeros in the file before compression is written into the file header of the compressed file, and the data is written into the file body of the compressed file. This reduces the storage overhead and reduces the bandwidth required for data transmission. It also facilitates file management and reading, reduces computational complexity, and facilitates recovery of the first file before compression.

[0249] S504, processing the compressed first file, that is, the second file.

[0250] In a possible embodiment, after executing the data compression operation in step S503, the electronic device may perform a series of processing operations on the second file.

[0251] For example, in the scenario of end-to-end cloning, the electronic device can send the second file to another electronic device to synchronize the data of the first file. Alternatively, the electronic device can also directly save the compressed file on the local storage medium of the electronic device. Since the number of bytes of the second file is reduced relative to the number of bytes of the first file after the segment with multiple consecutive zeros is compressed, the time taken to send the second file is shorter, the storage resources occupied by storing the second file are less, the utilization efficiency of the storage resources is higher, and the bandwidth occupied during transmission is reduced.

[0252] In a possible embodiment, the second file may be recompressed, and optionally, the data blocks in the second file may be compressed, so as to further reduce the storage space occupied by the file, reduce the bandwidth occupied by the transmission, and improve the transmission rate.

[0253] In a possible embodiment, the second file may be sent to the receiving device; or the encrypted second file may be sent to the receiving device. Optionally, an encryption algorithm such as AES may be used to protect user privacy and improve data security.

[0254] In a possible embodiment, the compression rate information can be obtained by recording the size of the segment with multiple consecutive zeros and the size of the file, and the compression benefit can be obtained. The compression rate or sparse rate can be used to represent the proportion of real data in the file, and can be calculated by the ratio of the space occupied by the real data to the space occupied by the file.

[0255] S505: directly process the first file without compressing the first file.

[0256] When the first file does not include a segment of multiple consecutive zeros, or when the first file does not include a segment of multiple consecutive zeros that meets a preset condition, the electronic device may not compress the first file and directly perform corresponding processing operations on the first file.

[0257] For example, in the end-to-end cloning scenario, the electronic device may directly send the first file to another electronic device and / or other storage device. Alternatively, the electronic device may also directly store the first file on a local storage medium of the electronic device.

[0258] In a possible embodiment, when the first file does not include a segment of multiple consecutive zeros, the first application reads the original file and uploads it to the cloud-side device. In this case, the compression type of the original file can be marked as "no compression", or the compression type of the original file can be marked as "0" to indicate that the original file is stored.

[0259] See also Fig.12 , which is a data decompression method provided by the present application. In this method, the electronic device can determine the compression method of the file data according to the file header of the compressed file, and then decompress the compressed data to achieve the recovery of the compressed data.

[0260] Step 601, obtaining a second file.

[0261] In a possible embodiment, the second file includes at least one third section and at least one fourth section, the third section includes information for indicating to compress at least one first section in the first file, and the fourth section includes a data block in the first file.

[0262] In a possible embodiment, the second file includes a linked list and a data block, wherein the linked list is used to indicate information for compressing holes or multiple consecutive zeros in the first file. The second file includes a file header and a file body, wherein the linked list is located in the file header and the data block is located in the file body. Figure 8 The relevant description is not repeated here.

[0263] For example, in some scenarios, when a user's electronic device is invaded by a virus, accidentally deleted, damaged, or has a hardware failure, or when the device is replaced, the backup data can be used to restore the user's personal data. The backup data here can be data stored in one or more of a cloud device, an electronic device, or other storage media (for example, other electronic devices), and the backup data can be decompressed and restored to the original file.

[0264] In a possible embodiment, the electronic device receives the second file from the sending device; receiving the compressed second file from the sending device saves the bandwidth required for data transmission. Or receiving the encrypted second file from the sending device, decrypting the encrypted second file, and obtaining the decrypted second file; improving the security of data transmission while protecting the privacy of the user. Or receiving the re-compressed second file from the sending device, decompressing the re-compressed second file, and obtaining the decompressed second file, re-compressing further saves storage space and improves transmission efficiency. Optionally, re-compression can be different from the above. Figure 6 and Fig. 9 The compression algorithms shown include, but are not limited to, compression algorithms for the entire contents of a file or compression algorithms for valid data.

[0265] In a possible embodiment, the second file may be uncompressed data (i.e., the original file), or may be data obtained by compressing file holes in the original file. The electronic device may determine the type of the second file by reading the file header. For example, the electronic device may read the data compression type information, and when the compression type is 0, determine that the current second file is an original file, or uncompressed data. When the compression type is not 0, determine that the current second file is a compressed file. When the current second file is an original file, the electronic device may directly write the second file to the corresponding file address.

[0266] In a possible embodiment, when the file system of the receiving device receiving the second file does not support sparse files, the receiving device stores the second file. In this embodiment, since when the file system of the receiving device receiving the second file does not support sparse files, the compressed second file is decompressed into the first file before compression, the empty part in the first file will occupy the actual storage space, so when the user has no need to view and use the second file, the second file can be stored in the receiving device without decompressing it, thereby saving storage space and improving storage utilization.

[0267] Step 602, decompress the second file to generate a first file.

[0268] In a possible embodiment, the first file includes at least one first segment and at least one second segment, wherein the first segment is a segment in the first file that includes a hole or multiple consecutive zeros, and the second segment is a segment in the first file that stores the data block.

[0269] In the case where the current second file is a compressed file, the electronic device can read the file header of the file data to determine the compression algorithm of the compressed file, thereby determining the decompression algorithm corresponding to the compression algorithm. Using the decompression algorithm, the electronic device can decompress the compressed file into the original file. After obtaining the original file, the electronic device writes the original file to the corresponding address.

[0270] In the embodiment of the present application, the compressed second file is restored to the first file before compression, so that the user can view and use it. The compressed second file saves the storage space occupied by holes or multiple consecutive zeros without valid information in the file, reduces the resource occupation and power consumption of the terminal side, reduces the bandwidth required for data transmission, and improves the transmission efficiency.

[0271] In a possible embodiment, the step 602 of decompressing the second file to generate the first file includes: Fig.13 The method shown:

[0272] Step 6021, obtain the linked list in the second file.

[0273] In a possible embodiment, the linked list is used to indicate the position of at least one first segment in the first file, and the first segment is a segment in the first file that contains a hole or a plurality of consecutive zeros;

[0274] In a possible embodiment, obtaining the linked list in the second file includes: obtaining the first linked list from the third section of the second file, the third section including the file header of the compressed file. Exemplarily, the file header information such as pdlz_head can be extracted from the file header of the compressed file.

[0275] In this embodiment, the positions of holes or consecutive zeros in the first file before compression are saved in the compressed second file using a linked list. The storage overhead occupied by the linked list is small, which reduces the resource occupation and power consumption on the terminal side and reduces the compression rate. The bandwidth required for data transmission is reduced, and the transmission efficiency is improved. In addition, the concentrated storage in the linked list facilitates the recovery of the compressed second file according to the linked list information to obtain the first file before compression, which facilitates the management and reading of the position information of holes or consecutive zeros, and reduces the resource overhead of repeated reading and writing.

[0276] Step 6022: Determine the position of the at least one first section in the first file according to the at least one first position in the linked list.

[0277] In a possible embodiment, at least one first position corresponds one-to-one to a position of at least one first section in the first file;

[0278] In a possible embodiment, the first position includes a first offset and a first length, and determining the position of at least one first segment in the first file according to at least one first position in the linked list includes: obtaining at least one first offset and first length in the first linked list, the at least one first offset and first length corresponding one-to-one to the at least one first segment; determining the position of the at least one first segment in the first file according to the at least one first offset and first length in the linked list.

[0279] In this embodiment, the offset and size of the hole or continuous multiple zeros in the first file before compression are saved in the second file after compression by a linked list. The offset and size as the position information of the hole or continuous multiple zeros can be obtained by calling a function or directly obtained through a mark in the file, which is convenient for acquisition and reduces the complexity of calculation. The storage overhead occupied by the linked list is small, storage resources are saved, and the bandwidth required for data transmission is reduced. And according to the offset and size of each hole or continuous multiple zeros, it is convenient to restore the compressed second file to the first file before compression according to the linked list information. At the same time, the storage overhead occupied by the linked list is small, storage resources are saved, and the bandwidth required for data transmission is reduced.

[0280] Step 6023, write a hole or multiple consecutive zeros into at least one first segment in the first file.

[0281] Step 6024, obtain at least one data block in the second file.

[0282] In a possible embodiment, obtaining at least one data block in the second file includes: obtaining a linked list from a fourth section of the second file, where the fourth section includes a file body of the second file.

[0283] In this embodiment, a linked list storing the offsets and sizes of holes or consecutive zeros in the file before compression is written in the file header of the compressed file, and the data blocks are written in the file body of the compressed file. This reduces the storage overhead and reduces the bandwidth required for data transmission. It is also convenient for file management and reading, and for restoring the first file before compression.

[0284] Step 6025: Write at least one data block into at least one second section in the first file to generate a first file.

[0285] In a possible embodiment, the second section is a section storing data blocks in the first file.

[0286] Based on similar principles, Fig.14 As shown, the embodiment of the present application also provides a data compression device 1400, including a method for executing Figure 6 or Fig.10 The data compression method shown or a module of a method in any possible embodiment of the data compression method.

[0287] Exemplarily, the device may include: an acquisition module 1401, configured to acquire a first file, the first file including at least one first segment and at least one second segment, wherein the first segment is a segment in the first file including a hole or multiple consecutive zeros, and the second segment is a segment in the first file storing a data block;

[0288] The processing module 1402 is used to compress at least one first segment in the first file to generate a second file, wherein the second file includes at least one third segment and at least one fourth segment, wherein the third segment contains information indicating that the at least one first segment is compressed, and the fourth segment contains a data block.

[0289] In a possible embodiment, compressing at least one first segment in a first file to generate a second file includes: obtaining at least one first position of at least one first segment in the first file, the at least one first position corresponding one-to-one to the position of at least one first segment in the first file; writing the at least one first position into a linked list, the linked list being used to indicate the position of at least one first segment in the first file; writing the linked list into the second file; and writing a data block in at least one second segment into the second file.

[0290] In a possible embodiment, the first position includes a first offset and a first length, and obtaining at least one first position of at least one first segment in the file to be sent includes: opening the first file; obtaining at least one first offset and first length of at least one first segment in the first file, and the at least one first offset and first length correspond one-to-one to the at least one first segment.

[0291] In a possible embodiment, writing at least one first position into a linked list includes: generating an i-th node of the linked list according to an i-th first offset and an i-th first length; writing the i-th node into the linked list; pointing a first pointer of the linked list to the i-th node; generating an i+1-th node of the linked list according to an i+1-th first offset and an i+1-th first length; writing the i-th node into the linked list; and linking the i-th node and the i+1-th node according to a second pointer, where i is a positive integer.

[0292] In a possible embodiment, it is characterized in that writing the linked list into the second file includes: writing the linked list into a third section of the second file, and the third section includes a file header of the second file.

[0293] In a possible embodiment, it is characterized in that writing data in at least one second segment into the second file includes: writing a data block in at least one second segment into a fourth segment of the second file, and the fourth segment includes a file body of the sent file.

[0294] In a possible embodiment, it is characterized in that the processing module is further used for: sending the second file to the receiving device; or sending the encrypted second file to the receiving device.

[0295] It should be understood that the data compression device and Figure 6 or Fig.10 The data compression method shown, or Figure 6 or Fig.10 The method in any possible embodiment of the data compression method shown corresponds to the method in the embodiment, so all relevant contents of each step involved in the above data compression method embodiment can be referred to the functional description of the corresponding functional module, which will not be repeated here.

[0296] Based on similar principles, Fig.15 As shown, the embodiment of the present application also provides a data decompression device 1500, which includes a method for executing Fig.12 The data decompression method shown or a module of a method in any possible embodiment of the data decompression method.

[0297] Exemplarily, the device may include: an acquisition module 1501, used to acquire a second file; the second file includes at least one third segment and at least one fourth segment, the third segment contains information for indicating compression of at least one first segment in the first file, and the fourth segment contains a data block.

[0298] Processing module 1502 is used to decompress the second file to generate a first file, wherein the first file includes at least one first segment and at least one second segment, wherein the first segment is a segment in the first file that includes a hole or multiple consecutive zeros, and the second segment is a segment in the first file that stores data blocks.

[0299] In a possible embodiment, decompressing the second file to generate the first file includes: obtaining a linked list in the second file, the linked list being used to indicate the position of at least one first segment in the first file, the first segment being a segment in the first file containing a hole or multiple consecutive zeros; determining the position of at least one first segment in the first file according to at least one first position in the linked list; at least one first position having a one-to-one correspondence with the position of at least one first segment in the first file; writing the hole or multiple consecutive zeros into at least one first segment in the first file; obtaining at least one data block in the second file; and writing at least one data block into at least one second segment in the first file to generate the second file, the second segment being a segment storing data blocks in the first file.

[0300] In a possible embodiment, the first position includes a first offset and a first length, and the position of at least one first segment in the first file is determined according to at least one first position in the linked list: at least one first offset and first length in the first linked list are obtained, and at least one first offset and first length correspond one-to-one to at least one first segment; according to at least one first offset and first length in the linked list, the position of at least one first segment in the first file is determined.

[0301] In a possible implementation, obtaining the linked list in the second file includes: obtaining the first linked list from the third section of the second file, where the third section includes a file header of the compressed file.

[0302] In a possible embodiment, obtaining at least one data block in the second file includes: obtaining a linked list from a fourth section of the second file, where the fourth section includes a file body of the second file.

[0303] In a possible embodiment, the processing module is further configured to: when a file system of a receiving device that receives the second file does not support sparse files, the receiving device stores the second file.

[0304] In a possible embodiment, the processing module is further used to: receive the second file from the sending device; or receive the encrypted second file from the sending device, decrypt the encrypted second file, and obtain the decrypted second file. Fig.12 The data decompression method shown, or Fig.12 The method in any possible embodiment of the data decompression method shown corresponds to the method in the embodiment, so all relevant contents of each step involved in the above data decompression method embodiment can be referred to the functional description of the corresponding functional module, which will not be repeated here.

[0305] The present application embodiment also provides an electronic device 1600, see Fig.16 , Fig.16 An example diagram of an electronic device provided in an embodiment of the present application is used to implement Figure 6 The method of data compression in the embodiment shown, or Fig.10 Another data compression method in the embodiment shown, or Fig.12 The method for data decompression in the illustrated embodiment.

[0306] like Fig.16 As shown, the electronic device 1600 may include a processor 1601 for executing a program or instruction stored in a memory 1602. When the program or instruction stored in the memory 1602 is executed, the processor 1601 is used to execute a program or instruction for implementing Figure 6 The method of data compression in the embodiment shown, or Fig.10 Another data compression method in the embodiment shown, or Fig.12 The method for data decompression in the illustrated embodiment.

[0307] Optionally, the electronic device 1600 may further include a communication interface 1603. Fig.16 The dashed lines indicate that the communication interface 1603 is optional for the electronic device 1600 .

[0308] Among them, the number of processors 1601, memories 1602 and communication interfaces 1603 does not constitute a limitation on the embodiments of the present application, and during specific implementation, they can be arbitrarily configured according to business requirements.

[0309] Optionally, the memory 1602 is located outside the electronic device 1600 .

[0310] Optionally, the electronic device 1600 includes the memory 1602, the memory 1602 is connected to the at least one processor 1601, and the memory 1602 stores instructions that can be executed by the at least one processor 1601. Fig.16 The dashed lines indicate that the memory 1602 is optional for the electronic device 1600 .

[0311] The processor 1601 and the memory 1602 may be coupled via an interface circuit or may be integrated together, which is not limited here.

[0312] The specific connection medium between the processor 1601, the memory 1602 and the communication interface 1603 is not limited in the embodiment of the present application. Fig.16 In the embodiment, the processor 1601, the memory 1602 and the communication interface 1603 are connected via a bus 1604. Fig.16 The connections between the other components are shown in bold lines, which are only for illustration and are not intended to be limiting. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.15 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0313] It should be understood that the processor mentioned in the embodiments of the present application can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented by software, the processor can be a general-purpose processor implemented by reading software code stored in a memory.

[0314] Exemplarily, the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0315] It should be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous link DRAM (SLDRAM) and direct memory bus random access memory (directrambus RAM, DR RAM).

[0316] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, the memory (storage module) can be integrated into the processor.

[0317] It should be noted that the memory described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0318] An embodiment of the present application also provides a computer-readable storage medium, including a program or an instruction. When the program or the instruction is run on a computer, the data compression method or the data decompression method in the embodiment of the present application is executed.

[0319] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0320] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0321] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0322] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0323] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for data compression, characterized in that: The method comprises: Acquire a first file, the first file comprising at least one first section and at least one second section, wherein the first section is a section in the first file containing a hole or a plurality of consecutive zeros, and the second section is a section in the first file storing a data block; The at least one first segment in the first file is compressed to generate a second file, wherein the second file includes at least one third segment and at least one fourth segment, the third segment contains information indicating that the at least one first segment is compressed, and the fourth segment contains the data block.

2. The method according to claim 1, characterized in that The compressing the at least one first section in the first file to generate the second file comprises: Obtain at least one first position of the at least one first section in the first file, wherein the at least one first position corresponds one-to-one to a position of the at least one first section in the first file; Writing the at least one first position into a linked list, wherein the linked list is used to indicate the position of the at least one first section in the first file; Writing the linked list into the second file; The data blocks in the at least one second section are written into the second file.

3. The method according to claim 2, characterized in that The first position includes a first offset and a first length, and obtaining at least one first position of the at least one first segment in the first file includes: Open the first file; At least one first offset and a first length of the at least one first segment in the first file are obtained, wherein the at least one first offset and the first length correspond one-to-one to the at least one first segment.

4. The method according to claim 2 or 3, characterized in that: Writing the at least one first position into the linked list comprises: Generate an i-th node of the linked list according to the i-th first offset and the i-th first length; Writing the i-th node into the linked list; Point the first pointer of the linked list to the i-th node; Generate the i+1th node of the linked list according to the i+1th first offset and the i+1th first length; Write the i+1th node into the linked list; The i-th node and the i+1-th node are linked according to the second pointer, where i is a positive integer.

5. The method according to any one of claims 2 to 4, characterized in that Writing the linked list into the second file comprises: The linked list is written into the third section of the second file, where the third section includes a file header of the second file.

6. The method according to any one of claims 2 to 5, characterized in that Writing the data in the at least one second section into the second file comprises: The data blocks in the at least one second section are written into the fourth section of the second file, wherein the fourth section includes a file body of the sending file.

7. The method according to any one of claims 2 to 6, characterized in that The method further comprises: sending the second file to a receiving device; Or sending the encrypted second file to the receiving device.

8. A method for decompressing data, characterized in that: The method comprises: Acquire a second file; the second file comprises at least one third section and at least one fourth section, the third section comprises information for indicating that at least one first section in the first file is compressed, and the fourth section comprises a data block in the first file; The second file is decompressed to generate a first file, wherein the first file includes the at least one first segment and the at least one second segment, wherein the first segment is a segment in the first file that includes a hole or multiple consecutive zeros, and the second segment is a segment in the first file that stores the data block.

9. The method according to claim 8, characterized in that The decompressing the second file to generate the first file comprises: Obtaining a linked list in the second file, the linked list being used to indicate a position of the at least one first segment in the first file; Determine the position of the at least one first section in the first file according to the at least one first position in the linked list; the at least one first position corresponds to the position of the at least one first section in the first file in a one-to-one manner; writing the hole or the plurality of consecutive zeros into the at least one first section in the first file; Obtain at least one data block in the second file; The at least one data block is written into at least one second section in the first file to generate the first file, where the second section is a section in the first file that stores data blocks.

10. The method according to claim 9, characterized in that The first position includes a first offset and a first length, and determining the position of the at least one first section in the first file according to the at least one first position in the linked list includes: Obtain at least one first offset and first length in the first linked list, wherein the at least one first offset and first length correspond one-to-one to the at least one first segment; The position of the at least one first section in the first file is determined according to the at least one first offset and the first length in the linked list.

11. The method according to any one of claims 8 to 10, characterized in that The obtaining of the linked list in the second file comprises: The first linked list is obtained from the third section of the second file, where the third section includes a file header of the compressed file.

12. The method according to any one of claims 8 to 11, characterized in that The obtaining of at least one data block in the second file comprises: The linked list is obtained from the fourth section of the second file, where the fourth section includes a file body of the second file.

13. The method according to any one of claims 8 to 12, characterized in that The method further comprises: When the file system of the receiving device that receives the second file does not support sparse files, the receiving device stores the second file.

14. The method according to any one of claims 8 to 13, characterized in that The method further comprises: receiving the second file from a sending device; Or receiving the encrypted second file from the sending device, decrypting the encrypted second file, and obtaining the decrypted second file.

15. A data compression device, characterized in that: The device comprises: An acquisition module, configured to acquire a first file, wherein the first file includes at least one first section and at least one second section, wherein the first section is a section in the first file that includes a hole or a plurality of consecutive zeros, and the second section is a section in the first file that stores a data block; A processing module is used to compress the at least one first segment in the first file to generate a second file, wherein the second file includes at least one third segment and at least one fourth segment, the third segment contains information indicating that the at least one first segment is to be compressed, and the fourth segment contains the data block.

16. A data decompression device, characterized in that: an acquisition module, configured to acquire a second file; the second file comprising at least one third section and at least one fourth section, the third section comprising information for indicating that at least one first section in the first file is compressed, and the fourth section comprising a data block; A processing module is used to decompress the second file to generate a first file, wherein the first file includes at least one first segment and at least one second segment, wherein the first segment is a segment in the first file that includes a hole or multiple consecutive zeros, and the second segment is a segment in the first file that stores the data block.

17. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the memory stores computer instructions; the processor is used to execute the computer instructions, so that the communication device executes the method according to any one of claims 1 to 7 or the method according to any one of claims 8 to 14.

18. A file transmission system, characterized in that: The system comprises one or more first ends and one or more second ends, wherein, The first end is used to perform the method according to any one of claims 1 to claim 7; The second end is used to execute the method according to any one of claims 8 to claim 14.

19. A computer readable medium, characterized in that The computer-readable medium comprises instructions which, when executed by a processor of a processing system, cause the processing system to perform a method according to any one of claims 1 to 7 or to perform a method according to any one of claims 8 to 14.

20. A computer program, characterized in that The computer program comprises instructions which, when executed by a processor of a processing system, cause the processing system to perform a method according to any one of claims 1 to 7 or to perform a method according to any one of claims 8 to 14.

Images