Data compression method and related equipment

Abstract

The embodiment of the invention discloses a data compression method and related equipment, and the method comprises the steps: obtaining an image set which comprises a plurality of to-be-compressed images; dividing each to-be-compressed image into a plurality of sub-image blocks; obtaining a semantic feature set, wherein the semantic feature set comprises semantic features of sub-image blocks of each to-be-compressed image in the image set; performing clustering operation on the semantic feature set to obtain a semantic base set; reconstruction information of each to-be-compressed image is acquired, the reconstruction information comprises identifiers of multiple semantic bases in a semantic base set, and the multiple semantic bases correspond to clustering results of the semantic features of each sub-image block of the to-be-compressed image; and sending compressed data of the image set to a receiving end device, wherein the compressed data comprises the semantic base set and the reconstruction information.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computers, and in particular to a data compression method and related equipment. Background Art

[0002] The demand for data transmission in wireless networks has experienced explosive growth in the past few years, and image and video transmission is one of the main applications that consume network communication resources. In future wireless networks, the demand for image and video transmission from AI-enabled tasks such as industrial IoT and smart driving will merge with the further growing demand for traditional image and video transmission, posing more stringent challenges to the deployment of network communication resources.

[0003] In recent years, semantic-based communication ideas have provided a new perspective for the optimization design of image and video transmission solutions. Semantic communication takes "expression" as its basic goal, focusing on the content of transmitted information rather than coding symbols. On the basis of achieving background knowledge synchronization between the sender and the receiver, it can fully meet communication needs while reducing the volume of information representation. In semantic communication, the information to be transmitted needs to construct and select matching semantic bases from the source / task semantic knowledge base based on its own characteristics and under the guidance of the communication task requirements, and then map it to the semantic space based on this, thereby improving the efficiency of information transmission.

[0004] However, current semantic communication does not provide a way to construct and apply semantic bases in image and video transmission scenarios. Summary of the invention

[0005] The present application provides a data compression method and related equipment for improving the compression efficiency of images or videos, achieving more ideal reconstruction quality at the same compression ratio, or achieving more ideal compression ratio at the same reconstruction quality.

[0006] The first aspect of the present application provides a data compression method, which is applied to a sending end device:

[0007] An image set is obtained, the image set including a plurality of images to be compressed; each image to be compressed is divided into a plurality of sub-image blocks; a semantic feature set is obtained, the semantic feature set including the semantic features of the sub-image blocks of each image to be compressed in the image set; a clustering operation is performed on the semantic feature set to obtain a semantic base set; reconstruction information of each image to be compressed is obtained, the reconstruction information including identifiers of a plurality of semantic bases in the semantic base set, the plurality of semantic bases respectively corresponding to the clustering results of the semantic features of each sub-image block of the image to be compressed; and compressed data of the image set, including the semantic base set and the reconstruction information, is sent to a receiving device.

[0008] In the present application, by clustering the semantic feature set, the redundant information within and between the images to be compressed can be fully mined to obtain the semantic base set containing the key information of the image set. By using the identifier of the semantic base corresponding to the semantic feature of each sub-image block of the image to be compressed as the reconstruction information of the image to be compressed, and adding the semantic base set to form the compressed data of the image set, data transmission based on semantic communication can improve the compression efficiency of images or videos, achieve a more ideal reconstruction quality at the same compression ratio, or achieve a more ideal compression ratio at the same reconstruction quality.

[0009] In a possible implementation, the method further includes:

[0010] Each semantic base in the semantic base set is visualized to generate a semantic base image block set; a plurality of semantic base image blocks are obtained from the semantic base image block set according to reconstruction information; the plurality of semantic base image blocks are spliced ​​into a reconstructed image of the image to be compressed; a compensation operation is performed on the reconstructed image with the image to be compressed as a compensation target to obtain a compensated image; a difference between the compensated image and the image to be compressed is obtained; the compressed data also includes the difference and compensation information of each image to be compressed, and the compensation information is used to indicate the compensation operation.

[0011] In the present application, the compressed data also includes difference and compensation information, so that the receiving device can further fill in more details for the image based on this information, thereby improving the quality of image reconstruction.

[0012] In a possible implementation, the semantic basis set is sent to the receiving device via a reliable transmission protocol, and the reconstruction information, difference and compensation information are sent to the receiving device via an unreliable transmission protocol.

[0013] In this application, since the semantic base set and the reconstruction information, difference, and compensation information have different importance levels, different transmission protocols can be used for transmission to ensure the effective transmission of important information and avoid unnecessary overhead.

[0014] In a possible implementation, the method further includes:

[0015] The semantic base set is stored in a long-term memory, and the reconstruction information, difference and compensation information are stored in a short-term memory; after sending the semantic base set and the reconstruction information, difference and compensation information of each image to be compressed in the image set to the receiving device, and when receiving information indicating successful reception of the semantic base set from the receiving device, the semantic base set is deleted from the long-term memory; after sending the reconstruction information, difference and compensation information of each image to be compressed to the receiving device, the reconstruction information, difference and compensation information of the image to be compressed are deleted from the short-term memory.

[0016] In the present application, the reconstruction information, difference and compensation information correspond to a single image to be compressed, and the semantic base set corresponds to the entire image set. Therefore, after the reconstruction information, difference and compensation information of a single image to be compressed are sent, the reconstruction information, difference and compensation information of the image to be compressed can be deleted to avoid resource occupation; after the semantic base set and the reconstruction information, difference and compensation information of all images to be compressed are sent, and it is determined that the semantic base set has been successfully received by the receiving device, the semantic base set can be deleted to avoid data loss when retransmission is required.

[0017] In a possible implementation manner, a semantic base in the semantic base set is obtained by clustering multiple semantic features.

[0018] The second aspect of the present application provides a data decompression method, which is applied to a receiving device:

[0019] Receive compressed data of an image set sent by a transmitting device, the compressed data including a semantic base set and reconstruction information of each original image in the image set, the semantic base set is obtained by clustering a semantic feature set, the semantic feature set includes semantic features of sub-image blocks of each original image in the image set, and each original image is divided into multiple sub-image blocks; the reconstruction information includes identifiers of multiple semantic bases in the semantic base set, and the multiple semantic bases correspond to clustering results of semantic features of each sub-image block of the original image; visualize each semantic base in the semantic base set to generate a semantic base image block set; obtain a reconstructed image of each original image, the reconstructed image is obtained by selecting multiple semantic base image blocks from the semantic base image block set according to the reconstruction information, and splicing the multiple semantic base image blocks.

[0020] In the present application, clustering operations are performed on the semantic feature set to fully exploit the redundant information within and between the original images, and obtain a semantic base set containing the key information of the image set. The semantic base set is visualized to obtain a semantic base image block set, and the reconstructed image of the original image can be obtained by combining the reconstruction information of each original image. Data transmission based on such a semantic communication method can improve the compression efficiency of images or videos, achieve a more ideal reconstruction quality at the same compression ratio, or achieve a more ideal compression ratio at the same reconstruction quality.

[0021] In a possible implementation, the compressed data further includes compensation information and a difference value of each of the multiple original images, the compensation information is used to indicate a compensation operation performed on the reconstructed image with the original image as a compensation target, the difference value is a difference between the compensated image and the original image, and the compensated image is obtained after the compensation operation is performed on the reconstructed image;

[0022] The method also includes:

[0023] A compensated image is obtained for each original image; and a high-quality reconstructed image is obtained for each original image, wherein the high-quality reconstructed image is obtained based on the compensated image and the difference.

[0024] In the present application, the compressed data also includes difference and compensation information, so the receiving device can further fill in more details for the image based on this information to improve the quality of image reconstruction.

[0025] In a possible implementation, a semantic basis set sent by a transmitting device is received through a reliable transmission protocol, and reconstruction information, difference and compensation information sent by the transmitting device are received through an unreliable transmission protocol.

[0026] In this application, since the semantic base set and the reconstruction information, difference, and compensation information have different importance levels, different transmission protocols can be used for transmission to ensure the effective transmission of important information and avoid unnecessary overhead.

[0027] In a possible implementation, the method further includes:

[0028] The semantic base set is stored in a long-term memory, and the reconstruction information, difference and compensation information are stored in a short-term memory; after a high-quality reconstructed image of the original image is obtained based on the semantic base set, the reconstruction information, difference and compensation information of the original image, the reconstruction information, difference and compensation information of the original image are deleted from the short-term memory; after a high-quality reconstructed image of each original image in the image set is obtained, the semantic base set is deleted from the long-term memory.

[0029] In the present application, the reconstruction information, difference and compensation information correspond to a single original image, and the semantic basis set corresponds to the entire image set. Therefore, after obtaining a high-quality reconstructed image of the original image based on the reconstruction information, difference and compensation information of the single original image, the reconstruction information, difference and compensation information of the original image can be deleted to avoid resource occupation; after obtaining high-quality reconstructed images of all original images, the semantic basis set can be deleted to avoid data missing.

[0030] The third aspect of the present application provides a sending end device:

[0031] The method comprises a plurality of functional modules, which interact with each other to implement the method in the first aspect and its embodiments. The plurality of functional modules can be implemented based on software, hardware or a combination of software and hardware, and the plurality of functional modules can be arbitrarily combined or divided based on the specific implementation.

[0032] A fourth aspect of the present application provides a receiving end device:

[0033] The method comprises a plurality of functional modules, which interact with each other to implement the method in the second aspect and its respective embodiments. The plurality of functional modules can be implemented based on software, hardware or a combination of software and hardware, and the plurality of functional modules can be arbitrarily combined or divided based on the specific implementation.

[0034] A fifth aspect of the present application provides a transmitting end device:

[0035] It includes a processor, the processor is coupled to a memory, and the memory is used to store instructions. When the instructions are executed by the processor, the sending end device executes the method in the first aspect as mentioned above.

[0036] A sixth aspect of the present application provides a receiving end device:

[0037] It includes a processor, the processor is coupled to a memory, and the memory is used to store instructions. When the instructions are executed by the processor, the receiving end device executes the method in the second aspect as mentioned above.

[0038] A seventh aspect of the present application provides a computer-readable storage medium:

[0039] A computer program or instruction is stored thereon, and it is characterized in that when the computer program or instruction is executed, the computer program or instruction is stored thereon, and when the computer program or instruction is executed, the computer executes the method as described in the first aspect or the second aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A schematic diagram of an application scenario of this application;

[0041] Figure 2 A schematic diagram of the flow chart of the data compression method of the present application;

[0042] Figure 3 A schematic diagram of the process of processing an image set;

[0043] Figure 4 A schematic diagram of dividing an image to be compressed into sub-image blocks;

[0044] Figure 5 A schematic diagram of clustering semantic features into semantic bases;

[0045] Figure 6 A schematic diagram of the process of obtaining residual semantic information of an image;

[0046] Figure 7 Schematic diagram of sending semantic base and image residual semantic information;

[0047] Figure 8 is a schematic diagram of an image to be compressed, a high-quality reconstructed image and a reconstructed image;

[0048] Fig. 9 The curves of PSNR of reconstructed images with various compression schemes in the transmission scenario of image sets with complex correlations versus the number of bits per pixel BPP;

[0049] Fig.10 The curves of the multi-scale structural similarity MS-SSIM of the reconstructed images of various compression schemes in the transmission scenario of image sets with complex correlations and the number of bits per pixel BPP;

[0050] Fig.11 The PSNR of the reconstructed image varies with BPP for various compression schemes in a typical video clip transmission scenario;

[0051] Fig.12 The curves of MS-SSIM versus BPP of the reconstructed image of various compression schemes in a typical video clip transmission scenario;

[0052] Fig.13 A schematic diagram of the structure of the transmitting end device in this application;

[0053] Fig.14 A schematic diagram of the structure of a receiving device in this application;

[0054] Fig.15 Another structural diagram of the transmitting device and the receiving device in this application. DETAILED DESCRIPTION

[0055] The following describes the embodiments of the present application in conjunction with the accompanying drawings. Obviously, the described embodiments are only embodiments of a part of the present application, rather than all embodiments. It is known to those skilled in the art that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0056] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0057] The demand for data transmission in wireless networks has experienced explosive growth in the past few years, and image and video transmission is one of the main applications that consume network communication resources. In future wireless networks, the demand for image and video transmission from AI-enabled tasks such as industrial IoT and smart driving will merge with the further growing demand for traditional image and video transmission, posing more stringent challenges to the deployment of network communication resources. How to achieve high-quality, low-cost transmission of images and videos with complex correlations is a key technical issue to improve user experience and break through network performance bottlenecks.

[0058] To support the above-mentioned diversified image and video transmission needs, the technologies currently used in the industry include image and video compression technologies. Image compression technology takes a single image as the object, and by mining the statistical laws of pixel values, removes the information redundancy contained in the image to seek the optimal balance between image reconstruction quality and compression ratio. It can be regarded as a practical application of variable-length source coding. Image compression technology mainly includes three main links: feature transformation, quantization, and entropy coding. Its guiding ideology is: first decouple the inter-pixel correlation of the information contained, and then select the information according to the physiological characteristics of the human eye, task requirements, etc. Image compression technology only considers the internal redundancy of the image, ignores the correlation between images, and has a limited performance upper limit. Video compression technology processes image (frame) sequences with high mutual correlation as the object, and realizes compression by comprehensively mining the statistical laws contained in the image and between images, which can obtain a further compression gain compared with image compression technology. In the implementation logic of image compression, video compression technology introduces the inter-frame prediction method to extract the information that can be predicted by the reference frame from the video frame to be compressed, and completes the subsequent processing of the remaining information through image compression technology. Video compression technology has strict requirements on the statistical characteristics of image sources. For image sequences containing significant correlations, video compression technology can achieve compression gains several times higher than image compression technology, but its performance is not ideal when directly used to compress general image sources. The technical characteristics of image and video compression are summarized in Table 1:

[0059] Table 1

[0060]

[0061] In recent years, the idea of ​​communication based on semantics has provided a new perspective for the optimization design of image and video transmission schemes. Semantic communication takes "expression" as its basic goal, focusing on the content of transmitted information rather than coding symbols. On the basis of realizing background knowledge synchronization between the sender and the receiver, it can fully meet the communication needs while reducing the volume of information representation. In semantic communication, the information to be transmitted needs to be constructed and selected from the source / task semantic knowledge base according to its own characteristics under the guidance of the communication task requirements, and mapped to the semantic space based on this, so as to improve the efficiency of information transmission. The prior art (CN113705245A) proposes a semantic base representation framework for the intention of the communication object, and designs a point-to-point semantic communication system based on this framework. However, the above-mentioned prior art does not provide a semantic base construction and application method for specific image and video transmission tasks, and cannot give full play to the flexibility of semantic information in representation granularity and representation efficiency.

[0062] The present application provides a data compression method and related equipment for improving the compression efficiency of images or videos, achieving more ideal reconstruction quality at the same compression ratio, or achieving more ideal compression ratio at the same reconstruction quality.

[0063] This application can be applied to Figure 1 In the system architecture shown, in a wireless communication network with a large number of image or video transmission requirements, by deploying corresponding data compression algorithms and data decompression algorithms at the transmitting end and the receiving end, efficient compression transmission of images or videos can be achieved. For example, for industrial production line monitoring cameras or drone cameras, the collected data passes through switches, wireless access points and other devices to the network aggregation node at the source compression position, and the network aggregation node compresses and encodes these export flows and sends the compressed flows to the 5G network. The compressed flows are transmitted through the 5G network and arrive at the receiving network aggregation node, which compresses and decodes these flows. The compressed and decoded data passes through the switch and reaches the console PC in the factory or the central control of the drone.

[0064] The sending device and the receiving device in this application can be any form of computing device, wherein the sending device can be, for example, Figure 1 The network aggregation node of the sending end shown in FIG. , the receiving end device may be, for example, Figure 1 The network aggregation node at the receiving end is shown in .

[0065] The present application can be applied to image set transmission scenarios with complex correlations and typical video clip transmission scenarios, wherein the image set transmission scenario with complex correlations refers to images from different task scenarios being aggregated at the originating network access point waiting for transmission, the correlation between images in the same task scenario may be high or low, and the image set presents certain clustering characteristics. The typical video clip transmission scenario refers to multiple video sequences acquired from different cameras being aggregated at the originating network access point waiting for transmission, the correlation between images in the same video sequence is high, and the image set presents obvious clustering characteristics.

[0066] See also Figure 2 , the following is an introduction to a process of the data compression method in this application:

[0067] 201. Acquire an image set, where the image set includes a plurality of images to be compressed;

[0068] The receiving end device acquires a plurality of images to be compressed. These images to be compressed are, for example, acquired by different cameras in a factory. After acquiring the images to be compressed, these cameras send these images to be compressed to the sending end device.

[0069] The receiving device in the present application includes an image set splitting module, which adopts a Simsiam network structure and includes a ResNet-50 backbone network and two fully connected layers. The image set splitting module performs correlation analysis on multiple images to be compressed and divides them into multiple image sets, wherein the correlation between the images to be compressed in the same image set is greater than the correlation between the images to be compressed in different image sets, for example, the images to be compressed collected by camera A are divided into image set A, and the images to be compressed collected by camera B are divided into image set B.

[0070] 202. Divide each to-be-compressed image into a plurality of sub-image blocks;

[0071] The receiving device in the present application also includes an image processing module, which performs the same operation on each image set. Figure 3 , the following introduces the process of processing image set A.

[0072] The image processing module first divides each image to be compressed in the image set A into blocks, see Figure 4 , each image to be compressed can be divided into multiple sub-image blocks.

[0073] 203. Acquire a semantic feature set, where the semantic feature set includes semantic features of each sub-image block of the image to be compressed in the image set;

[0074] The sub-image blocks of each image to be compressed in the image set A are input into the semantic feature generation network, which is composed of a convolutional layer and a GDN activation function. The semantic feature generation network is used to extract the semantic features of each sub-image block of each image to be compressed, thereby obtaining a set of semantic features of the image set A, which contains important visual information of the corresponding sub-image block.

[0075] 204. Perform clustering operation on the semantic feature set to obtain a semantic base set;

[0076] The semantic feature generation network inputs the semantic feature set of the image set A into the feature space for clustering operation. The essence of clustering operation is to cluster multiple semantic features into a semantic base. In the above clustering operation process, the image to be compressed can be represented by the semantic features of all the sub-image blocks divided from it. This part of information is recorded as representation information. For example, please refer to Figure 5 Taking the image 1 to be compressed and the image 2 to be compressed as examples, the image 1 to be compressed can be represented by semantic features 1 to 6, and the image 2 to be compressed can be represented by semantic features A to F.

[0077] If sub-image block 1, sub-image block 2 and sub-image block 3 of the image 1 to be compressed are relatively close, the semantic features 1, 2 and 3 of the image 1 to be compressed are clustered to obtain a semantic base 1.

[0078] If the sub-image block 4 of the image to be compressed 1 is relatively close to the sub-image block D of the image to be compressed, the semantic feature 4 of the image to be compressed 1 and the semantic feature D of the image to be compressed 2 are clustered to obtain the semantic base 2 .

[0079] If the sub-image block 5 of the image to be compressed 1 is relatively close to the sub-image block E of the image to be compressed, the semantic feature 5 of the image to be compressed 1 and the semantic feature E of the image to be compressed 2 are clustered to obtain the semantic base 3 .

[0080] If the sub-image block 6 of the image to be compressed 1 is relatively close to the sub-image block F of the image to be compressed, the semantic feature 6 of the image to be compressed 1 and the semantic feature F of the image to be compressed 2 are clustered to obtain a semantic base 4 .

[0081] If the sub-image block A, the sub-image block B and the sub-image block C of the image to be compressed 2 are relatively close, the semantic features A, the semantic features B and the semantic features C of the image to be compressed 2 are clustered to obtain a semantic base 5 .

[0082] Taking the example that the image set A only includes the image 1 to be compressed and the image 2 to be compressed, the semantic base set of the image set A includes the above-mentioned semantic base 1, semantic base 2, semantic base 3, semantic base 4 and semantic base 5.

[0083] 205. Obtain reconstruction information of each image to be compressed, where the reconstruction information includes identifiers of multiple semantic bases in the semantic base set, and the multiple semantic bases respectively correspond to clustering results of semantic features of each sub-image block of the image to be compressed;

[0084] The obtained semantic base is used to replace the semantic features that aggregate the semantic base, that is, the representation information of the image to be compressed is updated. Specifically, semantic base 1 is obtained by clustering semantic features 1, 2 and 3, so semantic base 1 replaces semantic features 1, 2 and 3; semantic base 2 is obtained by clustering semantic features 4 and D, so semantic base 2 replaces semantic features 4 and D; semantic base 3 is obtained by clustering semantic features 5 and E, so semantic base 2 replaces semantic features 4 and E; semantic base 4 is obtained by clustering semantic features 6 and F, so semantic base 2 replaces semantic features 6 and F; semantic base 5 is obtained by clustering semantic features A, B and C, so semantic base 2 replaces semantic features A, B and C.

[0085] After performing the above operations, the reconstruction information of each image to be compressed is obtained, and the reconstruction information includes a sequence composed of the identifiers of each semantic basis of the updated representation information of the image to be compressed. For example, the reconstruction information of image 1 to be compressed includes 1 (semantic basis 1), 1 (semantic basis 1), 1 (semantic basis 1), 2 (semantic basis 2), 3 (semantic basis 3), and 4 (semantic basis 4); the reconstruction information of image 2 to be compressed includes 5 (semantic basis 5), 5 (semantic basis 5), 5 (semantic basis 5), 6 (semantic basis 6), 7 (semantic basis 7), and 8 (semantic basis 8).

[0086] Please continue reading Figure 3 , the image processing module will also input the semantic base set of image set A into the image block generation network, which is composed of a deconvolution layer and an IGDN activation function. The image block generation network outputs the semantic base image block corresponding to each semantic base, that is, the semantic base is visualized to obtain a semantic base image block set. After that, the corresponding semantic base image block is selected from the semantic base image block set according to the reconstruction information of each image to be compressed. For example, the reconstruction information of the image 1 to be compressed includes 1, 1, 1, 2, 3, 4, then the semantic base image block corresponding to the above semantic base is selected from the semantic base image block set, and these semantic base image blocks are combined into the reconstructed image of the image to be compressed.

[0087] See also Figure 6, the transmitting end device also includes an image residual coding module, the image processing module inputs the reconstructed image of each image to be compressed in the image subset A into the image residual coding module, and the image residual coding module also obtains each image to be compressed in the image subset A. Taking the processing of the image 1 to be compressed as an example, the image residual coding module takes the image 1 to be compressed as the compensation target, and performs a compensation operation on the reconstructed image of the image 1 to be compressed. For example, it can be to obtain a vector describing pixel movement information for efficient image compression, and superimpose it on the image, so as to obtain a compensated image of the image 1 to be compressed, and the image residual coding module also records compensation information, which is used to indicate the above-mentioned compensation operation. The image residual coding module also calculates the difference between the above-mentioned compensated image and the image 1 to be compressed, and obtains the difference between the compensated image and the image 1 to be compressed. In an optional manner, the image residual coding module also compresses and encodes the above-mentioned compensation information and the difference, and the above-mentioned compensation information and the difference are recorded as image residual semantic information. Correspondingly, the receiving end device includes an image residual decoding module. After the receiving end device obtains the reconstructed image of the image to be compressed 1, the image residual decoding module first performs a compensation operation on the reconstructed image according to the compensation information to obtain a compensated image. Then, the compensated image is added to the difference to obtain a high-quality reconstructed image of the image to be compressed 1.

[0088] In summary, the sending end device obtains the semantic base set of image set A, the reconstruction information of each image to be compressed in image set A, and the image residual semantic information, which are recorded as the compressed data of image set A. Similarly, the sending end device obtains the above compressed data for each other image set.

[0089] 206. Send the compressed data of the image set to the receiving device, including the semantic basis set and the reconstruction information.

[0090] See also Figure 7, the sending device and the receiving device transmit data according to the application layer protocol, network layer protocol, data link layer protocol and physical layer protocol. Among them, after the sending device obtains the compressed data of each image set, it stores the semantic base set of each image set in the local long-term memory, and stores the reconstruction information and image residual semantic information of each image to be compressed in the local short-term memory. The sending device will send the compressed data of each image set to the receiving device. Taking the compressed data of image set A as an example, the sending device sends the semantic base set of image set A to the receiving device through a reliable transmission protocol, and sends the reconstruction information and image residual semantic information of each image to be compressed in image set A to the receiving device through an unreliable transmission protocol. Specifically, when the transmitting end device completes the transmission of the reconstruction information and image residual semantic information of each image to be compressed, the reconstruction information and image residual semantic information of the image to be compressed are deleted from the short-term memory; when the transmitting end device completes the transmission of the compressed data of the image set A and receives the information indicating the successful reception of the semantic base set of the image set A from the receiving end device, the semantic base set of the image set A is deleted from the long-term memory. Optionally, in the message header of the message used to transmit the above-mentioned information, the semantic base set, the reconstruction information and the image residual semantic information can also be distinguished by different protocol fields.

[0091] It should be understood that the manner in which the transmitting device sends compressed data of other image sets is similar to that described above, and will not be described in detail.

[0092] The following describes the processing flow after the receiving device receives the compressed data of the image set:

[0093] Similarly, the receiving device stores the semantic basis set of each image set in the long-term memory, and stores the reconstruction information and image residual semantic information of each image to be compressed in the local short-term memory.

[0094] Still taking the image set A as an example, the receiving end device also includes an image processing module similar to the above. The receiving end device first inputs the semantic base set of image set A into the image block generation network in the image processing module, and the image block generation network outputs the corresponding semantic base image block set. Afterwards, the receiving end device selects the corresponding semantic base image block from the above semantic base image block set according to the reconstruction information of the image to be compressed. Taking the image to be compressed 1 as an example, since the reconstruction information of the image to be compressed 1 includes 1, 1, 1, 2, 3, 4, the semantic base image blocks corresponding to semantic base 1, semantic base 2, semantic base 3, and semantic base 4 are selected, and the above semantic base image blocks are spliced ​​into the reconstructed image of the image to be compressed 1 in the order of the above sequence.

[0095] Then, the reconstructed image is supplemented according to the supplementary information of the image to be compressed 1 to obtain a compensated image of the image to be compressed 1. Finally, the difference of the image to be compressed 1 is added to the compensated image to obtain a high-quality reconstructed image of the image to be compressed 1.

[0096] Based on the above operations, the receiving device can obtain a high-quality reconstructed image of each image to be compressed in each image set. Figure 7 When the receiving device obtains a high-quality reconstructed image of an image to be compressed, the receiving device deletes the reconstruction information and residual semantic information of the image to be compressed in the short-term memory; when the high-quality reconstructed images of all images to be compressed in an image set are obtained, the receiving device deletes the semantic basis set of the image set in the long-term memory.

[0097] In one possible implementation, the transmitting device may also only send the semantic basis set of the image set and the reconstruction information of each image to be compressed in the image set to the receiving device. After receiving the above information, the receiving device obtains the reconstructed image of each image to be compressed in the manner described above. The above method can be applicable to scenarios where the image and video transmission quality is not high.

[0098] See also Figure 8 , comparing the image to be compressed, the reconstructed image and the high-quality reconstructed image, it is not difficult to see that the reconstructed image has restored the main information of the image to be compressed, and the peak signal-to-noise ratio PSNR of the image to be compressed is 29.63dB. The high-quality reconstructed image has further supplemented the detail information, and the peak signal-to-noise ratio PNSR of the image to be compressed is 39.89dB, which has a higher reconstruction accuracy.

[0099] See also Fig. 9 , Fig. 9The PSNR of the reconstructed image of various compression schemes in the transmission scenario of an image set with complex correlation varies with the bit number per pixel BPP. Wherein, the comparison scheme 1 is JPEG2000, and the comparison scheme 2 can be specifically referred to ".Balle, D.Minnen, S.Singh, SJHwang, and N.Johnston, "Variationalimage compression with a scale hyperprior," International Conference onLearning Representations, 2018." J; the comparison scheme 3 can be specifically referred to "G.Lu, W.Ouyang, D.Xu et al., "Dvc: An end-to-end deep video compression framework," Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2019.". Among them, "JPEG2000" corresponds to the comparison scheme 1, "DL w / o corr" corresponds to the comparison scheme 2, "DL w / corr" corresponds to the comparison scheme 3, and "Proposed" corresponds to the compression scheme of this application. It should be noted that the ordinate corresponding to the curve of the compression scheme of the present application is the PSNR of the aforementioned high-quality reconstructed image. It is not difficult to see that the compression scheme of the present application can improve the PSNR by 0.5-1.5dB compared with Comparative Schemes 1, 2 and 3.

[0100] See also Fig.10 , Fig.10 The graph shows the variation curve of the multi-scale structural similarity MS-SSIM of the reconstructed image of various compression schemes in the transmission scenario of an image set with complex correlations with the number of bits per pixel BPP. It should be noted that the ordinate corresponding to the curve of the compression scheme of the present application is the MS-SSIM of the aforementioned high-quality reconstructed image. It is not difficult to see that the compression scheme of the present application is not inferior to Comparative Scheme 1, Comparative Scheme 2, and Comparative Scheme 3 in terms of MS-SSIM.

[0101] See also Fig.11 and Fig.12 , Fig.11 and Fig.12The curves of PSNR of the reconstructed image with BPP and the curves of MS-SSIM with BPP of various compression schemes in a typical video clip transmission scenario are shown respectively. It should be noted that the ordinate corresponding to the curve of the compression scheme of the present application is the PSNR of the aforementioned high-quality reconstructed image. It is not difficult to see that the compression scheme of the present application is close to the performance of comparison scheme 2, is better than other comparison schemes, and has a more obvious performance advantage under low BPP conditions.

[0102] In the present application, clustering operations are performed on the semantic feature set at the transmitting end device, which can fully exploit the redundant information within and between the images to be compressed, and obtain a semantic base set containing the key information of the image set. The identifier of the semantic base corresponding to the semantic feature of each sub-image block of the image to be compressed is used as the reconstruction information of the image to be compressed, and the semantic base set is added to form the compressed data of the image set. The semantic base set is visualized at the receiving end device to obtain a semantic base image block set, and the reconstructed image of the original image can be obtained in combination with the reconstruction information of each original image. Data transmission based on such a semantic communication method can improve the compression efficiency of images or videos, achieve a more ideal reconstruction quality at the same compression ratio, or achieve a more ideal compression ratio at the same reconstruction quality.

[0103] The following is an introduction to the sending device and the receiving device in this application:

[0104] See also Fig.13 The sending end device 1300 in the present application includes an acquisition module 1301, a processing module 1302, and a sending module 1303.

[0105] An acquisition module 1301 is used to acquire an image set, where the image set includes a plurality of images to be compressed;

[0106] The processing module 1302 is used to divide each image to be compressed into a plurality of sub-image blocks;

[0107] The processing module 1302 is further used to obtain a semantic feature set, where the semantic feature set includes semantic features of each sub-image block of the image to be compressed in the image set;

[0108] The processing module 1302 is further used to perform clustering operation on the semantic feature set to obtain a semantic base set;

[0109] The processing module 1302 is further used to obtain reconstruction information of each image to be compressed, where the reconstruction information includes identifiers of multiple semantic bases in the semantic base set, and the multiple semantic bases respectively correspond to clustering results of semantic features of each sub-image block of the image to be compressed;

[0110] The sending module 1303 is used to send the compressed data of the image set, including the semantic basis set and reconstruction information, to the receiving end device.

[0111] In one possible implementation,

[0112] The processing module 1302 is further used to visualize each semantic base in the semantic base set to generate a semantic base image block set;

[0113] The processing module 1302 is further configured to obtain a plurality of semantic base image blocks from the semantic base image block set according to the reconstruction information;

[0114] The processing module 1302 is further used to splice the multiple semantic base image blocks into a reconstructed image of the image to be compressed;

[0115] The processing module 1302 is further used to perform a compensation operation on the reconstructed image with the image to be compressed as a compensation target to obtain a compensated image;

[0116] The processing module 1302 is further used to obtain the difference between the compensated image and the image to be compressed;

[0117] The compressed data also includes the difference and compensation information of each image to be compressed, and the compensation information is used to indicate the compensation operation.

[0118] In one possible implementation,

[0119] The sending module 1303 is specifically used to send the semantic basis set to the receiving end device through a reliable transmission protocol, and send the reconstruction information, difference and compensation information to the receiving end device through an unreliable transmission protocol.

[0120] In a possible implementation, the transmitting end device 1300 further includes a storage module 1304 .

[0121] The storage module 1304 is further used to store the semantic base set through the long-term memory and store the reconstruction information, difference and compensation information through the short-term memory;

[0122] The storage module 1304 is further configured to send the semantic base set, the reconstruction information, the difference and the compensation information of each image to be compressed in the image set to the receiving end device, and upon receiving information indicating that the semantic base set is successfully received from the receiving end device, delete the semantic base set from the long-term storage;

[0123] The storage module 1304 is further configured to delete the reconstruction information, difference and compensation information of each image to be compressed from the short-term memory after sending the reconstruction information, difference and compensation information of each image to be compressed to the receiving end device.

[0124] In a possible implementation manner, a semantic base in the semantic base set is obtained by clustering multiple semantic features.

[0125] See also Fig.14 The receiving device 1400 in the present application includes a receiving module 1401, a generating module 1402, and an acquiring module 1403.

[0126] The receiving module 1401 is used to receive compressed data of an image set sent by a transmitting device, wherein the compressed data includes a semantic base set and reconstruction information of each original image in the image set, wherein the semantic base set is obtained by performing a clustering operation on a semantic feature set, and the semantic feature set includes semantic features of a sub-image block of each original image in the image set, wherein each original image is divided into a plurality of sub-image blocks;

[0127] The reconstruction information includes identifiers of multiple semantic bases in the semantic base set, and the multiple semantic bases respectively correspond to clustering results of semantic features of each sub-image block of the image to be compressed;

[0128] A generating module 1402, configured to visualize each semantic base in the semantic base set to generate a semantic base image block set;

[0129] The acquisition module is used to acquire a reconstructed image of each original image. The reconstructed image is obtained by selecting a plurality of semantic base image blocks from a set of semantic base image blocks according to reconstruction information and splicing the plurality of semantic base image blocks.

[0130] In a possible implementation, the compressed data further includes compensation information and a difference value of each of the multiple original images, the compensation information is used to indicate a compensation operation performed on the reconstructed image with the original image as a compensation target, the difference value is a difference between the compensated image and the original image, and the compensated image is obtained after the compensation operation is performed on the reconstructed image;

[0131] The acquisition module is also used to acquire a compensated image of each original image;

[0132] The acquisition module is also used to acquire a high-quality reconstructed image of each original image, where the high-quality reconstructed image is obtained based on the compensated image and the difference.

[0133] In one possible implementation,

[0134] The receiving module 1401 is used to receive the semantic basis set sent by the sending end device through a reliable transmission protocol, and to receive the reconstruction information, difference and compensation information sent by the sending end device through an unreliable transmission protocol.

[0135] In a possible implementation, the receiving device 1400 further includes a storage module 1404 .

[0136] The storage module 1404 stores the semantic base set in a long-term memory and stores the reconstruction information, difference and compensation information in a short-term memory;

[0137] After obtaining a high-quality reconstructed image of the original image according to the semantic basis set, the reconstruction information, the difference and the compensation information of the original image, the storage module 1404 deletes the reconstruction information, the difference and the compensation information of the original image in the short-term memory;

[0138] The storage module 1404 is used to delete the semantic basis set in the long-term memory after obtaining a high-quality reconstructed image of each original image in the image set.

[0139] Fig.15 1 is a schematic diagram of the structure of a device provided in an embodiment of the present application. The device 1500 may be the aforementioned sending end device or receiving end device, including one or more central processing units (CPU) 1501 and a memory 1505, wherein the memory 1505 stores one or more application programs or data. Among them, the memory 1505 may be a volatile storage or a persistent storage. The program stored in the memory 1505 may include one or more modules, each of which may include a series of instruction operations on the server. Furthermore, the central processing unit 1501 may be configured to communicate with the memory 1505, and execute a series of instruction operations in the memory 1505 on the device 1500. The device 1500 may also include one or more power supplies 1502, one or more wired or wireless network interfaces 1503, one or more input and output interfaces 1504, and / or, one or more operating systems, such as WindowsServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc. The central processor 1501 can execute the operations of the sending end device or the receiving end device in the illustrated embodiment, and the details are not repeated here.

[0140] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0141] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0142] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0143] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0144] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

Claims

1. A data compression method, applied to a sending end device, characterized in that: include: Acquire an image set, wherein the image set includes a plurality of images to be compressed; Dividing each of the to-be-compressed images into a plurality of sub-image blocks; Acquire a semantic feature set, wherein the semantic feature set includes semantic features of the sub-image blocks of each of the images to be compressed in the image set; Performing clustering operation on the semantic feature set to obtain a semantic base set; Acquire reconstruction information of each of the images to be compressed, the reconstruction information including identifiers of a plurality of semantic bases in the semantic base set, the plurality of semantic bases respectively corresponding to clustering results of the semantic features of each of the sub-image blocks of the image to be compressed; The compressed data of the image set, including the semantic basis set and the reconstruction information, is sent to a receiving device.

2. The method according to claim 1, characterized in that The method further comprises: Visualizing each semantic base in the semantic base set to generate a semantic base image block set; Acquire a plurality of semantic base image blocks from the semantic base image block set according to the reconstruction information; splicing the multiple semantic base image blocks into a reconstructed image of the image to be compressed; Performing a compensation operation on the reconstructed image with the image to be compressed as a compensation target to obtain a compensated image; Obtaining a difference between the compensated image and the image to be compressed; The compressed data also includes the difference value and compensation information of each of the images to be compressed, and the compensation information is used to indicate the compensation operation.

3. The method according to claim 2, characterized in that The step of sending the compressed data of the plurality of images to be compressed to the receiving end device comprises: The semantic basis set is sent to the receiving end device through a reliable transmission protocol, and the reconstruction information, the difference and the compensation information are sent to the receiving end device through an unreliable transmission protocol.

4. The method according to claim 3, characterized in that The method further comprises: The semantic base set is stored in a long-term memory, and the reconstruction information, the difference and the compensation information are stored in a short-term memory; After sending the semantic base set, the reconstruction information, the difference and the compensation information of each of the to-be-compressed images in the image set to the receiving end device, and upon receiving information indicating that the semantic base set is successfully received from the receiving end device, deleting the semantic base set in the long-term memory; After sending the reconstruction information, the difference value and the compensation information of each image to be compressed to the receiving end device, the reconstruction information, the difference value and the compensation information of the image to be compressed are deleted in the short-term memory.

5. The method according to any one of claims 1 to 4, characterized in that A semantic base in the semantic base set is obtained by clustering a plurality of the semantic features.

6. A data decompression method, applied to a receiving device, characterized in that: include: Receive compressed data of an image set sent by a transmitting end device, the compressed data including a semantic base set and reconstruction information of each original image in the image set, the semantic base set is obtained by performing a clustering operation on a semantic feature set, the semantic feature set includes semantic features of a sub-image block of each original image in the image set, and each original image is divided into a plurality of sub-image blocks; The reconstruction information includes identifiers of a plurality of semantic bases in the semantic base set, and the plurality of semantic bases respectively correspond to clustering results of the semantic features of each of the sub-image blocks of the image to be compressed; Visualizing each semantic base in the semantic base set to generate a semantic base image block set; A reconstructed image of each of the original images is obtained, where the reconstructed image is obtained by selecting a plurality of semantic base image blocks from the semantic base image block set according to the reconstruction information and splicing the plurality of semantic base image blocks.

7. The method according to claim 6, characterized in that The compressed data further includes compensation information and a difference value of each of the multiple original images, wherein the compensation information is used to indicate a compensation operation performed on the reconstructed image with the original image as a compensation target, the difference value is a difference between a compensated image and the original image, and the compensated image is obtained after performing the compensation operation on the reconstructed image; The method further comprises: Acquire the compensated image of each of the original images; A high-quality reconstructed image is obtained for each of the original images, wherein the high-quality reconstructed image is obtained according to the compensated image and the difference.

8. The method according to claim 7, characterized in that The receiving device sends compressed data of an image set including: The semantic basis set sent by the sending end device is received through a reliable transmission protocol, and the reconstruction information, the difference and the compensation information sent by the sending end device are received through an unreliable transmission protocol.

9. The method according to claim 8, characterized in that The method further comprises: The semantic base set is stored in a long-term memory, and the reconstruction information, the difference and the compensation information are stored in a short-term memory; After obtaining the high-quality reconstructed image of the original image according to the semantic basis set, the reconstruction information of the original image, the difference and the compensation information, deleting the reconstruction information of the original image, the difference and the compensation information in the short-term memory; After the high-quality reconstructed image of each original image in the image set is obtained, the semantic basis set is deleted in the long-term memory.

10. A computing device, used as a sending end device, characterized in that: It comprises a plurality of functional modules, and the plurality of functional modules interact with each other to implement any one of the methods 1 to 5 above.

11. A computing device, used as a receiving device, characterized in that: It comprises a plurality of functional modules, and the plurality of functional modules interact with each other to implement any one of the methods 6 to 9 above.

12. A computing device, used as a sending end device, characterized in that: The device comprises a processor coupled to a memory, wherein the memory is used to store instructions. When the instructions are executed by the processor, the computing device performs the method according to any one of claims 1 to 5.

13. A computing device, used as a receiving device, characterized in that: The device comprises a processor coupled to a memory, wherein the memory is used to store instructions. When the instructions are executed by the processor, the computing device performs the method according to any one of claims 6 to 9.

14. A computer-readable storage medium, characterized in that: A computer program or instruction is stored thereon, wherein when the computer program or instruction is executed, the computer is caused to execute the method as claimed in any one of claims 1 to 9.

Images