Semantic information compression and transmission method and video image restoration method

Abstract

The present invention provides a method for compressing and transmitting semantic information and a method for restoring video images, the method for compressing semantic information comprising: extracting the potential features of each frame image from a video to be compressed by a potential feature extractor, generating a plurality of potential feature maps accordingly, using the potential feature maps as semantic information, each potential feature map comprising a plurality of channels, and the plurality of channels of different potential feature maps correspond to each other one by one; calculating the correlation between the plurality of corresponding channels of the potential feature map of each two adjacent frames of images based on the semantic information of each two adjacent frames of images, and obtaining a plurality of correlation values; sorting the plurality of correlation values ​​from large to small, and selecting the plurality of corresponding channels of the potential feature map of each two adjacent frames of images corresponding to the correlation in the front sorting by a channel merging module based on a preset compression ratio to merge, so as to realize the compression of the semantic information in the video to be compressed. Through the above scheme, the compression efficiency of the semantic information of the video can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of communication and data processing, and in particular to a semantic information compression and transmission method and a video image restoration method. Background Art

[0002] The development of science and technology has greatly enriched people's living standards. The evolution of media communication has made user experience closer to the real world. The means of communication have become more popular and modular. The role of video in production and life has become increasingly prominent, and people are spending more time on streaming media. Compared with text, the amount of video data is many times that of text, so the amount of video data on the entire network is close to 70%.

[0003] Semantic communication uses computing power at both the sending and receiving ends at the expense of computing power. Generally, the feature vector of the video stream is extracted through a neural network, the feature vector is transmitted through the channel, and then the neural network is used to restore the transmitted image.

[0004] At present, some researchers calculate the entropy of feature vectors from the perspective of information entropy, and judge the importance of vector elements based on the size of entropy. Information with low entropy is not transmitted during communication transmission, and the transmitted data is compressed. However, this method requires the transmission of the index of the transmission vector element, and in an unreliable channel, the transmitted index will be wrong, resulting in image recovery failure. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a semantic information compression and transmission method and a video image restoration method to eliminate or improve one or more defects existing in the prior art.

[0006] One aspect of the present invention provides a semantic information compression method, comprising:

[0007] A latent feature extractor extracts latent features of each frame image from the compressed video, and generates a plurality of latent feature maps accordingly. The latent feature maps are used as semantic information. Each latent feature map includes a plurality of channels, and the plurality of channels of different latent feature maps correspond to each other one by one.

[0008] Calculating correlations between multiple corresponding channels of potential feature maps of each of the two adjacent frames of images based on semantic information of each of the two adjacent frames of images, and obtaining multiple correlation values;

[0009] The values ​​of multiple correlations are sorted from large to small, and a channel merging module selects multiple corresponding channels of the potential feature map of each two adjacent frames of images corresponding to the correlations sorted first based on a preset compression ratio to merge to form a shared channel. The semantic information corresponding to the shared channel and the respective non-shared channels constitutes the semantic information of each two adjacent frames of images after compression, thereby achieving compression of the semantic information in the video to be compressed.

[0010] In some embodiments of the present invention, the step of calculating the correlation between multiple corresponding channels of the potential feature map of each two adjacent frames of images based on the semantic information of each two adjacent frames of images includes:

[0011] The latent feature maps of multiple corresponding channels in the latent feature map of each of two adjacent frames of images are converted into multiple pairs of semantic vectors, wherein the two semantic vectors of each corresponding channel form a pair of semantic vectors;

[0012] The Pearson correlation coefficient between each pair of semantic vectors is calculated to obtain the correlation between multiple corresponding channels of the potential feature map of each two adjacent frames of images.

[0013] In some embodiments of the present invention, the feature extractor includes multiple convolutional layers and multiple GDN activation functions, wherein a GDN activation function is connected between every two convolutional layers, the first convolutional layer serves as the input of the feature extractor, and the last convolutional layer serves as the output of the feature extractor. The multiple convolutional layers are used to downsample each frame image of the compressed video, and correspondingly generate multiple potential feature maps, increase the number of channels of the image, and reduce the size of the potential feature map of each channel.

[0014] Another aspect of the present invention provides a semantic information transmission method, comprising:

[0015] The sending end compresses the semantic information of each frame image in the video to be transmitted using the aforementioned semantic information compression method;

[0016] The source-channel joint encoder encodes the semantic information of each frame image in the compressed video to be transmitted to obtain a semantic code;

[0017] The channel transmits the semantic coding of each frame image in the compressed video to be transmitted.

[0018] In some embodiments of the present invention, the source-channel joint encoder includes multiple convolutional layers and multiple GDN activation functions, wherein a GDN activation function is connected between every two convolutional layers, the first convolutional layer serves as the input of the source-channel joint encoder, and the last convolutional layer serves as the output of the source-channel joint encoder, and the multiple convolutional layers are used to encode semantic information.

[0019] In some embodiments of the present invention, the step of transmitting the semantic coding of each frame image in the compressed video to be transmitted through a channel also includes: transmitting the index of the shared channel of the potential feature map of each frame image in the compressed video to be transmitted.

[0020] Another aspect of the present invention provides a video image restoration method, comprising:

[0021] The receiving end receives the semantic coding of each frame image in the compressed video from the sending end, and the source channel joint decoder decodes the semantic coding to obtain semantic information of each frame image in the compressed video, wherein the semantic information includes the compressed shared channel and respective non-shared channels of two adjacent frame images before compression corresponding to each frame image, and the respective non-shared channels are respectively a first non-shared channel and a second non-shared channel;

[0022] The channel recovery module extracts the shared channel, the first non-shared channel and the second non-shared channel from the semantic information of each frame of the compressed video based on a preset compression ratio;

[0023] The semantic information of the shared channel is combined with the semantic information corresponding to the first non-shared channel and the second non-shared channel respectively to restore the semantic information of each two adjacent frames of images before compression, thereby obtaining the semantic information of each frame of the image in the decompressed video;

[0024] The latent feature restorer restores each frame of the video based on the semantic information of each frame in the decompressed video, thereby obtaining the transmitted video.

[0025] In some embodiments of the present invention, the source-channel joint decoder includes multiple convolutional layers and multiple GDN activation functions, wherein a GDN activation function is connected between every two convolutional layers, the first convolutional layer serves as the input of the source-channel joint decoder, and the last convolutional layer serves as the output of the source-channel joint decoder, and the multiple convolutional layers of the source-channel joint decoder are used to decode semantic coding.

[0026] In some embodiments of the present invention, the latent feature restorer includes multiple transposed convolutional layers, multiple GDN activation functions, convolutional layers and Tanh activation functions, wherein the number of the transposed convolutional layers and the GDN activation functions is the same, a GDN activation function is connected between every two transposed convolutional layers, the output of the last GDN activation function is connected to the convolutional layer and the Tanh activation function in sequence, the first transposed convolutional layer serves as the input of the latent feature restorer, and the Tanh activation function serves as the output of the latent feature restorer, and the multiple transposed convolutional layers are used to upsample the semantic information of each frame image in the decompressed video, reduce the number of channels of the image, and increase the size of the image of each channel.

[0027] Another aspect of the present invention provides an electronic device, which includes: a computer device, the computer device including a processor and a memory, the memory storing computer instructions, the processor being used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the electronic device implements the steps of the aforementioned semantic information compression method, semantic information transmission method and video image restoration method.

[0028] Another aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned semantic information compression method, semantic information transmission method and video image restoration method.

[0029] The semantic information compression and transmission method and video image restoration method of the present invention extract shared potential features based on the correlation of feature maps or feature matrices, merge them from the dimension of the channels of the feature maps or feature matrices, and then realize the compression of semantic information, which can improve the compression efficiency and has good compression performance.

[0030] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be achieved and obtained by the structures specifically indicated in the specification and the accompanying drawings.

[0031] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0033] Figure 1A flowchart of an implementation method of the semantic information compression method of the present invention;

[0034] Figure 2 A schematic diagram of a network structure of an implementation method of a potential feature extractor of the present invention;

[0035] Figure 3 This is a flow chart of an implementation of step S120 in the semantic information compression method of the present invention;

[0036] Figure 4 A schematic diagram of an example of channel incorporation for the present invention;

[0037] Figure 5 A flowchart of an implementation method of the semantic information transmission method of the present invention;

[0038] Figure 6 A schematic diagram of a network structure of an implementation of a source-channel joint encoder and a source-channel joint decoder of the present invention;

[0039] Figure 7 A schematic diagram of a process of an implementation method of a video image restoration method of the present invention;

[0040] Figure 8 A schematic diagram of an example of channel recovery of the present invention;

[0041] Fig. 9 A schematic diagram of a network structure of an implementation mode of a potential feature restorer of the present invention;

[0042] Fig.10 A schematic diagram of an example of the semantic information compression, transmission and video image restoration process of the present invention;

[0043] Fig.11 A schematic diagram comparing the effects of the video image restoration method of the present invention and the restoration of a transmitted image in the prior art;

[0044] Fig.12 The figure is a schematic diagram comparing the index amount transmitted in the semantic information transmission method of the present invention and that in the prior art. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0046] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0047] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0048] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0049] In view of the problem of video image compression in the process of semantic communication, and how to restore the compressed video image and improve the accuracy of video image restoration, and even restore the transmitted original video image and original video, the embodiments of the present invention disclose a semantic information compression method, a semantic information transmission method and a video image restoration method.

[0050] Figure 1 is a flowchart of a semantic information compression method according to an embodiment of the present invention. Figure 1 The semantic information compression method of the embodiment of the present invention comprises the following steps:

[0051] Step S110, a latent feature extractor extracts latent features of each frame image from the video to be compressed, and generates a plurality of latent feature maps accordingly. The latent feature maps are used as semantic information. Each latent feature map includes a plurality of channels, and the multiple channels of different latent feature maps correspond to each other one by one.

[0052] In an embodiment of the present invention, the potential features of each frame image extracted from the video to be compressed in step S110 constitute respective potential feature maps, and the potential feature maps can be represented by a potential feature matrix, that is, the potential feature matrix of each frame image can also be used as respective semantic information. Since each frame image in the input video to be compressed comes from the same background knowledge base, the semantic information obtained by the potential feature extractor comes from the same information representation space, so the number of channels of the potential feature map of each frame image in the video to be compressed is the same and corresponds one to one. In one embodiment of the present invention, the channel of the potential feature map can be each row of the potential feature matrix.

[0053] In an optional embodiment of the present invention, the potential feature extractor in step S110 includes multiple convolutional layers and multiple GDN activation functions, wherein a GDN activation function is connected between every two convolutional layers, the first convolutional layer serves as the input of the potential feature extractor, and the last convolutional layer serves as the output of the potential feature extractor, and the multiple convolutional layers are used to downsample each frame image in the video to be compressed, and generate multiple potential feature maps accordingly, thereby increasing the number of channels of the image and reducing the size of the potential feature map of each channel.

[0054] In the embodiment of the present invention, the potential feature map contained in each channel is the channel feature of the respective channel.

[0055] For example, see Figure 2 The latent feature extractor includes four convolutional layers and three GDN activation functions. A GDN activation function is connected between every two convolutional layers. The first convolutional layer is the input of the latent feature extractor, and the last convolutional layer is the output of the latent feature extractor. Two adjacent frames of the video to be compressed with a size of 3*512*1024 are input into the first convolutional layer of the latent feature extractor. The four convolutional layers downsample the images input to each layer. The size of the two latent feature maps output by the fourth convolutional layer is 64*64*128, so that the latent feature map output by the last convolutional layer has an increased number of channels (from 3 to 64) compared to the image input to the first convolutional layer, and the size of the feature map on each channel becomes smaller (from 512*1024 to 64*128). Similarly, the other two adjacent frames of the video to be compressed are also processed in the same way, which will not be repeated here.

[0056] Step S120, calculating the correlation between multiple corresponding channels of the potential feature map of each two adjacent frames of images based on the semantic information of each two adjacent frames of images, and obtaining multiple correlation values.

[0057] In an embodiment of the present invention, since each frame image in the input video to be compressed comes from the same background knowledge base, the semantic information obtained by the latent feature extractor comes from the same information representation space and has certain similarities. Therefore, the latent features or semantic information of the video image can be compressed according to the similarity between each two adjacent frames of images. By calculating the correlation between multiple corresponding channels (i.e., each row of the latent feature matrix) in the latent feature map of each two adjacent frames of images, the dimension of the information representation space can be effectively reduced.

[0058] In the optional embodiment of the present invention, please refer to Figure 3 The specific step of calculating the correlation between multiple corresponding channels of the potential feature map of each two adjacent frames of images based on the semantic information of each two adjacent frames of images in step S120 also includes:

[0059] Step S121, converting the latent feature maps of multiple corresponding channels in the latent feature maps of each of two adjacent frames of images into multiple pairs of semantic vectors, wherein the two semantic vectors of each corresponding channel form a pair of semantic vectors;

[0060] Step S122, calculating the Pearson correlation coefficient between each pair of semantic vectors, and obtaining the correlation between multiple corresponding channels of the potential feature map of each two adjacent frames of images.

[0061] In the embodiment of the present invention, step S121 is to expand each row of the two potential feature matrices of each two adjacent frames of images and convert them into multiple pairs of one-dimensional semantic vectors. In step S122, the Pearson correlation coefficient between the vectors is calculated for each pair of one-dimensional semantic vectors according to the following formula:

[0062]

[0063] Among them, r represents the Pearson correlation coefficient, X i and Y i Represent the elements of the two semantic vectors in each pair of one-dimensional semantic vectors, and Respectively represent the average values ​​of the elements in the two semantic vectors in each pair of one-dimensional semantic vectors, i = 1, ..., n, i represents the index of the semantic vector, and n represents the length of the semantic vector. The larger the value of the calculated Pearson correlation coefficient, the greater the correlation.

[0064] For example, the correlation between two potential feature maps of size 64*64*128 output by the potential feature extractor is calculated. Since the two potential feature maps have 64 channels, the correlation needs to be calculated 64 times to obtain 64 Pearson correlation coefficients. Similarly, the correlation needs to be calculated 64 times for other two adjacent frames in the compressed video.

[0065] Step S130, sorting the values ​​of multiple correlations from large to small, and the channel merging module selects multiple corresponding channels of the potential feature map of each two adjacent frames of images corresponding to the correlations sorted first based on a preset compression ratio to merge to form a shared channel, and the semantic information corresponding to the shared channel and the respective non-shared channels constitute the semantic information of each two adjacent frames of images after compression, thereby achieving compression of the semantic information in the video to be compressed.

[0066] In the embodiment of the present invention, the channel features contained in each channel in the shared channel (Channel) - the potential feature map (semantic information) is the shared potential feature. The non-shared channel is the channel that is not merged in the potential feature map of each two adjacent frames of images.

[0067] Exemplarily, the preset compression ratio is 50%, and the channel merging module is used to sort the values ​​of the 64 Pearson correlation coefficients of the two potential feature maps with a size of 64*64*128 from large to small, and the corresponding channels of the two potential feature maps corresponding to the 32 Pearson correlation coefficients ranked first (some rows corresponding to the two potential feature matrices) are selected for merging, and the potential feature maps of these channels are merged as shared potential features. The size of the output merged feature map is 96*64*128, and the compression of the two adjacent frames of the video to be compressed with a size of 3*512*1024 is realized. Please refer to Figure 4 , the feature on the left Figure 1 (above) and features Figure 2 (Below) are schematic diagrams of two potential feature maps of size 64*64*128. Figure 1 and Features Figure 2 The merged figure on the right is a schematic diagram of a feature map of size 96*64*128. Each row in these three figures represents a channel.

[0068] It can be seen that the semantic information compression method of the embodiment of the present invention extracts shared potential features based on the correlation of feature maps or feature matrices, merges them from the dimension of the channels of the feature maps or feature matrices, and then realizes the compression of semantic information, which can improve the compression efficiency and has good compression performance. By transmitting the semantic information of the video image compressed by the semantic information compression method, the receiving end can restore the image and video without considering the index of each channel transmitted, thereby improving the stability of communication.

[0069] Figure 5 is a schematic diagram of a semantic information transmission method provided by an embodiment of the present invention. Figure 5 , a method for transmitting semantic information according to an embodiment of the present invention comprises the following steps:

[0070] Step S210: The sending end compresses the semantic information of each frame image in the video to be transmitted using the aforementioned semantic information compression method.

[0071] For example, using Figure 1 The semantic information compression method shown compresses each frame image of size 3*512*1024 in the video to be transmitted, and compresses each two adjacent frame images into multiple feature maps of size 96*64*128 through a potential feature extractor and a channel merging module in sequence.

[0072] Step S220: The source-channel joint encoder encodes the semantic information of each frame image in the compressed video to be transmitted to obtain a semantic code.

[0073] In an optional embodiment of the present invention, the source-channel joint encoder in step S220 includes multiple convolutional layers and multiple GDN activation functions, wherein a GDN activation function is connected between every two convolutional layers, the first convolutional layer serves as the input of the source-channel joint encoder, and the last convolutional layer serves as the output of the source-channel joint encoder, and the multiple convolutional layers are used to encode semantic information.

[0074] In an embodiment of the present invention, the source-channel joint encoder can improve the noise resistance of semantic information to cope with interference or noise in the transmission channel, and the size of the encoded feature map is the same as that before encoding (the number and size of image channels are not changed).

[0075] For example, see Figure 6 The source-channel joint encoder includes two convolutional layers and a GDN activation function. The GDN activation function is connected to the two convolutional layers, one of which is used as the input of the source-channel joint encoder and the other as the output of the source-channel joint encoder. The feature map with a size of 96*64*128 is input into the source-channel joint encoder for encoding, and the size of the feature map after output encoding is still 96*64*128. The source-channel joint encoder does not change the size of the feature map, which can improve the anti-noise ability.

[0076] Step S230: The channel transmits the compressed semantic coding of each frame image in the video to be transmitted.

[0077] In an optional embodiment of the present invention, in step S230, the step of transmitting the semantic coding of each frame image in the compressed video to be transmitted through the channel also includes: transmitting the index of the shared channel of the potential feature map of each frame image in the compressed video to be transmitted. That is, the two potential feature maps of each two adjacent frames of the video to be transmitted are merged into one image through the shared channel, thereby realizing the compression of the video semantic information, and the index of this shared channel can also be transmitted through the channel.

[0078] Figure 7 is a schematic diagram of a video image restoration method provided by an embodiment of the present invention. Figure 7 , a video image restoration method according to an embodiment of the present invention comprises the following steps:

[0079] Step S310, the receiving end receives the semantic coding of each frame image in the compressed video from the sending end, and the source channel joint decoder decodes the semantic coding to obtain semantic information of each frame image in the compressed video, wherein the semantic information includes the compressed shared channel and the respective non-shared channels of the two adjacent frame images before compression corresponding to each frame image, and the respective non-shared channels are respectively a first non-shared channel and a second non-shared channel.

[0080] In an embodiment of the present invention, a receiving end receives semantic coding of each frame image in a compressed video transmitted by a sending end through the aforementioned semantic information transmission method, wherein the compressed video is obtained through the aforementioned semantic information compression method, and each frame image in the compressed video is obtained by merging channels of latent feature maps of two adjacent frames of images before compression, wherein the first non-shared channel is a channel of the latent feature map of one of the two adjacent frames of images that has not been merged or compressed, and the second non-shared channel is a channel of the latent feature map of the other of the two adjacent frames of images that has not been merged or compressed.

[0081] In an optional embodiment of the present invention, the source-channel joint decoder in step S310 includes multiple convolutional layers and multiple GDN activation functions, wherein a GDN activation function is connected between every two convolutional layers, the first convolutional layer serves as the input of the source-channel joint decoder, and the last convolutional layer serves as the output of the source-channel joint decoder, and the multiple convolutional layers are used to decode semantic coding.

[0082] In the embodiment of the present invention, the source-channel joint decoder can further improve the noise resistance of semantic information, and the size of the decoded feature map is the same as the size of the encoded feature map (without changing the number and size of image channels).

[0083] For example, see Figure 6 The source-channel joint decoder includes two convolutional layers and a GDN activation function. The GDN activation function is connected to the two convolutional layers, one of which is used as the input of the source-channel joint decoder, and the other is used as the output of the source-channel joint decoder. The transmitted encoded feature map with a size of 96*64*128 is input into the source-channel joint decoder for decoding, and the size of the decoded feature map is still 96*64*128. The source-channel joint decoder does not change the size of the feature map, which can improve the noise resistance.

[0084] Step S320: A channel recovery module extracts the shared channel, the first non-shared channel, and the second non-shared channel from semantic information of each frame of the compressed video based on a preset compression ratio.

[0085] Step S330, the semantic information of the shared channel is combined with the semantic information corresponding to the first non-shared channel and the second non-shared channel respectively to restore the semantic information of each two adjacent frames of images before compression, thereby obtaining the semantic information of each frame of the image in the decompressed video.

[0086] Exemplarily, the decoded feature map of size 96*64*128 is input into the channel recovery module, and the preset compression ratio is 50%. The channel recovery module is used to extract the 32 shared channels of the two potential feature maps before compression and the remaining 32 channels of each of the two potential feature maps except the shared channels from the feature map. The remaining 32 channels except the shared channels and the potential feature map parts corresponding to the 32 shared channels respectively form two potential feature maps of size 64*64*128, thereby achieving decompression. For a schematic diagram of the two potential feature maps of two adjacent frames of 3*512*1024 images after decompression, please refer to Figure 8 The feature on the right Figure 1 (above) and features Figure 2 (below), these two latent feature maps are compared to the original latent feature map output by the latent feature extractor (see Figure 4 The feature on the left Figure 1 (above) and features Figure 2 (Schematic diagram of the bottom), the order of the channels changes.

[0087] Step S340: The latent feature restorer restores each frame of the video based on the semantic information of each frame of the decompressed video, thereby obtaining the transmitted video.

[0088] In an optional embodiment of the present invention, the potential feature restorer in step S340 includes multiple transposed convolutional layers, multiple GDN activation functions, convolutional layers and Tanh activation functions, wherein the number of transposed convolutional layers and GDN activation functions is the same, a GDN activation function is connected between every two transposed convolutional layers, and the output of the last GDN activation function is connected to the convolutional layer and the Tanh activation function in sequence. The first transposed convolutional layer serves as the input of the potential feature restorer, and the Tanh activation function serves as the output of the potential feature restorer. The multiple transposed convolutional layers are used to upsample the semantic information of each frame image in the decompressed video, reduce the number of channels of the image, and increase the size of the image of each channel.

[0089] For example, see Fig. 9The latent feature restorer includes three transposed convolutional layers, three GDN activation functions, a convolutional layer and a Tanh activation function. A GDN activation function is connected between every two transposed convolutional layers. The output of the third GDN activation function is connected to the convolutional layer and the Tanh activation function in turn. The first transposed convolutional layer is used as the input of the latent feature restorer, and the Tanh activation function is used as the output of the latent feature restorer. The two latent feature maps of size 64*64*128 with changed channel order after decompression are input into the first transposed convolutional layer of the latent feature restorer as the input. After upsampling through three transposed convolutional layers, two images of size 3*512*1024 are finally restored and output. The two images of size 3*512*1024 restored here are different from the two original images input to the latent feature extractor only in terms of image quality.

[0090] It can be seen that the semantic information transmission method of the embodiment of the present invention does not need to transmit the index of each channel in the latent feature map of the image (that is, the index of each semantic vector in the latent feature matrix), so that the video image restoration method of the embodiment of the present invention can restore the semantic information of each frame image of the transmitted video. The semantic information of each frame image of the restored video is the same as the original image of each frame of the original video, which can ensure the accuracy, precision and stability of the semantic information of each frame image of the restored video. At the same time, it can avoid the problem of transmitted index error caused by the need to transmit the index of each semantic vector in the latent feature matrix, and the security, reliability and stability of information transmission are high.

[0091] The existing technology calculates the entropy of the feature vector from the perspective of information entropy, and judges the importance of the vector elements according to the size of the entropy. Information with low entropy is not transmitted during the communication transmission process, and the transmitted data is compressed by transmitting the index of the vector elements. Fig.11 Schematic diagram comparing the video image restoration method of the embodiment of the present invention and the effect of restoring the transmitted image in the prior art. Fig.11 The two images in the first row are original images, the two images in the second row are transmission images restored by the video image restoration method of an embodiment of the present invention, and the two images in the third row are transmission images restored by the prior art. It can be seen that in the transmission process of the prior art, index errors lead to aliasing of the transmission image, while the semantic information transmission method of the embodiment of the present invention does not need to transmit the index of each channel, which can improve the stability of communication transmission and the restoration quality of the video image by the video image restoration method. Fig.12 Schematic diagram of comparison between the semantic information transmission method of the embodiment of the present invention and the index amount transmitted in the prior art. Fig.12The PC-HEM in the figure represents the index amount transmitted by the semantic information transmission method of the embodiment of the present invention, and the ED-HEM represents the index amount transmitted by the prior art for images of different sizes. It can be seen that compared with the prior art, the index transmission amount of the semantic information transmission method of the embodiment of the present invention is 0, which saves resources, greatly reduces the transmission amount, and improves the transmission efficiency.

[0092] In an optional embodiment of the present invention, when the above-mentioned semantic information transmission method transmits the index of the shared channel through the channel, the index is received by the receiving end. The channel recovery module in the video image restoration method of the embodiment of the present invention can also restore the semantic information of each frame image before merging the shared channel using the above-mentioned semantic information compression method according to the index of the shared channel, that is, the original latent feature map obtained by the latent feature extractor.

[0093] It can be seen that the semantic information transmission method of the embodiment of the present invention transmits the index of the shared channel of each adjacent two frames of images and the semantic coding of each frame of images in the compressed video to be transmitted to the receiving end, so that the video image restoration method of the embodiment of the present invention uses the channel recovery module to perform channel recovery on the semantic information of each frame of images in the compressed video according to the index of the shared channel to obtain the potential feature map of each frame of images in the transmitted original video, and further enables the potential feature restorer to restore the original images of each frame of the original video according to the original potential feature map restored by the channel recovery module, thereby restoring the original video.

[0094] Corresponding to the above method, an electronic device of an embodiment of the present invention includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the electronic device implements the steps of the above-mentioned semantic information compression method, semantic information transmission method and video image restoration method.

[0095] In the embodiments of the present invention, see Fig.10 The above-mentioned semantic information compression, transmission and video image restoration process is realized by using a latent feature extractor, a correlation calculation module (not shown in the figure), a channel merging module, a source channel joint encoder, a channel, a source channel joint decoder, a channel recovery module and a latent feature restorer. Among them, the latent feature extractor can be an autoencoder or a transformer encoder, and the latent feature restorer can be an autoencoder or a transformer decoder.

[0096] The embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the aforementioned semantic information compression method, semantic information transmission method and video image restoration method are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0097] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0098] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0099] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0100] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A semantic information compression method, characterized in that: include: A latent feature extractor extracts latent features of each frame image from the compressed video, and generates a plurality of latent feature maps accordingly. The latent feature maps are used as semantic information. Each latent feature map includes a plurality of channels, and the plurality of channels of different latent feature maps correspond to each other one by one. Calculating correlations between multiple corresponding channels of potential feature maps of each of the two adjacent frames of images based on semantic information of each of the two adjacent frames of images, and obtaining multiple correlation values; The values ​​of multiple correlations are sorted from large to small, and a channel merging module selects multiple corresponding channels of the potential feature map of each two adjacent frames of images corresponding to the correlations sorted first based on a preset compression ratio to merge to form a shared channel. The semantic information corresponding to the shared channel and the respective non-shared channels constitutes the semantic information of each two adjacent frames of images after compression, thereby achieving compression of the semantic information in the video to be compressed.

2. The method according to claim 1, characterized in that The step of calculating the correlation between multiple corresponding channels of the potential feature map of each two adjacent frames of images based on the semantic information of each two adjacent frames of images includes: The latent feature maps of multiple corresponding channels in the latent feature map of each of two adjacent frames of images are converted into multiple pairs of semantic vectors, wherein the two semantic vectors of each corresponding channel form a pair of semantic vectors; The Pearson correlation coefficient between each pair of semantic vectors is calculated to obtain the correlation between multiple corresponding channels of the potential feature map of each two adjacent frames of images.

3. The method according to claim 1, characterized in that The feature extractor includes multiple convolutional layers and multiple GDN activation functions, wherein a GDN activation function is connected between every two convolutional layers, the first convolutional layer serves as the input of the feature extractor, and the last convolutional layer serves as the output of the feature extractor. The multiple convolutional layers are used to downsample each frame image of the video to be compressed, and correspondingly generate multiple potential feature maps, increase the number of channels of the image, and reduce the size of the potential feature map of each channel.

4. A semantic information transmission method, characterized in that: include: The sending end compresses the semantic information of each frame image in the video to be transmitted using the method as described in any one of claims 1 to 3; The source-channel joint encoder encodes the semantic information of each frame image in the compressed video to be transmitted to obtain a semantic code; The channel transmits the semantic coding of each frame image in the compressed video to be transmitted.

5. The method according to claim 4, characterized in that The source-channel joint encoder includes multiple convolutional layers and multiple GDN activation functions, wherein a GDN activation function is connected between every two convolutional layers, the first convolutional layer serves as the input of the source-channel joint encoder, and the last convolutional layer serves as the output of the source-channel joint encoder, and the multiple convolutional layers are used to encode semantic information.

6. The method according to claim 4, characterized in that The step of transmitting the semantic coding of each frame image in the compressed video to be transmitted through the channel also includes: transmitting the index of the shared channel of the potential feature map of each frame image in the compressed video to be transmitted.

7. A video image restoration method, characterized in that: include: The receiving end receives the semantic coding of each frame image in the compressed video from the sending end, and the source channel joint decoder decodes the semantic coding to obtain semantic information of each frame image in the compressed video, wherein the semantic information includes the compressed shared channel and respective non-shared channels of two adjacent frame images before compression corresponding to each frame image, and the respective non-shared channels are respectively a first non-shared channel and a second non-shared channel; The channel recovery module extracts the shared channel, the first non-shared channel and the second non-shared channel from the semantic information of each frame of the compressed video based on a preset compression ratio; The semantic information of the shared channel is combined with the semantic information corresponding to the first non-shared channel and the second non-shared channel respectively to restore the semantic information of each two adjacent frames of images before compression, thereby obtaining the semantic information of each frame of the image in the decompressed video; The latent feature restorer restores each frame of the video based on the semantic information of each frame in the decompressed video, thereby obtaining the transmitted video.

8. The method according to claim 7, characterized in that The source-channel joint decoder comprises a plurality of convolutional layers and a plurality of GDN activation functions, wherein a GDN activation function is connected between every two convolutional layers, the first convolutional layer is used as the input of the source-channel joint decoder, and the last convolutional layer is used as the output of the source-channel joint decoder, and the plurality of convolutional layers of the source-channel joint decoder are used to decode the semantic coding; The potential feature restorer includes multiple transposed convolutional layers, multiple GDN activation functions, convolutional layers and Tanh activation functions, wherein the number of the transposed convolutional layers and the GDN activation functions is the same, a GDN activation function is connected between every two transposed convolutional layers, the output of the last GDN activation function is sequentially connected to the convolutional layer and the Tanh activation function, the first transposed convolutional layer serves as the input of the potential feature restorer, and the Tanh activation function serves as the output of the potential feature restorer, and the multiple transposed convolutional layers are used to upsample the semantic information of each frame image in the decompressed video, reduce the number of channels of the image, and increase the size of the image of each channel.

9. An electronic device comprising a processor and a memory, characterized in that: The memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the electronic device implements the steps of the method as claimed in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images