A restricted communication image transmission method and device based on image mask reconstruction

Abstract

The application provides a limited communication image transmission method and device based on image mask reconstruction, wherein the method applied to an image sending end and an image receiving end comprises: the image sending end processes an original image to be sent to obtain frame data, a mean matrix, a standard deviation matrix and a mask matrix; the image sending end packs a timestamp, the frame data, the statistical quantity matrix and the mask matrix into a protocol data unit, and sends the protocol data unit to the image receiving end; and the image receiving end processes the received protocol data unit to obtain a reconstructed image. The application can significantly reduce bandwidth demand by reducing the amount of transmission data, adapt to low bandwidth and unstable network environment, efficiently reconstruct a missing image part, maintain high image quality, reduce transmission delay and save bandwidth cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image transmission, in particular to a restricted communication image transmission method and device based on image mask reconstruction. BACKGROUND

[0002] In the field of image and video transmission, traditional image and video compression technology mainly realizes efficient compression by reducing redundant information and optimizing encoding strategy, and these technologies generally have no obvious problems under good network environment conditions. With the increasing number of wireless devices and the increasing scarcity of spectrum resources, when long-range transmission or high concurrent number of intra-regional communication is performed, it is difficult to guarantee that all connections have low latency and high bandwidth transmission conditions. The traditional image and video compression still mainly takes fidelity as the primary optimization index, and has significant limitations under extremely low bandwidth conditions. When an extremely low bit rate optimization goal is specified, a large amount of semantic loss (such as color change, object contour, texture detail, etc.) often occurs, which greatly reduces the understanding and recognition ability of the transmitted image content.

[0003] Existing compression algorithms mainly fall into two categories: one is a compression method of manually designing a feature and rule set, and the other is a compression method based on deep learning. The former mainly represented by H.264, H.265, etc. mainly idea is based on block-oriented motion compensation, which divides the single frame content into different size macroblocks, compresses the blocks, and then removes the redundant information of the video in time and space based on the inter-frame relationship. This method has slightly lower computational complexity and has hardware encoder support, and can be efficiently encoded and decoded under lower power consumption and performance requirements, but the compression rate is still limited, and it is not suitable for specific texture and motion scenes, and block effect and blur are more likely to occur at low code rate. The latter can automatically learn features through neural networks and can optimize specific scenes, and has strong adaptability. For example, ROI-DVC inputs the region of interest (ROI) mask into the encoder, so that the model focuses on the area important to the user; considering the transmission characteristics, the VQ-DeepVSC framework extracts key frames and difference modules through self-adaptation, combines semantic vector quantization codec and multi-channel spatial condition vector quantization method, and can significantly improve the multi-scale structural similarity (MS-SSIM) and learning perceptual image block similarity (LPIPS) compared with H.265. However, the compression method based on deep learning also has the characteristics of large amount of calculation and high deployment difficulty, and generally needs to rely on specific hardware modules such as npu to run the compression algorithm on the sending end, which has a high requirement on device performance and a low efficiency ratio. SUMMARY

[0004] Therefore, the application provides a limited communication image transmission method and device based on image mask reconstruction to solve the above technical problems.

[0005] In a first aspect, the application provides a limited communication image transmission method based on image mask reconstruction, applied to an image sending end and an image receiving end, comprising:

[0006] The image sending end processes the original image to be sent to obtain frame data, a mean matrix, a standard deviation matrix, and a mask matrix;

[0007] The image sending end packs the timestamp, frame data, statistical quantity matrix, and mask matrix into a protocol data unit, and sends the protocol data unit to the image receiving end;

[0008] The image receiving end processes the received protocol data unit to obtain a reconstructed image.

[0009] In a possible implementation, the image sending end processes the original image to be sent to obtain frame data, a mean matrix, a standard deviation matrix, and a mask matrix, comprising:

[0010] Downsample the original image I1 to obtain an image I2 with a size of 224x224;

[0011] Divide the image I2 into 14x14 tiles with a size of 16x16 pixels, and each tile is represented as where i and j represent the row and column indices of the tile in the image, respectively, 1≤i≤14, and 1≤j≤14;

[0012] Calculate the mean vector of each tile, and generate a mean matrix P mean from the mean vectors of all tiles.

[0013] Calculate the standard deviation vector of each tile, and generate a standard deviation matrix P std from the standard deviation vectors of all tiles.

[0014] Mask the tiles according to the set masking rate, generate a mask matrix and an unmasked image according to the set masking rate.

[0015] Generate frame data based on the unmasked image.

[0016] In a possible implementation, calculating the mean vector of each tile comprises:

[0017] The mean vector of the tile is:

[0018] ​

[0019] wherein, is the mean of the R channel:

[0020]

[0021] wherein, is the tile the R channel value at pixel (x, y);

[0022] is the mean of the G channel:

[0023]

[0024] wherein, is the tile the G channel value at pixel (x, y);

[0025] is the mean of the B channel:

[0026]

[0027] wherein, is the tile the B channel value at pixel (x, y).

[0028] In one possible implementation, a standard deviation vector of each tile is calculated, including:

[0029] tile standard deviation vector is:

[0030]

[0031] wherein, is the standard deviation of the R channel:

[0032]

[0033] is the standard deviation of the G channel:

[0034]

[0035] is the standard deviation of the B channel:

[0036]

[0037] In one possible implementation, the tiles are masked according to a set masking rate, generating a mask matrix and an unmasked image, including:

[0038] According to the predetermined masking rate, the number N of tiles that need to be retained is calculated:

[0039] N = round((1 - mask ratio ) x 14 x 14)

[0040] wherein, mask_ratio is the mask rate; round(·) is the floor function;

[0041] Based on the number of tiles N, the reserved tile position is determined according to the pre-prepared strategy, and a mask matrix M with a size of 14x14 is generated, each element of which is 1 or 0, wherein 1 indicates that the tile at the corresponding position is reserved, and 0 indicates that the tile at the corresponding position is masked;

[0042] The reserved tiles are spliced into an unmasked image I umasked in horizontal order, wherein the height is 16 and the width is 16xN.

[0043] In one possible implementation, based on the unmasked image, frame data is generated, including:

[0044] The unmasked image I umasked is binary encoded by a standard encoder to obtain frame data in binary form.

[0045] In one possible implementation, the image receiving end processes the received protocol data unit to obtain a reconstructed image, including:

[0046] The received protocol data unit is unpacked to extract the timestamp, frame data, mean matrix P mean , standard deviation matrix P std and mask matrix M;

[0047] The frame data is decoded by a standard decoder to obtain the unmasked image I umasked ;

[0048] Based on the mask matrix M and the unmasked image I umasked , a partially masked image I partial with a size of 224x224 is generated; wherein the unmasked part of the image I partial is the image I umasked , and the masked part of the image I partial is empty;

[0049] The image I partial is processed by a pre-trained masking autoencoder to obtain a first reconstructed image I reconstructed , and the masked part of the first reconstructed image I reconstructed is the reconstructed image I masked , and the unmasked part remains unchanged;

[0050] The first reconstructed image Ireconstructed Multiply by the standard deviation matrix P std Then, it is compared with the mean matrix P. mean By adding the two images together, we obtain the second reconstructed image I with normal colors. reconstructed ;

[0051] The second reconstructed image I with normal colors reconstructed Upsampling is performed to obtain a third reconstructed image I with the same resolution as the original image. final .

[0052] In one possible implementation, the method further includes:

[0053] Image I is ordered according to timestamps. final Arrange them sequentially to generate a complete video.

[0054] Secondly, embodiments of this application provide a restricted communication image transmission device based on image mask reconstruction, comprising: an image transmitting end and an image receiving end.

[0055] The image transmitting end is used to process the original image to be transmitted to obtain frame data, mean matrix, standard deviation matrix and mask matrix; package the timestamp, frame data, statistical matrix and mask matrix into protocol data unit and send the protocol data unit to the image receiving end;

[0056] The image receiving end is used to process the received protocol data units to obtain the reconstructed image.

[0057] This application significantly reduces bandwidth requirements by reducing the amount of data transmitted, making it suitable for low-bandwidth and unstable network environments; it can efficiently reconstruct missing image parts, maintain high image quality, reduce transmission latency, and save bandwidth costs. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0059] Figure 1 A flowchart of a restricted communication image transmission method based on image mask reconstruction provided in an embodiment of this application;

[0060] Figure 2 A functional structure diagram of a restricted communication image transmission device based on image mask reconstruction provided in an embodiment of this application. Detailed Implementation

[0061] To make the purposes, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0062] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without making creative efforts are within the scope of protection of the present application.

[0063] The technical solutions provided by the embodiments of the present application will be described below.

[0064] As shown in Figure 1 The embodiments of the present application provide a restricted communication image transmission method based on image mask reconstruction, applied to an image sending end and an image receiving end, comprising:

[0065] Step 101: The image sending end processes the original image to be sent to obtain frame data, a mean matrix, a standard deviation matrix, and a mask matrix;

[0066] Step 102: The image sending end packs the timestamp, frame data, statistical quantity matrix, and mask matrix into a protocol data unit, and sends the protocol data unit to the image receiving end;

[0067] Step 103: The image receiving end processes the received protocol data unit to obtain a reconstructed image.

[0068] The embodiments of the present application mask and reassemble the original video frame at the image sending end, and after being encoded and packaged by a standard encoder, the video frame is transmitted to the image receiving end through a protocol. The image receiving end decodes and sends the masked part to a masked autoencoder (Masked Autoencoder, MAE) for reconstruction, and finally forms a visible video stream through post-processing and upsampling. The embodiments of the present application can solve the problems of video data transmission feasibility under extremely low bandwidth (5-100 kbps) and high semantic loss in traditional video compression methods.

[0069] The method of the embodiments of the present application significantly reduces bandwidth demand by reducing the amount of transmitted data, and is suitable for low-bandwidth and unstable network environments. It can efficiently reconstruct missing image parts, maintain high image quality, reduce transmission delay, and save bandwidth costs. In addition, the method has good adaptability and robustness, and is suitable for various application scenarios.

[0070] In some embodiments, the image sending end processes the original image to be sent to obtain frame data, a mean matrix, a standard deviation matrix and a mask matrix, including:

[0071] Down-sampling the original image I1 to obtain an image I2 with a size of 224x224;

[0072] Dividing the image I2 into 14x14 tiles each with a size of 16x16 pixels, each tile being denoted as where i and j represent the row and column indices of the tile in the image respectively, 1≤i≤14, 1≤j≤14;

[0073] Calculating a mean vector of each tile, and generating a mean matrix P from the mean vectors of all tiles mean ;

[0074] Calculating a standard deviation vector of each tile, and generating a standard deviation matrix P from the standard deviation vectors of all tiles std ;

[0075] Masking the tiles according to a set masking rate, and generating a mask matrix and an unmasked image according to the set masking rate;

[0076] Generating frame data based on the unmasked image.

[0077] In some embodiments, the mean vector of each tile includes:

[0078] The mean vector of the tile is:

[0079]

[0080] where is the mean of the R channel:

[0081]

[0082] where is the value of the R channel of the tile at pixel (x, y);

[0083] is the mean of the G channel:

[0084]

[0085] where is the value of the G channel of the tile at pixel (x, y);

[0086] is the mean of the B channel:​

[0087]

[0088] wherein, is a tile B channel value at pixel (x, y).

[0089] In some embodiments, a standard deviation vector of each tile is calculated, including:

[0090] tile standard deviation vector of is:

[0091]

[0092] wherein, is the standard deviation of R channel:

[0093]

[0094] is the standard deviation of G channel:

[0095]

[0096] is the standard deviation of B channel:

[0097]

[0098] In some embodiments, the tiles are masked according to a set masking rate, generating a mask matrix and an unmasked image, including:

[0099] According to a predetermined masking rate, the number of tiles to be retained N is calculated:

[0100] N = round((1-mask ratio ) x 14 x 14)

[0101] wherein, mask_ratio is the masking rate; round(·) is the floor function;

[0102] Based on the number of tiles N, the retained tile positions are determined according to a pre-prepared strategy, generating a mask matrix M of size 14 x 14, each element of which is 1 or 0, wherein 1 indicates that the tile at the corresponding position is retained, and 0 indicates that the tile at the corresponding position is masked;

[0103] The retained tiles are spliced into an unmasked image I umasked , wherein the height is 16 and the width is 16 x N.

[0104] In some embodiments, based on the unmasked image, frame data is generated, including:

[0105] The unmasked image I umasked is encoded by a standard encoder to obtain frame data in binary form.

[0106] In some embodiments, the received protocol data unit is unpacked to extract the timestamp, frame data, mean matrix P mean , standard deviation matrix P std and mask matrix M;

[0107] The frame data is decoded by a standard decoder to obtain the unmasked image I umasked .

[0108] Based on the mask matrix M and the unmasked image I umasked , a partially masked image I partial of size 224x224 is generated; wherein the unmasked part of the image I partial is the image I umasked , and the masked part of the image I partial is empty.

[0109] The image I partial is processed by a pre-trained masking autoencoder to obtain a first reconstructed image I reconstructed , and the masked part of the first reconstructed image I reconstructed is the reconstructed image I masked , while the unmasked part remains unchanged.

[0110] The first reconstructed image I reconstructed is multiplied by the standard deviation matrix P std , and then added to the mean matrix P mean to obtain a second reconstructed image I reconstructed of normal color.

[0111] The second reconstructed image I reconstructed of normal color is upsampled to obtain a third reconstructed image I final of the same resolution as the original image.

[0112] In addition, Gaussian blur needs to be applied to the completed tile edge part of the second reconstructed image I reconstructed to smooth the transition between the completed area and the original area; the kernel size and standard deviation of the Gaussian blur are adjusted according to the actual effect.

[0113] In some embodiments, the method further comprises:

[0114] The images I final are arranged in order of the timestamp to generate a complete video.

[0115] Based on the above embodiment, the application provides a limited communication image transmission device based on image mask reconstruction, referring to Figure 2 As shown in the figure, the application provides a limited communication image transmission device 200 based on image mask reconstruction, which at least comprises an image sending end and an image receiving end, wherein,

[0116] The image sending end is configured to process an original image to be sent to obtain frame data, a mean matrix, a standard deviation matrix and a mask matrix; package a timestamp, the frame data, the statistical quantity matrix and the mask matrix into a protocol data unit, and send the protocol data unit to the image receiving end.

[0117] The image receiving end is configured to process the received protocol data unit to obtain a reconstructed image.

[0118] It should be noted that the principle of solving the technical problem of the limited communication image transmission device 200 based on image mask reconstruction provided by the application is similar to the method provided by the application, therefore, the implementation of the limited communication image transmission device 200 based on image mask reconstruction provided by the application can refer to the implementation of the method provided by the application, and the repeated parts will not be described here.

[0119] It should be noted that although several units or sub-units of the device are mentioned in the above detailed description, such division is only exemplary and not mandatory. In fact, according to the embodiments of the application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units for embodiment.

[0120] In addition, although the operations of the method of the application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in that specific order, or that all of the shown operations must be performed to achieve the desired result. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be divided into multiple steps.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the application and not to limit it. Although the application has been described in detail with reference to the embodiments, those skilled in the art should understand that the technical solutions of the application can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the application, and they should be covered in the scope of the claims of the application.

Claims

1. A constrained communication image transmission method based on image mask reconstruction, applied to an image sending end and an image receiving end, characterized in that, include: The image transmitting end processes the original image to be transmitted to obtain frame data, mean matrix, standard deviation matrix and mask matrix; The image transmitting end packages the timestamp, frame data, statistical matrix, and mask matrix into a protocol data unit, and sends the protocol data unit to the image receiving end; The image receiving end processes the received protocol data units to obtain the reconstructed image; The image transmitter processes the raw image to be transmitted, obtaining frame data, a mean matrix, a standard deviation matrix, and a mask matrix, including: For the original image Perform downsampling to obtain a size of Image ; Image Divided into The size is A pixel tile, each tile is represented as ,in and These represent the row and column indices of the tile in the image, respectively. , ; Calculate the mean vector for each tile, and generate a mean matrix based on the mean vectors of all tiles. ; Calculate the standard deviation vector for each tile, and generate a standard deviation matrix based on the standard deviation vectors of all tiles. ; The tiles are masked according to the set masking rate, and a mask matrix and an unmasked image are generated. Generate frame data based on the unmasked image; The tiles are masked according to a set masking rate, generating a mask matrix and an unmasked image, including: Calculate the number of tiles that need to be retained based on the predetermined concealment rate. : in, For concealment rate; To round down; Based on the number of tiles The positions of the tiles to be retained are determined according to a pre-defined strategy, generating a size of... mask matrix Each element is either 1 or 0, where 1 indicates that the tile at the corresponding position is retained. This indicates that the tile at the corresponding location is covered; Arrange the remaining tiles horizontally to form an unmasked image. The height is 16 and the width is ; The image receiving end processes the received protocol data units to obtain a reconstructed image, including: The received protocol data units are unpacked to extract the timestamp, frame data, and mean matrix. Standard deviation matrix and mask matrix ; The frame data is decoded using a standard decoder to obtain the unmasked image. ; Based on mask matrix and unmasked images Generate a partial mask of size . Image Among them, images The unmasked portion is the image. ,image The concealed portion is empty; Using a pre-trained masking autoencoder to process images The image is processed to obtain the first reconstructed image. First reconstructed image The masked part is the reconstructed image The unmasked parts remain unchanged; The first reconstructed image Multiply by the standard deviation matrix Then, it is compared with the mean matrix. By adding the two images together, a second reconstructed image with normal colors is obtained. ; Second reconstructed image with normal colors Upsampling is performed to obtain a third reconstructed image with the same resolution as the original image. .

2. The method according to claim 1, characterized in that, Calculate the mean vector for each tile, including: roof tiles mean vector for: in, The mean of the R channel: in, For tiles In pixels The R channel value at that location; The mean of the G channel: in, For tiles In pixels The G channel value at that location; The mean of channel B: in, For tiles In pixels The B channel value at that location.

3. The method according to claim 2, characterized in that, Calculate the standard deviation vector for each tile, including: roof tiles standard deviation vector for: in, The standard deviation of the R channel: The standard deviation of the G channel: The standard deviation of channel B: 。 4. The method according to claim 1, characterized in that, Based on the unmasked image, generate frame data, including: Unmasked images are processed using a standard encoder. Binary encoding is performed to obtain frame data in binary form.

5. The method according to claim 1, characterized in that, The method further includes: Images in order of timestamp Arrange them sequentially to generate a complete video.

6. A constrained communication image transmission device based on image mask reconstruction, characterized in that, include: Image transmitter and image receiver, The image transmitting end is used to process the original image to be transmitted to obtain frame data, mean matrix, standard deviation matrix and mask matrix; The timestamp, frame data, statistical matrix, and mask matrix are packaged into a protocol data unit, and the protocol data unit is sent to the image receiving end. The image receiving end is used to process the received protocol data units to obtain a reconstructed image; The raw image to be sent is processed to obtain frame data, mean matrix, standard deviation matrix, and mask matrix, including: For the original image Perform downsampling to obtain a size of Image ; Image Divided into The size is A pixel tile, each tile is represented as ,in and These represent the row and column indices of the tile in the image, respectively. , ; Calculate the mean vector for each tile, and generate a mean matrix based on the mean vectors of all tiles. ; Calculate the standard deviation vector for each tile, and generate a standard deviation matrix based on the standard deviation vectors of all tiles. ; The tiles are masked according to the set masking rate, and a mask matrix and an unmasked image are generated. Generate frame data based on the unmasked image; The tiles are masked according to a set masking rate, generating a mask matrix and an unmasked image, including: Calculate the number of tiles that need to be retained based on the predetermined concealment rate. : in, For concealment rate; To round down; Based on the number of tiles The positions of the tiles to be retained are determined according to a pre-defined strategy, generating a size of... mask matrix Each element is either 1 or 0, where 1 indicates that the tile at the corresponding position is retained. This indicates that the tile at the corresponding location is covered; Arrange the remaining tiles horizontally to form an unmasked image. The height is 16 and the width is ; The received protocol data units are processed to obtain a reconstructed image, including: The received protocol data units are unpacked to extract the timestamp, frame data, and mean matrix. Standard deviation matrix and mask matrix ; The frame data is decoded using a standard decoder to obtain the unmasked image. ; Based on mask matrix and unmasked images Generate a partial mask of size . Image Among them, images The unmasked portion is the image. ,image The concealed portion is empty; Using a pre-trained masking autoencoder to process images The image is processed to obtain the first reconstructed image. First reconstructed image The masked part is the reconstructed image The unmasked parts remain unchanged; The first reconstructed image Multiply by the standard deviation matrix Then, it is compared with the mean matrix. By adding the two images together, a second reconstructed image with normal colors is obtained. ; Second reconstructed image with normal colors Upsampling is performed to obtain a third reconstructed image with the same resolution as the original image. .

Images