Video transmission device and method

Abstract

The present invention provides a video transmission device, the device comprising: a preprocessing module, which is configured at a transmitting end, and is used to decompose the original target image in the video to be transmitted into a foreground factor, a background factor and a posture factor; a coding system, which is configured at the transmitting end, and is used to encode the original image based on the foreground factor, the background factor and the posture factor to obtain the foreground information embedding feature, the background information embedding feature and the posture information embedding feature and transmit them to a decoding system; a decoding system, which is configured at a receiving end, and is used to decode the received embedded features to generate a reconstructed image. The present invention deconstructs the image data into multiple factors at the transmitting end and encodes the input image into an embedded feature based on each decomposition factor, and then reconstructs the image based on the embedded feature, and then uses a deep convolution adversarial generation network to generate a more realistic image. Under the premise of reducing the transmission bandwidth and storage resources, the quality of the generated image is improved and the transmission quality is guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of image generation, more specifically, to the field of feature extraction and image generation technology in a video transmission process, and more specifically, to a video transmission device and method with low bandwidth occupancy and high-quality image generation. Background Art

[0002] In the IT3.0 era of human-machine-object interconnection, there will be tens of billions of intelligent terminals and hundreds of billions of physical devices, which will be integrated into many application fields such as education, transportation, industry, agriculture, production, and entertainment. As people's demand for communication continues to increase, video communication has gradually replaced voice as the basic carrier of communication content, and a large number of devices will communicate based on video.

[0003] Compared with other methods, video communication brings greater transmission and storage pressure. Under existing technologies, the main method to alleviate transmission and storage pressure is video compression codec technology. Video coding technology reduces redundancy in the video through methods such as quantization, discrete cosine transform, and motion estimation, thereby achieving data compression. However, with the continuous increase in data volume, video compression codec technology is gradually approaching its performance limit. Every increase in compression efficiency by 1 times often results in more than 5 times the computing overhead. The speed of improving the compression efficiency of video codec technology is far behind the growth rate of video data. Therefore, in the face of the rapid growth of video big data, transmission, storage, and analysis and processing are facing severe challenges.

[0004] In recent years, a new band-strip transmission method has been proposed to cope with the video coding and decoding technology that has reached its limit. Video is composed of several images. The essence of video transmission is image transmission. By deconstructing the image data into semantic information at the sending end, the key information can be transmitted in text form. At the receiving end, the text semantic information is reconstructed into a reconstructed image that is consistent with the original target image semantics based on image generation technology, such as Figure 1 As shown, the transmitting end collects and processes video data to realize image acquisition to obtain the original image, extracts the foreground, human posture, and text key point coordinate information from the original image to obtain text information, and then sends it to the receiving end. The receiving end generates a background image and a human torso image based on the text semantic information, completes image generation based on the human torso to generate a pedestrian image, and fuses the background image and the pedestrian image to obtain a reconstructed image. This method assigns different forms of computing power to each part of the communication network structure, specifically by assigning the semantic computing power of video data features to the transmitting end and the computing power of semantically generated images to the receiving end, converting the original local centralized computing communication mode into a collaborative diversity computing communication mode, thereby greatly reducing bandwidth and storage resources and improving transmission, storage and retrieval efficiency.

[0005] Although the band-strip transmission method has overturned the traditional video transmission technology, it still has many problems. Specifically, because the image reconstruction part of this method uses generative technology instead of restoration technology, the generated image quality is poor. Compared with the original image, there are problems such as low clarity and blur. The reason for this problem is that the image generation technology uses the conditional generative adversarial network (CGAN), which only considers whether the image is generated or not, but does not consider the quality of the generated image. Summary of the invention

[0006] Therefore, the purpose of the present invention is to overcome the defects of the prior art and provide a video transmission device and method that can generate high-quality images and significantly reduce bandwidth resource usage.

[0007] According to a first aspect of the present invention, a video transmission device is provided, the device comprising: a preprocessing module, which is configured at a transmitting end, and is used to decompose an original target image in a video to be transmitted so as to decompose it into a foreground factor, a background factor and a posture factor; an encoding system, which is configured at a transmitting end, and is used to encode the original image based on the foreground factor, the background factor and the posture factor to obtain a foreground information embedding feature, a background information embedding feature and a posture information embedding feature and transmit them to a decoding system; a decoding system, which is configured at a receiving end, and is used to decode the received embedded features to generate a reconstructed image.

[0008] Preferably, the encoding system comprises: a posture encoder for encoding the original target image based on the posture factor to obtain the posture information embedding feature; a background encoder for encoding the original target image based on the foreground factor to obtain the foreground information embedding feature; a foreground encoder for encoding the original target image based on the background factor to obtain the background information embedding feature

[0009] In some embodiments of the present invention, the posture encoder includes: a human posture model, which is used to extract the key points of the human skeleton to obtain the corresponding key point coordinate information and key point visibility information; a human posture automatic encoding model, which is used to encode and reconstruct the posture information based on the key point coordinate information and the key point visibility information and decode the embedded posture features based on the reconstructed posture information. Preferably, the human posture automatic encoding model includes 4 residual blocks that are fully connected in sequence, a bottleneck layer, and 4 residual blocks that are fully connected in sequence, wherein each residual block includes two fully connected layers.

[0010] In some embodiments of the present invention, the foreground encoder includes: a foreground feature extraction module, which is used to obtain a rough posture mask based on the human skeleton key point information extracted from the human posture model by using morphology and perform feature mapping based on the mask to obtain foreground features containing posture information from the original target image; multiple human ROI models, which are used to encode posture invariant features for the foreground features containing posture information respectively; a foreground coding model, which is used to encode the foreground features based on the posture invariant features to obtain foreground information embedded features, wherein the foreground coding model includes multiple sequentially connected convolution residual blocks. Preferably, the foreground encoder includes 7 human ROI models, and the foreground coding model includes 5 sequentially connected convolution residual blocks.

[0011] In some embodiments of the present invention, the background encoder includes: a background feature extraction module, which is used to obtain background features by using an inverse posture mask based on a coarse posture mask obtained by a foreground feature extraction module; and a background coding model, which is used to obtain background information embedding features by encoding based on the background features, wherein the background coding model includes a plurality of sequentially connected convolution residual blocks. Preferably, the background coding model includes 5 sequentially connected convolution residual blocks.

[0012] Preferably, the decoding system includes: an appearance feature processing module, used for connecting and tiling foreground information embedding features and background information embedding features to obtain appearance features; and a U-net network, used for generating a reconstructed image based on appearance features and posture information embedding features.

[0013] Preferably, the device also includes a deep convolutional generative adversarial network, which also includes a generator for generating a difference map with the original image based on the reconstructed image generated by the decoding system, and connecting the difference map with the reconstructed image generated by decoding to obtain the final reconstructed image, wherein the generator of the generative adversarial network is obtained by training the deep convolutional generative adversarial network based on a training set of original images and reconstructed images of the original images.

[0014] According to a second aspect of the present invention, there is provided a video transmission method based on the video transmission device according to the first aspect of the present invention, wherein a transmitting end transmits an original target image to a receiving end, and the method comprises: S1, decomposing the original target image to be transmitted at the transmitting end so as to decompose it into a foreground factor, a background factor and a posture factor; S2, encoding the original image based on the foreground factor, background factor and posture factor distribution decomposed in step S1 to obtain embedded features of foreground information, background information and posture information and transmitting the embedded features to a decoding system; S3, generating a reconstructed image based on the embedded features at the receiving end.

[0015] Preferably, step S2 includes: S21, encoding the original target image based on the posture factor to obtain the posture information embedded feature; wherein, the posture information embedded feature is obtained in the following manner: extracting the key points of the human skeleton to obtain the corresponding key point coordinate information and the key point visibility information and encoding and reconstructing the posture information based on the key point coordinate information and the key point visibility information, and decoding the embedded posture feature based on the reconstructed posture information; S22, encoding the original target image based on the foreground factor to obtain the foreground information embedded feature, wherein the foreground information embedded feature is obtained in the following manner: using morphology to obtain a coarse posture mask based on the human skeleton key point information, performing feature mapping based on the mask to obtain a foreground feature containing the posture information from the original target image, encoding the foreground feature containing the posture information into a posture invariant feature, and encoding the foreground feature based on the posture invariant feature to obtain the foreground information embedded feature; S23, encoding the original target image based on the background factor to obtain the background information embedded feature, wherein the background information embedded feature is obtained in the following manner: using an inverse posture mask based on the coarse posture mask to obtain the background feature and encoding based on the background feature to obtain the background information embedded feature.

[0016] In some embodiments of the present invention, the method also includes: S4, using a generator in a pre-trained deep convolutional generative adversarial network to generate a difference map between the reconstructed image and the original target image based on the reconstructed image generated in step S3, and connecting the difference map with the reconstructed image generated in step S3 to obtain the final reconstructed image.

[0017] Preferably, the deep convolutional generative adversarial network is pre-trained in the following manner: T1, obtaining original images corresponding to multiple existing videos, T2, generating reconstructed images for images in existing videos, and obtaining a data set based on each original image and its corresponding reconstructed image; T3, using the data set in step T2 to train the deep convolutional generative adversarial network until convergence.

[0018] Compared with the prior art, the advantages of the present invention are as follows: the present invention provides an improved video transmission method for band-striping, which deconstructs the image data into foreground, background and posture factors at the transmitting end, and each decomposition factor encodes the input image into embedded features through a coding network, and then reconstructs the features into images through a decoding system, and then uses a deep convolution adversarial generative network to generate more realistic images. It can be seen that the present invention has made improvements on the traditional video transmission method for band-striping, and has improved the quality of the generated image and ensured the transmission quality while reducing the transmission bandwidth and storage resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0020] Figure 1A schematic diagram of the main process of a video encoding and decoding technology based on banding in the prior art according to an embodiment of the present invention;

[0021] Figure 2 A schematic diagram of the overall architecture of a video transmission device according to an embodiment of the present invention;

[0022] Figure 3 A schematic diagram of the principle of a video transmission method according to an embodiment of the present invention;

[0023] Figure 4 A schematic diagram of a video transmission method according to an embodiment of the present invention;

[0024] Figure 5 A schematic diagram of key points of a human body according to an embodiment of the present invention;

[0025] Figure 6 A schematic diagram of the working principle of a human body posture coding model according to an embodiment of the present invention;

[0026] Figure 7 is a schematic diagram of a mask according to an embodiment of the present invention;

[0027] Figure 8 A schematic diagram of the working principle of a foreground coding model according to an embodiment of the present invention;

[0028] Fig. 9 A schematic diagram of the working principle of a background coding model according to an embodiment of the present invention;

[0029] Fig.10 Schematic diagram of the working principle of a decoding system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail by specific embodiments below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0031] As mentioned in the background technology, the existing video coding and decoding technology based on banding only considers whether the image is generated, but does not consider the quality of the generated image, which greatly affects the quality of video image transmission. To solve this problem, the inventor found through research that the problem of poor image generation quality can be solved by improving the image generation strategy.

[0032] First, the technical idea of ​​the present invention is introduced.

[0033] The purpose of the present invention is to overcome the problem of low quality of generated images in the prior art and to propose an improved method for transmitting video in a band-modulated manner. The overall system architecture of the improved method for transmitting video in a band-modulated manner of the present invention is shown in FIG. Figure 2As shown, the original video is obtained at the sending end, and the encoding system is input to extract features to obtain image background information, human posture information, and image foreground information. Then, the feature semantics are performed and the semantic information is transmitted to the receiving end. The receiving end decodes the feature semantic information and reconstructs the video based on it. It can be seen that the present invention is different from the prior art in that the information in the original image is processed differently and features are extracted separately, such as Figure 3 As shown in the figure, after the sending end completes image acquisition and obtains the original image, the original image is factorized (decomposed into posture factors, background factors, and foreground factors), and then the embedded features corresponding to each factor are generated. At the receiving end, a reconstructed image is generated based on the embedded features, and a high-quality reconstructed image is generated by training a generative adversarial network.

[0034] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0035] like Figure 4 As shown, the present invention mainly includes the following stages: the transmitting end identifies and decomposes the original image into three factors, the three factors generate embedded features through the encoder, the receiving end receives the embedded features to generate a reconstructed image, the reconstructed image and the original image are trained with a DCGAN, and the receiving end uses the trained DCGAN to generate a high-quality reconstructed image. Each stage is described in detail below.

[0036] 1. The sender identifies and decomposes the original image into three factors

[0037] According to one embodiment of the present invention, the transmitting end identifies and decomposes the original target image into three factors, so that the image becomes a combination of a foreground factor, a background factor and a posture factor.

[0038] 2. The three-part factors generate embedded features through the encoder

[0039] Each decomposition factor encodes the input image into an embedded feature through a corresponding encoding network, and then sends each embedded feature to the receiving end to upgrade the feature into a reconstructed image through a decoder.

[0040] According to one embodiment of the present invention, for the posture factor, firstly, a human posture model (such as an OpenPose model, the following embodiments of the present invention are described by taking the OpenPose model as an example) is called to extract 18 human skeleton key points. The so-called human skeleton key points are as follows: Figure 5As shown, it includes the right foot, right knee, right waist, right elbow, right hand, right shoulder, right ear, right eye, nose, head, left eye, left ear, left shoulder, left hand, left elbow, left waist, left knee, and left foot. Since some body areas may be blocked and invisible, the present invention introduces a visibility variable α∈{0,1}i=1,...,18, to represent the visibility of each key point. Therefore, the posture variable can be represented by a 54-dimensional vector (36-dimensional key point coordinates and 18-dimensional key point visibility α), wherein the 36-dimensional key point coordinates are the 18 key point coordinates of the human body extracted using the OpenPose model. Because each key point is represented by a two-dimensional coordinate (x, y) on a two-dimensional image, there are a total of 18*2=36-dimensional key point coordinates. The connected 54-dimensional vectors are input into the human posture automatic encoding model to reconstruct the posture information so that the embedded posture features can be decoded to obtain a heat map. According to one embodiment of the present invention, Figure 6 As shown in Figure 1, the human posture automatic encoding model consists of 4 fully connected residual blocks, a 32-dimensional bottleneck layer, and 4 fully connected residual blocks, each of which consists of two fully connected layers. Figure 6 In the figure (the vector is represented by dim), the 512-dimensional ones are all residual blocks, the 32-dimensional ones are bottleneck layers, the input is a 54-dimensional vector, and the output is a 54-dimensional reconstructed vector (i.e., a vector containing embedded features of human posture information). Since the basic units that make up the human posture automatic encoding model are common in this field, the specific details of the model will not be elaborated here.

[0041] According to one embodiment of the present invention, for the foreground factor, the present invention connects the key points of the human skeleton extracted during the posture factor processing to form a skeleton graph, and then uses morphological operations to obtain a rough posture such as Figure 7 The mask shown. In order to reduce the inaccuracy of the rough pose mask, the present invention directly applies the rough pose mask to the feature map instead of the input image. However, the foreground extracted in this way will include pose information. In order to decouple the pose information and the foreground, according to one embodiment of the present invention, the present invention uses 7 BodyROI models (Body Regions-of-Interest) models to encode pose invariant features, wherein for each ROI, a 48*48 feature map is extracted and passed to a foreground coding model with shared weights to improve learning efficiency. Finally, a 224-dimensional feature vector connected by 7 Body ROI embedded features is encoded, and the feature vector is the foreground information embedded feature. The foreground coding model shown is as follows. Figure 8 As shown, it mainly includes 5 convolution residual blocks. Figure 8The foreground ROI features are sequentially passed through the convolution residual blocks represented by 48*48, 24*24, 12*12, 6*6, and 3*3, and the foreground embedded features are output. The input is the ROI feature map. For each ROI, the corresponding 48*48 feature map must be subjected to feature extraction operations, and these 7 ROIs are operated synchronously, which can improve efficiency; therefore, the foreground coding model shares weights, and the last 7 32-dimensional outputs are combined into a 224-dimensional feature vector. Since the basic units that make up the foreground coding model are common in this field, the specific details of the model will not be elaborated here.

[0042] According to one embodiment of the present invention, for background factors, the present invention uses an inverse pose mask to obtain background features, and inputs the background features into a background coding model to obtain background information embedding features. According to one embodiment of the present invention, Fig. 9 As shown, the background encoder includes 5 sequentially connected convolution residual blocks, inputs 128*64*128 background features, and outputs 128-dimensional background embedding features. Since the basic units that make up the background encoding model are common in the art, the specific details of the model will not be elaborated here.

[0043] The foreground and background features are then concatenated and flattened into 128*64*352 appearance features and input into the decoding system together with the pose information embedded features.

[0044] 3. The receiving end receives the embedded features to generate a reconstructed image

[0045] The present invention takes into account that the features of the embedded space are low-dimensional and continuous. Image generation in the embedded space will save the computational overhead of the model. At the same time, compared with the actual data distribution, the features of the embedded space are easier to learn. Therefore, the present invention improves the previous method of directly generating images using Gaussian noise. The split embedded features are received at the receiving end, and then the appearance feature map and the extracted 18-channel posture key point heat map are connected and sent to a U-Net-based network, such as Fig.10 As shown, to generate a reconstructed image.

[0046] 4. The reconstructed image and the original image are used to train a generative adversarial network (DCGAN), and the trained generative adversarial network is used to generate high-quality reconstructed images.

[0047] According to one embodiment of the present invention, a deep convolutional generative adversarial network is trained in the following manner: T1, obtaining original images corresponding to multiple existing videos; T2, generating reconstructed images for images in existing videos, and obtaining a data set based on each original image and its corresponding reconstructed image; T3, using the data set in step T2 to train the deep convolutional generative adversarial network in an adversarial learning manner until convergence.

[0048] The present invention takes into account the image quality gap between the directly generated reconstructed image and the original image. After connecting the reconstructed image with the original image, the generator of the deep convolutional generative adversarial network (DCGAN, the generative adversarial network includes a generator and a discriminator) is used to obtain a difference map between the reconstructed image and the original image, and then the obtained difference map is connected with the reconstructed image to obtain a refined image, and then the original image and the refined image are input into the discriminator of the DCGAN together, so that its input is no longer a single real image and a forged image, but a paired image, and the DCGAN is trained in an adversarial learning manner to strengthen the network to generate a more realistic generated image. After the training is completed, the difference map generated by the trained DCGAN generator is connected with the reconstructed image obtained by decoding the embedded features to obtain a higher quality reconstructed image.

[0049] In addition, the training process of the encoding system and the decoding system is relatively common, taking the original image as input and the reconstructed image as output and training them until convergence, so it will not be repeated here.

[0050] The present invention provides an improved video transmission method for band-striping, which deconstructs image data into foreground, background and posture factors at the transmitting end, and each decomposition factor encodes the input image into embedded features through a coding network, and then reconstructs the features into images through a decoding system, and then uses a deep convolution adversarial generative network to generate more realistic images. It can be seen that the present invention has made improvements on the traditional video transmission method for band-striping, and has improved the quality of the generated image and ensured the transmission quality under the premise of reducing the transmission bandwidth and storage resources.

[0051] It should be noted that although the above describes the various steps in a specific order, it does not mean that the various steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0052] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0053] Computer readable storage medium can be a tangible device that holds and stores instructions used by an instruction execution device. Computer readable storage medium can include, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a protruding structure in a groove on which instructions are stored, and any suitable combination thereof.

[0054] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A video transmission device, characterized in that: The device comprises: A preprocessing module, which is configured at the transmitting end and is used to decompose the original target image in the video to be transmitted into a foreground factor, a background factor and a posture factor; The encoding system is configured at the transmitting end and is used to encode the original image based on the foreground factor, the background factor and the posture factor to obtain the foreground information embedding feature, the background information embedding feature and the posture information embedding feature and transmit them to the decoding system; A decoding system, which is configured at the receiving end and is used to decode the received embedded features to generate a reconstructed image; A deep convolutional generative adversarial network, the deep convolutional generative adversarial network also includes a generator, which is used to generate a difference map with the original image based on the reconstructed image generated by the decoding system, and superimpose the difference map with the reconstructed image generated by decoding to obtain a final reconstructed image, wherein the generator of the generative adversarial network is obtained by training the deep convolutional generative adversarial network based on a training set of original images and reconstructed images of the original images.

2. The device according to claim 1, characterized in that The coding system comprises: A posture encoder, used to encode the original target image based on the posture factor to obtain a posture information embedding feature; A background encoder is used to encode the original target image based on the foreground factor to obtain a foreground information embedding feature; The foreground encoder is used to encode the original target image based on the background factor to obtain the background information embedding feature.

3. The device according to claim 2, characterized in that The attitude encoder comprises: Human body posture model, used to extract human skeleton key points to obtain the corresponding key point coordinate information and key point visibility information; The human body posture automatic encoding model is used to encode and reconstruct posture information based on key point coordinate information and key point visibility information and decode embedded posture features based on the reconstructed posture information.

4. The device according to claim 3, characterized in that The human body posture automatic encoding model includes 4 residual blocks that are fully connected in sequence, a bottleneck layer, and 4 residual blocks that are fully connected in sequence, wherein each residual block includes two fully connected layers.

5. The device according to claim 3, characterized in that The foreground encoder comprises: A foreground feature extraction module, used to obtain a rough posture mask using morphology based on the human skeleton key point information extracted from the human posture model and perform feature mapping based on the mask to obtain a foreground feature containing posture information from the original target image; Multiple human ROI models are used to encode posture-invariant features for foreground features containing posture information respectively; A foreground coding model is used to encode foreground features based on posture invariant features to obtain foreground information embedded features, wherein the foreground coding model includes a plurality of sequentially connected convolution residual blocks.

6. The device according to claim 5, characterized in that The foreground encoder includes 7 human body ROI models, and the foreground encoding model includes 5 convolution residual blocks connected in sequence.

7. The device according to claim 5, characterized in that The background encoder comprises: A background feature extraction module, used for obtaining background features by using an inverse posture mask based on the coarse posture mask obtained by the foreground feature extraction module; A background coding model is used to obtain background information embedding features by encoding based on background features, wherein the background coding model includes a plurality of convolution residual blocks connected in sequence.

8. The device according to claim 7, characterized in that The background coding model includes 5 convolutional residual blocks connected in sequence.

9. The device according to claim 3, characterized in that The decoding system comprises: An appearance feature processing module, used for connecting and tiling foreground information embedded features and background information embedded features to obtain appearance features; U-net network is used to generate reconstructed images based on appearance features and posture information embedding features.

10. A video transmission method based on the video transmission device according to any one of claims 1 to 9, wherein the transmitting end transmits the original target image to the receiving end, characterized in that: The method comprises: S1. Decomposing the original target image to be transmitted at the transmitting end to decompose it into a foreground factor, a background factor, and a posture factor; S2, based on the foreground factor, background factor and posture factor distribution decomposed in step S1, the original image is encoded to obtain the foreground information, background information and posture information embedded features and transmitted to the decoding system; S3. Generate a reconstructed image based on the embedded features at the receiving end.

11. The method according to claim 10, characterized in that The step S2 comprises: S21, encoding the original target image based on the posture factor to obtain the posture information embedded feature; wherein the posture information embedded feature is obtained by: extracting the key points of the human skeleton to obtain the corresponding key point coordinate information and key point visibility information, encoding and reconstructing the posture information based on the key point coordinate information and the key point visibility information, and decoding the embedded posture feature based on the reconstructed posture information; S22, encoding the original target image based on the foreground factor to obtain a foreground information embedding feature, wherein the foreground information embedding feature is obtained by: obtaining a coarse posture mask based on human skeleton key point information using morphology, performing feature mapping based on the mask to obtain a foreground feature containing posture information from the original target image, encoding a posture invariant feature for the foreground feature containing posture information, and encoding the foreground feature based on the posture invariant feature to obtain the foreground information embedding feature; S23. Encode the original target image based on the background factor to obtain background information embedding features, wherein the background information embedding features are obtained by: obtaining background features based on the coarse posture mask using an inverse posture mask and encoding based on the background features to obtain background information embedding features.

12. The method according to claim 11, characterized in that The method further comprises: S4. Use the generator in the pre-trained deep convolutional generative adversarial network to generate a difference map between the reconstructed image and the original target image based on the reconstructed image generated in step S3, and superimpose the difference map with the reconstructed image generated in step S3 to obtain the final reconstructed image.

13. The method according to claim 12, characterized in that The deep convolutional generative adversarial network is pre-trained as follows: T1. Obtain the original images corresponding to multiple existing videos. T2, generate reconstructed images for images in existing videos, and obtain a data set based on each original image and its corresponding reconstructed image; T3. Use the data set in step T2 to train the deep convolutional generative adversarial network until convergence.

14. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of any method described in claims 10 to 13.

15. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the steps of the method as claimed in any one of claims 10 to 13.

Images