Image generation method and device based on all-pulse network, equipment and storage medium
The image generation method using a full pulse network, which processes images and pose images using an encoder, feature processor, and decoder, solves the problem of high computational cost in existing technologies and achieves the effect of rapidly generating high-fidelity images on edge devices.
Patent Information
- Application Number
- CN202211170135.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-22
- Publication Date
- 2026-02-27
- Estimated Expiration
- 2042-09-22
AI Technical Summary
In existing technologies, spiking neural networks require a large amount of computation for image generation tasks, making it difficult to quickly generate high-fidelity images on edge devices.
A full-pulse network is used to encode the original image and pose image through a first pulse encoder and a second pulse encoder. The image features are processed by a feature processor, and the target image is generated by a pulse decoder. The network includes multiple feature processing modules and pulse convolution blocks for information fusion and pulse residual processing.
It improves the realism of image generation, reduces computational load, and enables the rapid generation of high-fidelity images on edge devices.
Smart Images
Figure CN115631343B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of computers, and particularly relates to an image generation method and device based on a full spiking network, an equipment and a storage medium. BACKGROUND
[0002] Recently, a large number of studies have focused on improving the accuracy of spiking neural networks (SNN) in image classification, object detection and other tasks, achieving the same or even exceeding the effect of artificial neural networks (ANN) with fewer parameters and computational complexity. However, the image generation task implemented by ANN often requires more computational complexity, making it difficult to quickly generate high-fidelity images on edge devices. SUMMARY
[0003] The present application provides an image generation method and device based on a full spiking network, an equipment and a storage medium, aiming to improve the fidelity of generated images.
[0004] In a first aspect, the present application provides an image generation method based on a full spiking network, the full spiking network comprising a first spiking encoder, a second spiking encoder, a feature processor and a spiking decoder.
[0005] The image generation method based on a full spiking network comprises:
[0006] encoding at least one original image by the first spiking encoder to obtain original image features;
[0007] processing each original pose image and each target pose image by the second spiking encoder to obtain original pose features, wherein the target pose image and the original pose image are pose images of a target object collected at different sampling times;
[0008] processing the original image features and the original pose features by the feature processor to obtain target image features;
[0009] decoding the target image features by the spiking decoder to generate a target image of the target object;
[0010] wherein the at least one original image is obtained by copying a to-be-processed image, and the original pose image is a pose image corresponding to the original image.
[0011] In one embodiment, the feature processor comprises a plurality of identical feature processing modules connected in sequence.
[0012] The plurality of feature processing modules comprises a first feature processing module, a second feature processing module and a third feature processing module;
[0013] The processing of the original image features and the original pose features by the feature processor comprises:
[0014] The first image features and the first pose features are obtained by processing the original image features and the original pose features by the first feature processing module;
[0015] The second image features and the second pose features are obtained by processing the first image features and the first pose features by the second feature processing module;
[0016] The target image features are obtained by processing the second image features and the second pose features by the third feature processing module.
[0017] Any of the feature processing modules comprises a first pulse convolution block, a second pulse convolution block and a third pulse convolution block;
[0018] The processing of the second image features and the second pose features by the third feature processing module comprises:
[0019] The to-be-processed image features are obtained by processing the second image features by the first pulse convolution block;
[0020] The to-be-processed pose features are obtained by processing the second pose features by the second pulse convolution block;
[0021] The fusion image features are obtained by fusing the to-be-processed image features and the to-be-processed pose features by the third pulse convolution block;
[0022] The target image features are obtained based on the fusion image features.
[0023] Any of the feature processing modules further comprises a stacking module;
[0024] The fusing of the to-be-processed image features and the to-be-processed pose features by the third pulse convolution block comprises:
[0025] The stacked features are obtained by stacking the to-be-processed image features and the to-be-processed pose features along a channel by the stacking module;
[0026] The fusion image features are obtained by fusing the stacked features by the third pulse convolution block.
[0027] Any of the feature processing modules also each comprises an exclusive or module;
[0028] The obtaining the target image feature based on the fused image feature comprises:
[0029] The target image feature is obtained by performing pulse residual processing on the fused image feature and the second image feature through the exclusive or module.
[0030] The target image feature is T time steps of image features;
[0031] The decoding the target image feature through the pulse decoder to generate a target image of the target object comprises:
[0032] The T time steps of membrane potentials of the target image feature are weighted and summed through the last layer of the pulse decoder to generate a target image.
[0033] In a second aspect, the present application provides an image generation device based on a full pulse network, the full pulse network comprising a first pulse encoder, a second pulse encoder, a feature processor and a pulse decoder;
[0034] The image generation device based on a full pulse network comprises:
[0035] An encoding unit is configured to encode at least one original image through the first pulse encoder to obtain an original image feature;
[0036] A first processing unit is configured to process each original pose image and each target pose image through the second pulse encoder to obtain an original pose feature, wherein the target pose image and the original pose image are pose images of a target object collected at different sampling times;
[0037] A second processing unit is configured to process the original image feature and the original pose feature through the feature processor to obtain a target image feature;
[0038] A decoding unit is configured to decode the target image feature through the pulse decoder to generate a target image of the target object;
[0039] The at least one original image is obtained by copying a to-be-processed image, and the original pose image is a pose image corresponding to the original image.
[0040] In a third aspect, the present application further provides an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the image generation method based on a full pulse network according to the first aspect when executing the program.
[0041] In a fourth aspect, the present application also provides a non-transitory computer readable storage medium comprising a computer program which, when executed by the processor, implements the image generation method based on the full-pulse network according to the first aspect.
[0042] In a fifth aspect, the present application also provides a computer program product comprising a computer program which, when executed by the processor, implements the image generation method based on the full-pulse network according to the first aspect.
[0043] The image generation method based on the full-pulse network provided by the present application encodes at least one original image through a first pulse encoder to obtain original image features, processes each original pose image and each target pose image through a second pulse encoder to obtain original pose features, wherein the target pose image and the original pose image are pose images of a target object collected at different sampling times, processes the original image features and the original pose features through a feature processor to obtain target image features, and decodes the target image features through a pulse decoder to generate a target image of the target object, wherein the at least one original image is obtained by copying a to-be-processed image, and the original pose image is a pose image corresponding to the original image.
[0044] In the process of image generation based on the full-pulse network, the pulse encoder, the feature processor and the pulse decoder in the full-pulse network are combined with the multiple original images, the multiple original pose images and the multiple target pose images, and the target pose image can guide the generation of an image with a required degree of reality, thereby improving the reality of the generated image. BRIEF DESCRIPTION OF DRAWINGS
[0045] In order to more clearly illustrate the technical solutions of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.
[0046] Figure 1 is a flowchart of the image generation method based on the full-pulse network provided by the present application;
[0047] Figure 2 is a schematic diagram of the overall framework of the full-pulse network provided by the present application;
[0048] Figure 3 is a structural diagram of the feature processing module provided by the present application;
[0049] Figure 4is a structural diagram of an image generation device based on a full pulse network provided by the present application;
[0050] Figure 5 is a structural diagram of an electronic device provided by the present application. DETAILED DESCRIPTION
[0051] For the purpose of making the object, technical scheme and advantages of the present application more clear, the technical scheme of the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.
[0052] It should be noted that, in the description of the embodiments of the present application, the terms "comprise", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitation, the element defined by the statement "comprising a" does not exclude the presence of another identical element in the process, method, article or device comprising the element. The terms "upper", "lower" and the like indicate the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the indicated device or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. Unless otherwise explicitly specified and limited, the terms "mount", "connect", "connect" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium, or it can be connected inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.
[0053] The terms "first", "second", and the like in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally a category, and do not limit the number of objects, for example, the first object can be one or more. In addition, "and / or" in the specification and claims means at least one of the connected objects, and the character " / ", generally indicates that the front and rear associated objects are in an "or" relationship.
[0054] Further, in combination with Figures 1 to 5 The image generation method based on the full pulse network provided by the application, an image generation device based on the full pulse network, an image generation equipment based on the full pulse network and a storage medium are described. Figure 1 is a flowchart of the image generation method based on the full pulse network provided by the application; Figure 2 is a schematic diagram of the overall framework of the full pulse network provided by the application; Figure 3 is a structural diagram of the feature processing module provided by the application; Figure 4 is a structural diagram of the image generation device based on the full pulse network provided by the application; Figure 5 is a structural diagram of the electronic equipment provided by the application.
[0055] The embodiments of the image generation method based on the full pulse network provided by the application are provided, and it should be noted that although the logical order is shown in the flowchart, under certain data, the steps shown or described can be completed in an order different from that here.
[0056] The embodiments of the application take the electronic equipment as an execution main body for example, and the image generation system as the electronic equipment for example.
[0057] As Figure 1 , Figure 1 is a flowchart of the image generation method based on the full pulse network provided by the application. The image generation method based on the full pulse network provided by the embodiments of the application comprises:
[0058] S101, encoding at least one original image through the first pulse encoder to obtain original image features;
[0059] S102, processing each original pose image and each target pose image through the second pulse encoder to obtain original pose features, the target pose image and the original pose image being pose images of a target object collected at different sampling moments;
[0060] S103, processing the original image features and the original pose features through the feature processor to obtain target image features;
[0061] S104, decoding the target image features through the pulse decoder to generate a target image of the target object.
[0062] It should be noted that the image generation system provided by the application is a system carrying the full pulse network, and the image generation system is described with reference to Figure 2 as shown, Figure 2is a schematic diagram of a whole pulse network framework provided by the present application, the whole pulse network at least comprising a first pulse encoder, a second pulse encoder, a feature processor and a pulse decoder.
[0063] Further, the feature processor comprises T feature processing modules connected in sequence, the T feature processing modules being feature processing module 1, feature processing module 2 to feature processing module T respectively, and each of the feature processing module 1, the feature processing module 2 to the feature processing module T has the same internal structure and operating principle.
[0064] Further, referring to Figure 2 , the whole pulse network can be divided into two branches, and each branch has a pulse encoder at the input end. Further, one branch of the whole pulse network is used for processing image information, that is, an image branch. The other branch is used for processing pose images, that is, a pose branch. That is, it can be understood that the first pulse encoder in the whole pulse network is used for processing image information, and the second pulse encoder is used for processing pose images.
[0065] Before the original image and the pose image are input into the pulse encoder, the initial to-be-processed image of the target object (the target object includes but is not limited to animals, objects with joints and movable and swingable objects) needs to be collected, and the to-be-processed image is copied T times to obtain T original images which are the same. Since the to-be-processed image is copied T times, the size of the image input by the first pulse encoder in the image branch is C I *H I *W I *T, wherein C I is the number of image channels, H I is the height of the image, W I is the width of the image, and T is the number of time steps.
[0066] It should be noted that the pose information in each original image can be extracted, and the extracted pose information can be converted into an original pose image corresponding to the original image. Since the to-be-processed image is copied T times to obtain T original images which are the same, it can be understood that the original pose image is also copied T times to obtain T original pose images which are the same. Further, after the target object changes action, a target pose image after the target object changes action is collected, which can be understood as that the target pose image and the original pose image are collected at different sampling times. The original pose image is the pose image of the target object before the action changes, and the target pose image is the pose image of the target object after the action changes. At the same time, the target pose image is also copied T times to obtain T target pose images which are the same.
[0067] Further, the target pose image and the original pose image constitute a pose image pair, and since the pose image pair is copied T times, the size of the second pulse encoder input image in the pose branch is 2*C P *H I *W I *T, wherein C P is the number of pose joints. It should be further pointed out that the pose image includes but is not limited to pose information, pose joint information, and the number of pose joints.
[0068] Specifically, after the first pulse encoder of the image branch receives the input T original images, the T original images are encoded to obtain original image features Further, after the second pulse encoder of the pose branch receives the input T pose image pairs (target pose image and original pose image), the T pose image pairs (target pose image and original pose image) are encoded to obtain original pose features
[0069] Further, the first pulse encoder of the image branch transmits the original image features to the feature processor, and the second pulse encoder of the pose branch simultaneously transmits the original pose features to the feature processor. After the feature processor receives the original image features and the original pose features , multiple feature processing modules in the feature processor process the original image features and the original pose features to obtain target image features It should be further pointed out that the target image features are the image features of the image of the target object after the motion change, which are obtained based on the original pose image and the changed target pose image and are cut out from the original image.
[0070] Further, the feature processor transmits the obtained target image features to the pulse decoder. After the pulse decoder receives the target image features transmitted by the T feature processing modules, the target image features are decoded to generate a target image under a target pose. It should be further pointed out that the target image is the image of the target object after the motion change, which is obtained based on the original pose image and the changed target pose image and is cut out from the original image.
[0071] The image generation method based on the full-pulse network provided by the embodiment of the present application encodes at least one original image through a first pulse encoder to obtain original image features; processes each original pose image and each target pose image through a second pulse encoder to obtain original pose features, wherein the target pose image and the original pose image are pose images of a target object collected at different sampling moments; processes the original image features and the original pose features through a feature processor to obtain target image features; and decodes the target image features through a pulse decoder to generate a target image of the target object; wherein the at least one original image is obtained by copying a to-be-processed image, and the original pose image is a pose image corresponding to the original image.
[0072] In the process of image generation based on the full-pulse network, the pulse encoder, the feature processor and the pulse decoder in the full-pulse network are combined with the plurality of original images, the plurality of original pose images and the plurality of target pose images, and the target pose image can be used to guide the generation of an image with a required degree of reality, thereby improving the reality of the generated image. It can be further understood that, after the target object changes in motion, the original pose image and the changed target pose image can be used to generate a target image with a high degree of reality after the target object changes in motion on the basis of the original image under the guidance of the target pose image.
[0073] The specific analysis of S103, which records processing of the original image features and the original pose features through the feature processor to obtain target image features, is as follows:
[0074] The first image features and the first pose features are obtained by processing the original image features and the original pose features through the first feature processing module;
[0075] The second image features and the second pose features are obtained by processing the first image features and the first pose features through the second feature processing module;
[0076] The target image features are obtained by processing the second image features and the second pose features through the third feature processing module.
[0077] It should be noted that the first feature processing module, the second feature processing module and the third feature processing module in the embodiment of the present application are not representative of only three feature processing modules, and the first feature processing module, the second feature processing module and the third feature processing module are only representative of the front and back order of the feature processing modules. The second feature processing module in the embodiment of the present application can be an intermediate feature processing module in the feature processor, the first feature processing module can be all feature processing modules before the intermediate feature processing module, and the third feature processing module can be all feature processing modules after the intermediate feature processing module. The number of the feature processors in the embodiment of the present application is exemplified as T, the second feature processing module is an intermediate feature processing module t, the first feature processing module is feature processing module 1 to feature processing module t-1, and the third feature processing module is feature processing module t+1, feature processing module T-1 to feature processing module T.
[0078] Specifically, the feature processing module 1 receives original image features and original pose features After that, the original image features and the original pose features are processed, and the first image features and the first pose features are output. Further, the feature processing module 2 takes the first image features and the first pose features output by the feature processing module 1 as input, processes the first image features and the first pose features , and outputs the second image features and the second pose features . In turn, the feature processing module t takes the t-1 image features and the t-1 pose features output by the feature processing module t-1 as input, processes the t-1 image features and the t-1 pose features , and outputs the t image features and the t pose features .
[0079] In turn, the feature processing module T-1 takes the T-2 image features and the T-2 pose features output by the feature processing module T-2 as input, processes the T-2 image features and the T-2 pose features , and outputs the T-1 image features and the T-1 pose features . Further, the feature processing module T takes the T-1 image features and the T-1 pose feature for input, the T-1 image feature and the T-1 pose feature processing, output the target image feature
[0080] The embodiment of the present application processes the original image feature and the original pose feature in turn through the feature processing module in the feature processor, and the authenticity of the target image feature finally generated under the guidance of the target pose image meets the requirements, so that the image with the authenticity meeting the requirements can be generated, and the authenticity of the generated image is improved.
[0081] Further, the second image feature and the second pose feature are processed through the third feature processing module to obtain the target image feature, and the specific analysis is as follows:
[0082] The second image feature is processed through the first pulse convolution block to obtain a to-be-processed image feature;
[0083] The second pose feature is processed through the second pulse convolution block to obtain a to-be-processed pose feature;
[0084] The to-be-processed image feature and the to-be-processed pose feature are fused through the third pulse convolution block to obtain a fused image feature;
[0085] The target image feature is obtained based on the fused image feature.
[0086] Further, referring to Figure 3 , Figure 3 is the structure diagram of the feature processing module provided by the present application, that is, the internal structure and operation principle of each feature processing module in the feature processor are Figure 3 the structure. As shown in Figure 3 , the feature processing module also has two branches, one branch is used for processing the image feature, that is, the image branch. The other branch is used for processing the pose feature, that is, the pose branch. Further, the feature processing module includes a first pulse convolution block and a second pulse convolution block, that is, the image branch includes the first pulse convolution block, and the pose branch includes the second pulse convolution block, wherein the first pulse convolution block and the second pulse convolution block are two 3*3 pulse convolution blocks. Further, the feature processing module further includes two stacking modules, a third pulse convolution block and an XOR module, the third pulse convolution block is a 1*1 pulse convolution block, the pulse convolution block is composed of a convolution layer, a batch normalization layer and a LIF (Leaky Integrate-and-Fire) layer, and the XOR module is composed of two linear layers.
[0087] Specifically, the embodiment of the present application takes the tth feature processing module as an example, and the input of the tth feature processing module is the image feature output by the (t-1)th feature processing module and the pose feature That is, the second image feature is The second pose feature is The second image feature The size of the second image feature is C*H*W*T, and the size of the second pose feature is 2*C*H*W*T.
[0088] Further, in the image branch of the tth feature processing module, the second image feature is processed through a first pulse convolution block (two 3*3 pulse convolution blocks) to obtain a to-be-processed image feature. In the pose branch of the tth feature processing module, the second pose feature is processed through a second pulse convolution block (two 3*3 pulse convolution blocks) to obtain a to-be-processed pose feature. Further, the to-be-processed image feature and the to-be-processed pose feature are fused through a third pulse convolution block (1*1 pulse convolution block) in the tth feature processing module to obtain a fused image feature.
[0089] Further, the tth feature processing module fuses the second image feature the fused image feature and the to-be-processed pose feature to output the tth image feature and the tth pose feature , wherein the size of the tth image feature and the tth pose feature is the same as the size of the second image feature and the second pose feature , that is, the size of the tth image feature is C*H*W*T, and the size of the tth pose feature is 2*C*H*W*T. Finally, the target image feature is obtained according to the tth image feature output by the tth feature processing module and the tth pose feature
[0090] The embodiment of the present application processes the image feature and the pose feature through the first pulse convolution block, the second pulse convolution block and the third pulse convolution block in the feature processing module, and the authenticity of the target image feature finally generated under the guidance of the target pose image meets the requirements, so that an image with authenticity meeting the requirements can be generated, and the authenticity of the generated image is improved.
[0091] Further, the third pulse convolution block is used for fusing the to-be-processed image feature and the to-be-processed pose feature, and specific analysis of obtaining the fused image feature is as follows:
[0092] The to-be-processed image feature and the to-be-processed pose feature are stacked along a channel by the stacking module, and stacked features are obtained.
[0093] The stacked features are fused by the third pulse convolution block, and the fused image feature is obtained.
[0094] Specifically, the stacking module is used for stacking the to-be-processed image feature and the to-be-processed pose feature along a channel, and stacked features are obtained. Further, the 1*1 pulse convolution block in the feature processing module is used for fusing the stacked features, and the fused image feature is obtained.
[0095] The embodiment of the present application processes the image feature and the pose feature by the stacking module and the third pulse convolution block, and the authenticity of the target image feature generated under the guidance of the target pose image meets the requirements, so that the image with the required authenticity can be generated, and the authenticity of the generated image is improved.
[0096] Further, specific analysis of obtaining the target image feature based on the fused image feature is as follows:
[0097] The fused image feature and the second image feature are processed by the exclusive or module, and the target image feature is obtained.
[0098] It should be noted that the embodiment of the present application realizes the pulse residual error in the pulse network through the exclusive or operation of the exclusive or module, and the exclusive or operation is realized by two linear layers.
[0099] Specifically, the two inputs for calculating the residual error are tiled respectively, and features with a size of M*T are obtained. After stacking, the size of the two features is M*T*2. After passing through linear layers with weight sizes of 2*2 and 2*1, the result of the exclusive or operation (with a size of M*T*1) is obtained, and the size is adjusted to C*H*W*T.
[0100] Therefore, taking the tth feature processing module as an example, the tth feature processing module performs pulse residual error processing on the fused image feature and the second image feature to obtain the tth image feature Further, the tth feature processing module stacks the tth image feature and the to-be-processed pose feature obtained by processing the second pose feature by the first pulse convolution block along a channel by the stacking module, and the tth pose feature is obtained.
[0101] is sequentially performed until the Tth feature processing module determines the image feature of the image branch as the target image feature
[0102] The embodiment of the present application performs pulse residual processing on the fused image feature and the second image feature through the XOR module, and the authenticity of the target image feature generated under the guidance of the target pose image meets the requirements, so that an image with required authenticity can be generated, and the authenticity of the generated image is improved.
[0103] Further, the target image feature is decoded by the pulse decoder to generate the target image of the target object, and the specific analysis is as follows:
[0104] The membrane potentials of the T time steps of the target image feature are weighted and summed by the last layer of the pulse decoder to generate the target image.
[0105] Specifically, the last layer of the pulse decoder weights and sums the membrane potentials of the T time steps of the target image feature to generate the target image.
[0106] The embodiment of the present application weights and sums the membrane potentials of the T time steps to generate an image with required authenticity, and improves the authenticity of the generated image.
[0107] Further, the steps of the present application are analyzed as follows:
[0108] After the first pulse encoder of the image branch receives the input T original images, the T original images are encoded and processed to obtain original image features Further, the second pulse encoder of the pose branch receives the input T pose image pairs, encodes and processes the T pose image pairs to obtain original pose features
[0109] After the feature processing module 1 receives the original image features and the original pose features , the original image features are processed by the first pulse convolution block (two 3*3 pulse convolution blocks) to obtain the first to-be-processed image features, and the original pose features The first image feature is obtained by performing pulse residual processing on the first image feature and the first pose feature. The first image feature is obtained by performing pulse residual processing on the first image feature and the first pose feature. The first image feature is obtained by performing pulse residual processing on the first image feature and the first pose feature. The first image feature is obtained by performing pulse residual processing on the first image feature and the first pose feature.
[0110] The first image feature is obtained by performing pulse residual processing on the first image feature and the first pose feature. The first image feature is obtained by performing pulse residual processing on the first image feature and the first pose feature. The first image feature is obtained by performing pulse residual processing on the first image feature and the first pose feature.
[0111] The first image feature is obtained by performing pulse residual processing on the first image feature and the first pose feature. The first image feature is obtained by performing pulse residual processing on the first image feature and the first pose feature. The first image feature is obtained by performing pulse residual processing on the first image feature and the first pose feature. The first image feature is obtained by performing pulse residual processing on the first image feature and the first pose feature. The first image feature is obtained by performing pulse residual processing on the first image feature and the first pose feature. The first image feature is obtained by performing pulse residual processing on the first image feature and the first pose feature. The first image feature is obtained by performing pulse residual processing on the first image feature and the first pose feature. The first image feature is obtained by performing pulse residual processing on the first image feature and the first pose feature.
[0112] The first image feature is obtained by performing pulse residual processing on the first image feature and the first pose feature. The first image feature is obtained by performing pulse residual processing on the first image feature and the first pose feature.
[0113] The first image feature is obtained by performing pulse residual processing on the first image feature and the first pose feature. The first image feature is obtained by performing pulse residual processing on the first image feature and the first pose feature. The first image feature is obtained by performing pulse residual processing on the first image feature and the first pose feature. The first image feature is obtained by performing pulse residual processing on the first image feature and the first pose feature.
[0114] Feature processing module T receives the T-1th image features and T-1 attitude features Then, the features of the (T-1)th image are processed through the first pulse convolution block (two 3*3 pulse convolution blocks). The image is processed to obtain the T-th image features. These features are then processed using a second pulse convolutional block (two 3x3 pulse convolutional blocks) to refine the (T-1)-th pose features. The process is performed to obtain the T-th pose feature to be processed. Further, the feature processing module T stacks the T-th image feature and the T-th pose feature along the channel using a stacking module to obtain the stacked features. Further, the feature processing module T fuses the stacked features using a third pulse convolution block (a 1*1 pulse convolution block) to obtain the fused image features. Further, the feature processing module T uses an XOR module to combine the fused image features with the (T-1)-th image features. Pulse residual processing is performed to obtain the T-th image features. and the Tth image features Identified as target image features
[0115] The target loss function in this embodiment of the invention is L full Target loss function L full Due to L1 loss, L per Perceived loss and L GAN Generative adversarial loss is defined as the target loss function L. full For L full =argmin G max D λ1L GAN +λ2L1+λ3L per .
[0116] Furthermore, the image generation apparatus based on a full-pulse network provided by the present invention will be described below. The image generation apparatus based on a full-pulse network and the image generation method based on a full-pulse network can be referred to each other accordingly.
[0117] like Figure 4 As shown, Figure 4 This is a structural diagram of the image generation device based on a full-pulse network provided by the present invention. The full-pulse network includes a first pulse encoder, a second pulse encoder, a feature processor, and a pulse decoder.
[0118] An image generation device based on a full-pulse network includes:
[0119] The encoding unit 401 is used to encode at least one original image through the first pulse encoder to obtain original image features;
[0120] The first processing unit 402 is configured to process each original pose image and each target pose image by the second pulse encoder to obtain original pose features, wherein the target pose image is a pose image of the target object collected at a different sampling time.
[0121] The second processing unit 403 is configured to process the original image features and the original pose features by the feature processor to obtain target image features.
[0122] The decoding unit 404 is configured to decode the target image features by the pulse decoder to generate a target image of the target object.
[0123] The at least one original image is obtained by copying the to-be-processed image, and the original pose image is a pose image corresponding to the original image.
[0124] Further, the second processing unit 403 is further configured to:
[0125] process the original image features and the original pose features by the first feature processing module to obtain first image features and first pose features;
[0126] process the first image features and the first pose features by the second feature processing module to obtain second image features and second pose features;
[0127] process the second image features and the second pose features by the third feature processing module to obtain the target image features.
[0128] Further, the second processing unit 403 is further configured to:
[0129] process the second image features by the first pulse convolution block to obtain to-be-processed image features;
[0130] process the second pose features by the second pulse convolution block to obtain to-be-processed pose features;
[0131] fuse the to-be-processed image features and the to-be-processed pose features by the third pulse convolution block to obtain fused image features;
[0132] obtain the target image features based on the fused image features.
[0133] Further, the second processing unit 403 is further configured to:
[0134] The to-be-processed image features and the to-be-processed pose features are stacked along a channel by the stacking module to obtain stacked features.
[0135] The stacked features are fused by the third pulse convolution block to obtain the fused image features.
[0136] Further, the second processing unit 403 is further configured to:
[0137] The fused image features and the second image features are subjected to pulse residual processing by the XOR module to obtain the target image features.
[0138] Further, the decoding unit 404 is further configured to:
[0139] The membrane potentials of the T time steps of the target image features are weighted and summed by the last layer of the pulse decoder to generate a target image.
[0140] The specific embodiments of the image generation device based on the full-pulse network provided by the present application are basically the same as the embodiments of the image generation method based on the full-pulse network, and will not be repeated here.
[0141] Figure 5 An example of an entity structure diagram of an electronic device is shown in Figure 5 As shown, the electronic device can include a processor 510, a communications interface 520, a memory 530, and a communications bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other through the communications bus 540. The processor 510 can invoke the logical instructions in the memory 530 to execute the image generation method based on the full-pulse network, which includes:
[0142] At least one original image is encoded by the first pulse encoder to obtain original image features;
[0143] Each original pose image and each target pose image are processed by the second pulse encoder to obtain original pose features, wherein the target pose image and the original pose image are pose images of a target object collected at different sampling times;
[0144] The original image features and the original pose features are processed by the feature processor to obtain target image features;
[0145] The target image features are decoded by the pulse decoder to generate a target image of the target object;
[0146] The at least one original image is obtained by copying a to-be-processed image, and the original posture image is a posture image corresponding to the original image.
[0147] In addition, the logic instructions in the memory 530 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.
[0148] In another aspect, the present application also provides a computer program product, which comprises a computer program stored on a non-transitory computer readable storage medium, and the computer program comprises program instructions, when the program instructions are executed by a computer, the computer can execute the image generation method based on the full pulse network provided by the above-mentioned method, and the method comprises:
[0149] Encoding at least one original image through the first pulse encoder to obtain original image features;
[0150] Processing each original posture image and each target posture image through the second pulse encoder to obtain original posture features, wherein the target posture image and the original posture image are posture images of a target object collected at different sampling times;
[0151] Processing the original image features and the original posture features through the feature processor to obtain target image features;
[0152] Decoding the target image features through the pulse decoder to generate a target image of the target object;
[0153] The at least one original image is obtained by copying a to-be-processed image, and the original posture image is a posture image corresponding to the original image.
[0154] In yet another aspect, the present application also provides a non-transitory computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the above-provided image generation method based on a full-pulse network, the method comprising:
[0155] encoding, by the first pulse encoder, at least one original image to obtain original image features;
[0156] processing, by the second pulse encoder, each original pose image and each target pose image to obtain original pose features, wherein the target pose image is a pose image of a target object collected at a different sampling time than the original pose image;
[0157] processing, by the feature processor, the original image features and the original pose features to obtain target image features;
[0158] decoding, by the pulse decoder, the target image features to generate a target image of the target object;
[0159] wherein the at least one original image is obtained by copying a to-be-processed image, and the original pose image is a pose image corresponding to the original image.
[0160] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present embodiment according to actual needs. Those skilled in the art can understand and implement without creative labor.
[0161] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software and necessary universal hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in terms of contribution to the prior art, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.
[0162] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; and although the present application has been described in detail with reference to the foregoing embodiments, it should be appreciated by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features thereof can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. An image generation method based on a full-pulse network, characterized in that, The full pulse network includes a first pulse encoder, a second pulse encoder, a feature processor, and a pulse decoder; The image generation method based on a full-pulse network includes: At least one original image is encoded using the first pulse encoder to obtain the original image features; The second pulse encoder processes each original pose image and each target pose image to obtain the original pose features, wherein the target pose image and the original pose image are pose images of the target object acquired at different sampling times. The original image features and the original pose features are processed by the feature processor to obtain the target image features; The target image of the target object is generated by decoding the features of the target image using the pulse decoder. Wherein, the at least one original image is obtained by copying the image to be processed, and the original pose image is the pose image corresponding to the original image.
2. The image generation method based on a full-pulse network according to claim 1, characterized in that, The feature processor comprises multiple identical feature processing modules cascaded in sequence.
3. The image generation method based on a full-pulse network according to claim 2, characterized in that, The plurality of feature processing modules include a first feature processing module, a second feature processing module, and a third feature processing module; The step of processing the original image features and the original pose features through the feature processor to obtain the target image features includes: The first feature processing module processes the original image features and the original pose features to obtain the first image features and the first pose features. The second feature processing module processes the first image features and the first pose features to obtain the second image features and the second pose features. The target image features are obtained by processing the second image features and the second pose features through the third feature processing module.
4. The image generation method based on a full-pulse network according to claim 3, characterized in that, Each of the aforementioned feature processing modules includes a first pulse convolution block, a second pulse convolution block, and a third pulse convolution block; The process of processing the second image features and the second pose features through the third feature processing module to obtain the target image features includes: The second image features are processed by the first pulse convolution block to obtain the image features to be processed; The second pose feature is processed by the second pulse convolution block to obtain the pose feature to be processed; The image features to be processed and the pose features to be processed are fused by the third pulse convolution block to obtain fused image features; The target image features are obtained based on the fused image features.
5. The image generation method based on a full-pulse network according to claim 4, characterized in that, Each of the aforementioned feature processing modules further includes a stacking module; The process of fusing the image features to be processed and the pose features to be processed through the third pulse convolution block to obtain fused image features includes: The stacking module stacks the image features and pose features to be processed along the channel to obtain the stacked features. The stacked features are fused using the third pulse convolution block to obtain the fused image features.
6. The image generation method based on a full-pulse network according to claim 4, characterized in that, Each of the aforementioned feature processing modules further includes an XOR module; The process of obtaining the target image features based on the fused image features includes: The target image features are obtained by performing pulse residual processing on the fused image features and the second image features through the XOR module.
7. The image generation method based on a full-pulse network according to any one of claims 1 to 6, characterized in that, The target image features are image features at T time steps; The step of decoding the target image features using the pulse decoder to generate a target image of the target object includes: The final layer of the pulse decoder performs a weighted sum of the membrane potentials of the target image features over T time steps to generate the target image.
8. An image generation device based on a full-pulse network, characterized in that, The full pulse network includes a first pulse encoder, a second pulse encoder, a feature processor, and a pulse decoder; The image generation device based on a full-pulse network includes: An encoding unit is used to encode at least one original image using the first pulse encoder to obtain original image features; The first processing unit is used to process each original pose image and each target pose image through the second pulse encoder to obtain original pose features, wherein the target pose image and the original pose image are pose images of the target object acquired at different sampling times. The second processing unit is used to process the original image features and the original pose features through the feature processor to obtain the target image features; A decoding unit is used to decode the target image features through the pulse decoder to generate a target image of the target object; Wherein, the at least one original image is obtained by copying the image to be processed, and the original pose image is the pose image corresponding to the original image.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the image generation method based on a full-pulse network as described in any one of claims 1 to 7.
10. A non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the image generation method based on a full-pulse network as described in any one of claims 1 to 7.