A method and related equipment for generating cartoon images based on variational autoencoder

Abstract

The present invention provides a method for generating cartoon images based on a variational autoencoder and related equipment. An auxiliary feature extraction layer is set in the first convolution submodule of the decoder module to obtain a weight distribution for performing matrix dot multiplication with a feature matrix after convolution through a conventional convolution layer. The obtained feature matrix enhances the accurate decoding of image information. An adaptive hybrid algorithm and a reordering attention algorithm are integrated into the third convolution submodule. The adaptive hybrid algorithm is used to generate difficult samples from the feature extraction results output by the previous convolution submodule. According to the linear combination ratio of the feature extraction results output by the previous convolution submodule in the difficult samples, the distance between the difficult samples and the input initial cartoon image in the feature space is reduced to control the generation range of the feature map. The reordering attention algorithm is used to access the feature extraction results using a static random access memory during the convolution process, thereby improving the rate of generating cartoon images.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method for generating cartoon images based on a variational autoencoder and related equipment. Background Art

[0002] Anime, a combination of animation and comics, uses concise and exaggerated techniques to depict or interpret life and social current affairs, and through various methods, allows the audience to understand the profound meaning that the anime works want to express while being entertained. With the continuous development of science and technology, anime has gradually become one of the main ways for users to entertain themselves in their daily lives. During the production process of anime, a large number of anime images need to be drawn.

[0003] Currently, existing technologies for drawing cartoon images mainly rely on manual work. Although production staff have improved the efficiency of drawing cartoon images through auxiliary tools such as computers, it still requires deep human participation. Drawing cartoon images is still a job that requires a lot of manpower costs and time.

[0004] The existing technology provides a fine-tuned variational auto-encoder (VAE) variant in the artificial intelligence text-to-image model Stable Diffusion - ft_EMA and ft_MSE, where EMA is the exponential moving average and MSE is the mean square error. EMA generates clearer images, while MSE generates smoother images. This model takes the image information extracted by the upper unit network as input, and then decodes the abstract image information in a unique latent space. This function can better restore the subtle details of the image and help render tiny details such as eyes and skin. This technology is applied to multimodal artificial intelligence-generated content (AI-Generated Content, AIGC) scenarios, which can greatly improve the quality of multimodal image generation. Among current technologies, due to its improved accuracy, high efficiency, and good robustness, it can greatly expand the application areas of AIGC tasks in actual production and life. However, due to the complexity of the AIGC multimodal field and the variability of image scenes, this technology still cannot decode the information of some details of the image well, the recovery quality is poor, and it still consumes a relatively large amount of time in terms of the model generation rate. Summary of the Invention

[0005] The present invention provides a method for generating cartoon images based on a variational autoencoder and related equipment, the purpose of which is to control the generation range of images and improve the calculation rate of large-parameter network units.

[0006] To achieve the above-mentioned object, the present invention provides a method for generating cartoon images based on a variational autoencoder, wherein the variational autoencoder includes an encoder module and an improved decoder module. The method for generating cartoon images includes:

[0007] Step 1: Obtain multiple initial animation images to be processed;

[0008] Step 2: multiple encoding vectors, where the encoding vectors correspond one-to-one to the initial cartoon images to be processed;

[0009] Step 3: All code vectors are input into a latent space region for compressing the code vectors to convert them, thereby obtaining a plurality of latent space vectors, where the latent space vectors correspond one-to-one to the code vectors.

[0010] Step 4: Input all latent space vectors into the improved decoder module for decoding respectively to obtain multiple decoding results, and generate animated images based on the multiple decoding results;

[0011] The improved decoder module includes a first convolution submodule for feature extraction, a second convolution submodule for performing convolution processing on the feature extraction result, and a third convolution submodule for limiting the generation range of the feature extraction result output by the second convolution submodule and accessing the feature extraction result using a static random access memory during the convolution process.

[0012] The first convolution submodule incorporates an auxiliary feature extraction layer, which is used to perform k 1-dimensional convolution operations on the input image data;

[0013] The third convolution submodule incorporates an adaptive hybrid algorithm and a reordering attention algorithm. The adaptive hybrid algorithm is used to control the generation range of the feature map by reducing the distance difference between the feature extraction results output by the second convolution submodule. The reordering attention algorithm is used to access the feature extraction results using static random access memory during the convolution process.

[0014] Furthermore, the first convolution submodule includes a 16-dimensional convolution unit, a 32-dimensional convolution unit, and a 64-dimensional convolution unit connected in sequence, and an auxiliary feature extraction layer is provided in the 16-dimensional convolution unit, the 32-dimensional convolution unit, and the 64-dimensional convolution unit;

[0015] The input of the auxiliary feature extraction layer in the 16-dimensional convolution unit and the input of the 16-dimensional convolution unit are both connected to the output of the latent space region, and the output of the auxiliary feature extraction layer in the 16-dimensional convolution unit is connected to the input of the 32-dimensional convolution unit;

[0016] The input end of the auxiliary feature extraction layer in the 32-dimensional convolution unit is connected to the output end of the 16-dimensional convolution unit, and the output end of the auxiliary feature extraction layer in the 32-dimensional convolution unit is connected to the input end of the 64-dimensional convolution unit;

[0017] The input end of the auxiliary feature extraction layer in the 64-dimensional convolution unit is connected to the output end of the 32-dimensional convolution unit, and the output end of the auxiliary feature extraction layer in the 64-dimensional convolution unit and the output end of the 64-dimensional convolution unit are both connected to the input end of the second convolution sub-module.

[0018] Furthermore, the auxiliary feature extraction layer includes:

[0019] Global average pooling sublayer, 1D convolution sublayer, and splicing sublayer;

[0020] The input of the global average pooling sublayer is the input of the auxiliary feature extraction layer, and the output of the global average pooling sublayer is connected to the input of the 1D convolution sublayer. The output of the 1D convolution sublayer and the input of the global average pooling sublayer are both connected to the input of the splicing sublayer, and the output of the splicing sublayer is the output of the auxiliary feature extraction layer.

[0021] The auxiliary feature extraction layer only performs k 1-dimensional convolution operations each time, and the calculation formula for k is:

[0022]

[0023] Among them, C represents the number of channels of the auxiliary feature extraction layer, γ and b are hyperparameters, and k represents the number of one-dimensional convolutions.

[0024] Furthermore, the third convolution submodule includes:

[0025] 512-dimensional convolutional unit, 768-dimensional convolutional unit, and 1024-dimensional convolutional unit connected in sequence;

[0026] The input end of the 512-dimensional convolution unit is connected to the output end of the second convolution submodule;

[0027] The output of the 1024-dimensional convolution unit is the output of the third convolution submodule;

[0028] Adaptive hybrid algorithms and reordering attention algorithms are integrated into the 512-dimensional convolutional units, 768-dimensional convolutional units, and 1024-dimensional convolutional units.

[0029] More specifically, the adaptive hybrid algorithm is used to:

[0030] Using the formula Generate difficult samples from the feature extraction results output by the previous level convolution unit Among them, g(·) represents the linear combination function, x i Represents the feature extraction result output by the previous convolution unit, x j represents the decoding result, λ represents the hyperparameter sampled from the data distribution, Indicates difficult samples;

[0031] According to the linear combination ratio of the feature extraction results output by the previous convolution unit in the difficult sample, the distance between the difficult sample and the feature extraction results output by the previous convolution unit in the feature space is reduced. The calculation formula is:

[0032]

[0033] where σ represents the noise term sampled from the Gaussian distribution of the data, represents the metric function, and F(·) represents the information extractor in the model.

[0034] Furthermore, the reordering attention algorithm that uses static random access memory to access feature extraction results during the convolution process is specifically used to:

[0035] Obtaining the size of a preset input and output matrix in a static random access memory for storing feature extraction results;

[0036] Get the Q matrix, K matrix, and V matrix from the feature extraction results output by the previous level convolution unit, and split the Q matrix, K matrix, and V matrix into the same size as the input and output matrices;

[0037] The split Q matrix, K matrix, and V matrix are calculated in the static random access memory to obtain a Q result matrix, a K result matrix, and a V result matrix;

[0038] The Q result matrix, K result matrix, and V result matrix are concatenated and input into the next level convolution unit.

[0039] The present invention also provides a device for generating an animated image based on a variational autoencoder, wherein the variational autoencoder includes an encoder module and an improved decoder module, and the device for generating an animated image includes:

[0040] An acquisition module is used to obtain multiple initial animation pictures to be processed;

[0041] An encoding module is used to input all the initial cartoon images to be processed into an encoder module for encoding to obtain multiple encoding vectors, and the encoding vectors correspond one-to-one to the initial cartoon images to be processed;

[0042] a conversion module, configured to convert all code vectors input into a latent space region for compressing the code vectors to obtain a plurality of latent space vectors, wherein the latent space vectors correspond one-to-one to the code vectors;

[0043] A decoding module is used to input all latent space vectors into the improved decoder module for decoding respectively, obtain multiple decoding results, and generate animated images based on the multiple decoding results;

[0044] The improved decoder module includes a first convolution submodule for feature extraction, a second convolution submodule for performing convolution processing on the feature extraction result, and a third convolution submodule for limiting the generation range of the feature extraction result output by the second convolution submodule and accessing the feature extraction result using a static random access memory during the convolution process.

[0045] The first convolution submodule incorporates an auxiliary feature extraction layer, which is used to perform k 1-dimensional convolution operations on the input image data;

[0046] The third convolution submodule incorporates an adaptive hybrid algorithm and a reordering attention algorithm. The adaptive hybrid algorithm is used to control the generation range of the feature map by reducing the distance difference between the feature extraction results output by the second convolution submodule. The reordering attention algorithm is used to access the feature extraction results using static random access memory during the convolution process.

[0047] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements a method for generating cartoon images based on a variational autoencoder.

[0048] The present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a method for generating an animated image based on a variational autoencoder is implemented.

[0049] The above solution of the present invention has the following beneficial effects:

[0050] The present invention obtains multiple initial cartoon images to be processed by inputting them into an encoder module for encoding to obtain multiple coding vectors; inputs all coding vectors into a latent space region for conversion to obtain multiple latent space vectors; inputs all latent space vectors into an improved decoder module for decoding to obtain multiple decoding results, and generates cartoon images based on the multiple decoding results; compared with the prior art, the present invention provides an auxiliary feature extraction layer in the first convolution submodule of the decoder module without providing an additional data set, obtains the weight distribution of the channel dimension of the feature matrix, uses the weight distribution to perform matrix dot multiplication with the feature matrix after convolution through the conventional convolution layer, and obtains the feature matrix It can enhance the model's accurate decoding of image information and improve the quality of image detail information; the adaptive hybrid algorithm and the reordering attention algorithm are integrated into the third convolution submodule of the decoder module, and difficult samples are generated from the feature extraction results output by the previous convolution submodule through the adaptive hybrid algorithm. According to the linear combination ratio of the feature extraction results output by the previous convolution submodule in the difficult samples, the distance between the difficult samples and the input initial cartoon images in the feature space is reduced, thereby controlling the generation range of the feature map; the reordering attention algorithm uses static random access memory to access the feature extraction results during the convolution process, thereby improving the rate of generating cartoon images.

[0051] Other beneficial effects of the present invention will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 Schematic diagram of a framework of an embodiment of the present invention;

[0053] Figure 2 Schematic diagram of the structure of the upsampling convolution unit in the decoder module;

[0054] Figure 3 Schematic diagram of the structure of the auxiliary feature extraction layer. DETAILED DESCRIPTION

[0055] To make the technical problems, technical solutions, and advantages to be solved by the present invention more clear, the following is a detailed description with reference to the accompanying drawings and specific embodiments. It is obvious that the embodiments described are only some of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0056] In the description of the present invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They are not intended to indicate or imply that the devices or components referred to must have, be constructed, or operate in a specific orientation, and therefore should not be construed as limitations on the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0057] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood broadly. For example, they may refer to a locking connection, a detachable connection, or an integral connection; they may refer to a mechanical connection or an electrical connection; they may refer to a direct connection or an indirect connection through an intermediate medium; and they may refer to internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0058] In addition, the technical features involved in different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0059] In response to the existing problems, the present invention provides a method for generating cartoon images based on a variational autoencoder and related equipment.

[0060] like Figure 1 As shown, an embodiment of the present invention provides a method for generating an animated image based on a variational autoencoder, wherein the variational autoencoder includes an encoder module and an improved decoder module, and the method for generating an animated image includes:

[0061] Step 1: Obtain multiple initial animation images to be processed;

[0062] Step 2: Input all the initial cartoon images to be processed into the encoder module for encoding to obtain multiple image coding vectors, where the image coding vectors correspond one-to-one to the initial cartoon images to be processed;

[0063] Step 3: All code vectors are input into a latent space region for compressing the code vectors to convert them, thereby obtaining a plurality of latent space vectors, where the latent space vectors correspond one-to-one to the code vectors.

[0064] Step 4: Input all latent space vectors into the improved decoder module for decoding respectively to obtain multiple decoding results, and generate animated images based on the multiple decoding results;

[0065] The improved decoder module includes a first convolution submodule for feature extraction, a second convolution submodule for performing convolution processing on the feature extraction result, and a third convolution submodule for limiting the generation range of the feature extraction result output by the second convolution submodule and accessing the feature extraction result using a static random access memory during the convolution process.

[0066] The first convolution submodule incorporates an auxiliary feature extraction layer, which is used to perform k 1-dimensional convolution operations on the input image data;

[0067] The third convolution submodule incorporates an adaptive hybrid algorithm and a reordering attention algorithm. The adaptive hybrid algorithm is used to control the generation range of the feature map by reducing the distance difference between the feature extraction results output by the second convolution submodule. The reordering attention algorithm is used to access the feature extraction results using static random access memory during the convolution process.

[0068] It should be noted that the encoder module of the variational autoencoder in the embodiment of the present invention is a conventional encoder. The embodiment of the present invention does not involve any improvement to its internal structure. Encoding the image through the encoder is also a conventional technology in this field. Therefore, the embodiment of the present invention will no longer describe its structure and principle one by one.

[0069] In an embodiment of the present invention, a picture of a bear and a tiger is used as the initial cartoon picture input to the encoder module. After the above encoding, conversion, and decoding processes, multiple decoding results are obtained, and then a cartoon image is generated based on the decoding results, which is an image combining the bodies of a bear and a tiger.

[0070] Most preferably, Figure 2 As shown, the first convolution submodule includes a 16-dimensional convolution unit, a 32-dimensional convolution unit, and a 64-dimensional convolution unit connected in sequence, and the 16-dimensional convolution unit, the 32-dimensional convolution unit, and the 64-dimensional convolution unit are all provided with an auxiliary feature extraction layer;

[0071] The input of the auxiliary feature extraction layer in the 16-dimensional convolution unit and the input of the 16-dimensional convolution unit are both connected to the output of the latent space region, and the output of the auxiliary feature extraction layer in the 16-dimensional convolution unit is connected to the input of the 32-dimensional convolution unit;

[0072] The input end of the auxiliary feature extraction layer in the 32-dimensional convolution unit is connected to the output end of the 16-dimensional convolution unit, and the output end of the auxiliary feature extraction layer in the 32-dimensional convolution unit is connected to the input end of the 64-dimensional convolution unit;

[0073] The input end of the auxiliary feature extraction layer in the 64-dimensional convolution unit is connected to the output end of the 32-dimensional convolution unit, and the output end of the auxiliary feature extraction layer in the 64-dimensional convolution unit and the output end of the 64-dimensional convolution unit are both connected to the input end of the second convolution sub-module.

[0074] In the embodiment of the present invention, Figure 3 As shown, the auxiliary feature extraction layer includes:

[0075] Global average pooling (GAP), 1D convolutional sublayer, and splicing sublayer;

[0076] The input of the global average pooling sublayer is the input of the auxiliary feature extraction layer, and the output of the global average pooling sublayer is connected to the input of the 1D convolution sublayer. The output of the 1D convolution sublayer and the input of the global average pooling sublayer are both connected to the input of the splicing sublayer, and the output of the splicing sublayer is the output of the auxiliary feature extraction layer.

[0077] The core idea of the auxiliary feature extraction layer ECA-block is to cancel the dimensionality reduction operation in the traditional SE-block and perform only k 1-dimensional convolution operations each time, where the value of k is a nonlinear relationship mapped to the number of unit channels; the purpose is to enable the network to have good feature extraction capabilities without increasing the number of parameters.

[0078] Specifically, the auxiliary feature extraction layer only performs k 1-dimensional convolution operations each time, and the calculation formula for k is:

[0079]

[0080] Among them, C represents the number of channels of the auxiliary feature extraction layer, γ and b are both hyperparameters. Through a large number of network training experiments, it is found that γ = 2 and b = 0.5. k represents the number of one-dimensional convolutions, which is also the goal to be achieved in the embodiment of the present invention. The purpose is to obtain the optimal information extraction result with the minimum computational cost.

[0081] Most preferably, Figure 2 As shown, the second convolution submodule is used to perform further convolution processing on the feature extraction result output by the first convolution submodule, including:

[0082] The 128-dimensional convolutional unit, 256-dimensional convolutional unit, and 384-dimensional convolutional unit are connected in sequence.

[0083] Most preferably, Figure 2 As shown in Figure 2, the third convolution submodule is used to reconstruct and restore the detailed information of the image, including:

[0084] 512-dimensional convolutional unit, 768-dimensional convolutional unit, and 1024-dimensional convolutional unit connected in sequence;

[0085] The input end of the 512-dimensional convolution unit is connected to the output end of the second convolution submodule;

[0086] The output end of the 1024-dimensional convolution unit is the output end of the third convolution submodule, which is used to output the decoding result;

[0087] In order to control the stability of the network when reconstructing detailed information and improve the decoding rate, the adaptive mixing algorithm Adaptive Mix and the reordering attention algorithm flash-attention are integrated into the 512-dimensional convolution unit, 768-dimensional convolution unit, and 1024-dimensional convolution unit.

[0088] In the embodiment of the present invention, the core idea of the adaptive mixing algorithm is to dynamically reduce the distance between the training samples and their linear combinations. Adaptive Mix can shrink the area of training data in the feature space, thereby achieving stable training of the network and improving the quality of generated samples.

[0089] Specifically, the adaptive hybrid algorithm is used to:

[0090] Using the formula of the mixup algorithm Generate difficult samples from the feature extraction results output by the previous level convolution unit (e.g. 384-dimensional convolution unit) Among them, g(·) represents the linear combination function, x i Represents the feature extraction result output by the previous convolution unit, x j represents the decoding result, λ represents the hyperparameter sampled from the data distribution, Indicates difficult samples;

[0091] According to the linear combination ratio of the feature extraction results output by the previous convolution unit in the difficult sample, the distance between the difficult sample and the feature extraction results output by the previous convolution unit in the feature space is reduced. The calculation formula is:

[0092]

[0093] where σ represents the noise term sampled from the Gaussian distribution of the data, represents the metric function, F(·) represents the information extractor in the model, and L add The function is used to control the improved decoder module to generate images close to the original training sample images, avoiding the irregular diffusion of the generation range of the decoder module.

[0094] It should be noted that the mixup algorithm is essentially a data enhancement method that does not rely on adding new data. New training samples can be obtained by linear interpolation of existing training samples and corresponding labels. The linear interpolation expression is:

[0095]

[0096] in, Both represent linear interpolation results, the value range of λ is [0, 1], x i Represents the input sample of the current network, y i Indicates the label value of the current network input sample, x j represents the sample generated by the decoder, y j Indicates the label value of the sample generated by the decoder.

[0097] The embodiment of the present invention adopts L2 evaluation norm, function L ada The distance difference between the difficult sample and the training sample is calculated by measuring the evaluation function, and the loss is optimized during continuous training. The best expected loss is:

[0098]

[0099] The embodiment of the present invention controls the generation range of the feature map extracted by the information extractor through an adaptive hybrid algorithm, thereby reducing the absolute distance between samples and achieving precise control of the details of the generated image.

[0100] Specifically, the core idea of the reordering attention algorithm flash-attention is to perform block calculations on the self-attention mechanism. The self-attention calculation method is to store and retrieve data in the High Bandwidth Memory (HBM). Although HBM memory has a large storage scale, its throughput is only 1.5TB / s. The reordering attention algorithm instead uses the feature extraction results in the Static Random Access Memory (SRAM) for access operations, with a throughput of 19TB / s.

[0101] Specifically, the reordering attention algorithm that uses static random access memory to access the feature extraction results during the convolution process is specifically used to:

[0102] Obtaining the size of a preset input and output matrix in a static random access memory for storing feature extraction results;

[0103] Get the Q matrix, K matrix, and V matrix from the feature extraction results output by the previous level convolution unit, and split the Q matrix, K matrix, and V matrix into the same size as the input and output matrices;

[0104] The split Q matrix, K matrix, and V matrix are calculated in the static random access memory to obtain a Q result matrix, a K result matrix, and a V result matrix;

[0105] The Q result matrix, K result matrix, and V result matrix are concatenated and input into the next level convolution unit.

[0106] In an embodiment of the present invention, if the reordering attention algorithm is the reordering attention algorithm in a 512-dimensional convolution unit, the previous level convolution unit is a 384-dimensional convolution unit; if the reordering attention algorithm is the reordering attention algorithm in a 768-dimensional convolution unit, the previous level convolution unit is a 512-dimensional convolution unit; if the reordering attention algorithm is the reordering attention algorithm in a 1024-dimensional convolution unit, the previous level convolution unit is a 768-dimensional convolution unit.

[0107] The embodiment of the present invention obtains multiple encoding vectors by inputting multiple initial cartoon images to be processed into the encoder module for encoding; inputting all the encoding vectors into the latent space area for conversion to obtain multiple latent space vectors; inputting all the latent space vectors into the improved decoder module for decoding to obtain multiple decoding results, and generating cartoon images based on the multiple decoding results; compared with the prior art, the embodiment of the present invention provides an auxiliary feature extraction layer in the first convolution submodule of the decoder module without providing an additional data set, obtains the weight distribution of the channel dimension of the feature matrix, and uses the weight distribution to perform matrix dot multiplication with the feature matrix after convolution through the conventional convolution layer to obtain The feature matrix can enhance the model's accurate decoding of image information and improve the quality of image detail information; the adaptive hybrid algorithm and the reordering attention algorithm are integrated into the third convolution submodule of the decoder module. The adaptive hybrid algorithm is used to generate difficult samples from the feature extraction results output by the previous convolution submodule. According to the linear combination ratio of the feature extraction results output by the previous convolution submodule in the difficult samples, the distance between the difficult samples and the input initial cartoon image in the feature space is reduced, thereby controlling the generation range of the feature map; the reordering attention algorithm is used to access the feature extraction results using static random access memory during the convolution process, thereby improving the rate of generating cartoon images.

[0108] An embodiment of the present invention further provides a device for generating an animated image based on a variational autoencoder, wherein the variational autoencoder includes an encoder module and an improved decoder module, and the device for generating an animated image includes:

[0109] An acquisition module is used to obtain multiple initial animation pictures to be processed;

[0110] An encoding module is used to input all the initial cartoon images to be processed into an encoder module for encoding to obtain multiple image encoding vectors, and the image encoding vectors correspond one-to-one to the initial cartoon images to be processed;

[0111] a conversion module, configured to convert all code vectors input into a latent space region for compressing the code vectors to obtain a plurality of latent space vectors, wherein the latent space vectors correspond one-to-one to the code vectors;

[0112] A decoding module is used to input all latent space vectors into the improved decoder module for decoding respectively, obtain multiple decoding results, and generate animated images based on the multiple decoding results;

[0113] The improved decoder module includes a first convolution submodule for feature extraction, a second convolution submodule for performing convolution processing on the feature extraction result, and a third convolution submodule for limiting the generation range of the feature extraction result output by the second convolution submodule and accessing the feature extraction result using a static random access memory during the convolution process.

[0114] The first convolution submodule incorporates an auxiliary feature extraction layer, which is used to perform k 1-dimensional convolution operations on the input image data;

[0115] The third convolution submodule incorporates an adaptive hybrid algorithm and a reordering attention algorithm. The adaptive hybrid algorithm is used to control the generation range of the feature map by reducing the distance difference between the feature extraction results output by the second convolution submodule. The reordering attention algorithm is used to access the feature extraction results using static random access memory during the convolution process.

[0116] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiments of the embodiments of the present invention. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0117] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the embodiments of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0118] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, a method for generating an animated image based on a variational autoencoder is implemented.

[0119] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the embodiments of the present invention implement all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device capable of carrying computer program code to a construction device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0120] An embodiment of the present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a method for generating an animated image based on a variational autoencoder is implemented.

[0121] It should be noted that the terminal device may be a mobile phone, tablet computer, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), and other terminal devices. For example, the terminal device may be a station (ST, STAION) in a WLAN, a cellular phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a wireless local loop (WLL) station, a personal digital assistant (PDA), a handheld device with wireless communication capabilities, a computing device or other processing device connected to a wireless modem, a computer, a laptop computer, a handheld communication device, a handheld computing device, a satellite wireless device, etc. The embodiments of the present invention do not impose any restrictions on the specific type of the terminal device.

[0122] The processor may be a central processing unit (CPU), other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0123] In some embodiments, the memory may be an internal storage unit of the terminal device, such as a hard disk or memory of the terminal device. In other embodiments, the memory may also be an external storage device of the terminal device, such as a plug-in hard disk equipped on the terminal device, a smart memory card (SMC, Smart Media Card), a secure digital (SD, Secure Digital) card, a flash card, etc. Furthermore, the memory may include both an internal storage unit of the terminal device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory may also be used to temporarily store data that has been output or is to be output.

[0124] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiments of the embodiments of the present invention. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0125] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A method for generating cartoon images based on variational autoencoders, characterized in that: The variational autoencoder includes an encoder module and an improved decoder module; The method for generating an animated image comprises: Step 1: Obtain multiple initial animation images to be processed; Step 2: input all the initial cartoon images to be processed into the encoder module for encoding to obtain multiple encoding vectors, each of which corresponds to the initial cartoon images to be processed; Step 3: All code vectors are input into a latent space region for compressing the code vectors to convert them, thereby obtaining a plurality of latent space vectors, wherein the latent space vectors correspond one-to-one to the code vectors; Step 4: inputting all latent space vectors into the improved decoder module for decoding respectively to obtain multiple decoding results, and generating an animated image based on the multiple decoding results; The improved decoder module includes a first convolution submodule for feature extraction, a second convolution submodule for performing convolution processing on the feature extraction result, and a third convolution submodule for limiting the generation range of the feature extraction result output by the second convolution submodule and accessing the feature extraction result using a static random access memory during the convolution process. The first convolution submodule incorporates an auxiliary feature extraction layer for performing k one-dimensional convolution operations on the input image data; The third convolution submodule incorporates an adaptive hybrid algorithm and a reordering attention algorithm. The adaptive hybrid algorithm is used to control the generation range of the feature map by reducing the distance difference between the feature extraction results output by the second convolution submodule. The reordering attention algorithm is used to use static random access memory to access the feature extraction results during the convolution process.

2. The method for generating cartoon images based on variational autoencoders according to claim 1, characterized in that: The first convolution submodule includes a 16-dimensional convolution unit, a 32-dimensional convolution unit, and a 64-dimensional convolution unit connected in sequence, and the 16-dimensional convolution unit, the 32-dimensional convolution unit, and the 64-dimensional convolution unit are all provided with an auxiliary feature extraction layer; The input end of the auxiliary feature extraction layer in the 16-dimensional convolution unit and the input end of the 16-dimensional convolution unit are both connected to the output end of the latent space region, and the output end of the auxiliary feature extraction layer in the 16-dimensional convolution unit is connected to the input end of the 32-dimensional convolution unit; The input end of the auxiliary feature extraction layer in the 32-dimensional convolution unit is connected to the output end of the 16-dimensional convolution unit, and the output end of the auxiliary feature extraction layer in the 32-dimensional convolution unit is connected to the input end of the 64-dimensional convolution unit; The input end of the auxiliary feature extraction layer in the 64-dimensional convolution unit is connected to the output end of the 32-dimensional convolution unit, and the output end of the auxiliary feature extraction layer in the 64-dimensional convolution unit and the output end of the 64-dimensional convolution unit are both connected to the input end of the second convolution sub-module.

3. The method for generating cartoon images based on variational autoencoders according to claim 2, characterized in that: The auxiliary feature extraction layer includes: Global average pooling sublayer, 1D convolution sublayer, and splicing sublayer; The input end of the global average pooling sublayer is the input end of the auxiliary feature extraction layer, the output end of the global average pooling sublayer is connected to the input end of the 1D convolution sublayer, the output end of the 1D convolution sublayer and the input end of the global average pooling sublayer are both connected to the input end of the splicing sublayer, and the output end of the splicing sublayer is the output end of the auxiliary feature extraction layer; The auxiliary feature extraction layer only performs k 1-dimensional convolution operations each time, and the calculation formula of k is: Among them, C represents the number of channels of the auxiliary feature extraction layer, γ and b are hyperparameters, and k represents the number of one-dimensional convolutions.

4. The method for generating cartoon images based on a variational autoencoder according to claim 3, wherein: The third convolution submodule includes: 512-dimensional convolutional unit, 768-dimensional convolutional unit, and 1024-dimensional convolutional unit connected in sequence; The input end of the 512-dimensional convolution unit is connected to the output end of the second convolution submodule; The output end of the 1024-dimensional convolution unit is the output end of the third convolution submodule; The 512-dimensional convolution unit, the 768-dimensional convolution unit, and the 1024-dimensional convolution unit are all integrated with an adaptive hybrid algorithm and a reordering attention algorithm.

5. The method for generating cartoon images based on variational autoencoders according to claim 4, characterized in that: The adaptive hybrid algorithm is specifically used for: Using the formula Generate difficult samples from the feature extraction results output by the previous level convolution unit Among them, g(·) represents the linear combination function, x i Represents the feature extraction result output by the previous convolution unit, x j represents the decoding result, λ represents the hyperparameter sampled from the data distribution, Indicates difficult samples; According to the linear combination ratio of the feature extraction result output by the previous convolution unit in the difficult sample, the distance between the difficult sample and the feature extraction result output by the previous convolution unit in the feature space is reduced. The calculation formula is: Where v represents the noise term sampled from the Gaussian distribution of the data, represents the metric function, and F(·) represents the information extractor in the model.

6. The method for generating cartoon images based on variational autoencoders according to claim 5, characterized in that: The reordering attention algorithm that uses static random access memory to access the feature extraction results during the convolution process is specifically used to: Obtaining the size of a preset input and output matrix in a static random access memory for storing feature extraction results; Obtain the Q matrix, K matrix, and V matrix from the feature extraction results output by the previous level convolution unit, and split the Q matrix, K matrix, and V matrix into the same size as the input and output matrices; Calculating the split Q matrix, K matrix, and V matrix in the static random access memory to obtain a Q result matrix, a K result matrix, and a V result matrix; The Q result matrix, the K result matrix, and the V result matrix are concatenated and input into the next level convolution unit.

7. A device for generating cartoon images based on a variational autoencoder, characterized in that: The variational autoencoder includes an encoder module and an improved decoder module; The animation image generating device includes: An acquisition module is used to obtain multiple initial animation pictures to be processed; An encoding module, configured to input all the initial cartoon images to be processed into the encoder module for encoding to obtain a plurality of encoding vectors, wherein the encoding vectors correspond one-to-one to the initial cartoon images to be processed; a conversion module, configured to convert all code vectors input into a latent space region for compressing the code vectors to obtain a plurality of latent space vectors, wherein the latent space vectors correspond one-to-one to the code vectors; A decoding module, configured to input all latent space vectors into the improved decoder module for decoding respectively, obtain multiple decoding results, and generate an animated image based on the multiple decoding results; The improved decoder module includes a first convolution submodule for feature extraction, a second convolution submodule for performing convolution processing on the feature extraction result, and a third convolution submodule for limiting the generation range of the feature extraction result output by the second convolution submodule and accessing the feature extraction result using a static random access memory during the convolution process. The first convolution submodule incorporates an auxiliary feature extraction layer for performing k one-dimensional convolution operations on the input image data; The third convolution submodule incorporates an adaptive hybrid algorithm and a reordering attention algorithm. The adaptive hybrid algorithm is used to control the generation range of the feature map by reducing the distance difference between the feature extraction results output by the second convolution submodule. The reordering attention algorithm is used to use static random access memory to access the feature extraction results during the convolution process.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for generating an animated image based on a variational autoencoder according to any one of claims 1 to 6 is implemented.

9. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for generating an animated image based on a variational autoencoder according to any one of claims 1 to 6 is implemented.

Images