Image animation processing method, device, intelligent device and storage medium
Through the combination of deep learning and image segmentation algorithm, the image animation processing is performed using encoding networks and decoding networks, solving the problem of personalized animation effects in different regions, realizing high-quality animation image generation, and reducing training costs and sample collection difficulty.
Patent Information
- Application Number
- CN202011228595.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-11-05
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2040-11-05
AI Technical Summary
The prior art is difficult to achieve personalized animation effects of different image regions in image animation processing, and samples need to be paired during model training, which increases sample collection and training costs.
Deep learning method combined with image segmentation algorithm is adopted to introduce semantic segmentation mask, and image animation is performed through encoding network and decoding network generator. Unsupervised training scheme and multi-loss function constraint generator are used to realize animation processing in different regions. And adjust the image style through anime style vector.
The animation processing effect of different areas in the image is personalized, reducing the cost of model training and the difficulty of sample collection. The generated animation images are highly similar to the original image and are coordinated and natural.
Smart Images

Figure CN113393545B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to an image animation processing method, device, intelligent device and storage medium. Background Art
[0002] Image animation refers to converting a real image (such as a photo) into an animated image. Currently, the automatic conversion of real images into animated images based on computers is realized. Specifically, the trained model is used to animate the entire image, that is, the model for image animation is trained, and the original image is processed by the model to obtain the animated image. How to better use the model to animate images has become a hot research issue. Summary of the invention
[0003] The embodiments of the present invention provide an image animation processing method, apparatus, intelligent device and storage medium, which can combine masks, image features and style features of different image regions to obtain better animation effects.
[0004] In one aspect, an embodiment of the present invention provides a method for processing image animation, the method comprising:
[0005] Get the target image to be animated;
[0006] Performing segmentation processing on the target image, and determining a reference mask according to the segmentation processing result, wherein the target image is segmented to obtain at least a first region and a second region, and mask values of the first region and the second region on the reference mask are different;
[0007] Determining encoding features of the target image through an encoding network;
[0008] The encoding features determined by the encoding network, the reference mask and the determined animation style vector are processed by a decoding network to obtain an animated image corresponding to the target image.
[0009] In one aspect, an embodiment of the present invention provides an image animation processing device, the device comprising:
[0010] An acquisition module is used to acquire a target image to be processed into an animation;
[0011] A processing module, used for segmenting the target image;
[0012] A determination module, configured to determine a reference mask according to a segmentation processing result, wherein the target image is segmented to obtain at least a first region and a second region, and mask values of the first region and the second region on the reference mask are different;
[0013] The determination module is further used to determine the coding features of the target image through a coding network;
[0014] The determination module is further used to process the encoding features determined by the encoding network, the reference mask and the determined animation style vector through a decoding network to obtain an animated image corresponding to the target image.
[0015] On the one hand, an embodiment of the present invention provides an intelligent device, which includes a processor and a memory, wherein the memory stores a computer program, and the processor runs the computer program to implement the above-mentioned image animation processing method.
[0016] On the one hand, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the image animation processing method.
[0017] In an embodiment of the present invention, an intelligent device obtains a target image to be animated, and after segmenting the target image, a reference mask with different mask values can be determined, and two models are set up. The encoding features of the target image, that is, the features of the image itself, are determined through an encoding network; and then the encoding features, reference masks and determined animation style vectors determined by the encoding network are processed through a decoding network, so that different areas have different animation effects and corresponding animation styles, making the image animation effect richer. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.
[0019] Figure 1a It is a schematic diagram of a model structure of image animation processing provided by an embodiment of the present invention;
[0020] Figure 1b is a schematic diagram of a model structure of a generator according to an embodiment of the present invention;
[0021] Figure 2 is a flowchart of another method for processing image animation provided by an embodiment of the present invention;
[0022] Figure 3 is a schematic diagram of another image animation process processing method provided by an embodiment of the present invention;
[0023] Figure 4 is a schematic diagram of an image animation result provided by an embodiment of the present invention;
[0024] Figure 5 It is a flow chart of a model training method provided by an embodiment of the present invention;
[0025] Figure 6 is a schematic diagram of a network structure during model training provided by an embodiment of the present invention;
[0026] Figure 7 is a schematic diagram of a network structure during another model training provided by an embodiment of the present invention;
[0027] Figure 8 This is a schematic diagram of an image animation scene provided by an embodiment of the present invention;
[0028] Fig. 9 is a structural schematic diagram of an image animation processing device provided by an embodiment of the present invention;
[0029] Fig.10 It is a structural schematic diagram of a smart device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0030] The computer technology field involved in this solution specifically includes the field of artificial intelligence technology, among which artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline that involves a wide range of fields, including both hardware-level technology and software-level technology. Basic artificial intelligence technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. Artificial intelligence software technology mainly includes several major directions such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning. This solution specifically applies deep learning to the field of image analysis. The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0031] Image animation refers to converting a real image (such as a photo) into an animated image. In some application scenarios, only part of the image area can be greatly deformed, while other areas do not change much. For example, in a portrait animation scene, the face area can be greatly deformed, while other parts of the character and the background remain consistent in content or slightly deformed, or only the style is changed (maintaining the same animation style to avoid giving users an abrupt sensory experience), and the facial expression of the character remains consistent. This application proposes a method for automatically generating animated images by computer for the above application scenarios or other application scenarios. The model constructed based on deep learning and other methods and the image segmentation algorithm, and then adding animation style vectors, can generate animated images with better animation effects.
[0032] There are two ways to use models to animate images: supervised and unsupervised. Using supervised methods to train models to generate animated images can be very convenient in practical applications. In the process of training the model, a large number of paired samples (i.e., original images and animated images of corresponding styles) are used to train the model. Using unsupervised methods is relatively simple. You only need to collect real images and anime images of a specific style. There is no need for the two to have a paired relationship. You can just put them into the corresponding training framework for training.
[0033] The image animation processing method proposed in this scheme belongs to the unsupervised training scheme under the category of deep learning. It is easy to obtain samples and the cost is very low. In terms of the structure of the model, a semantic segmentation mask is introduced to control the fusion of features at all levels in the generator, and constraints on the image are added based on multiple loss functions during the training process. The generator obtained by this scheme can generate high-quality animated images in the task of image animation. It can not only complete large deformations in specific areas (such as facial areas) according to the specified style to generate beautiful animated areas, but also maintain the content in other areas of the original image without major changes, so that the converted animated image is coordinated and natural, and maintains a high degree of consistency with the original image in content.
[0034] In this solution, the image animation process can be specifically executed by the generator in the smart device, which includes an encoding network and a decoding network, that is, the generator is an encoder-decoder structure. Figure 1a FIG. 1 is a schematic diagram of a model structure of image animation processing provided by an embodiment of the present invention. Figure 1a In the embodiment of the present invention, the animation processing flow under this model structure includes: ① obtaining a target image, specifically obtaining a target image to be animated. ② segmenting the target image to obtain a reference mask of the target image, specifically, segmenting the target image through a trained segmentation model to obtain a reference mask of the target image, wherein at least a first region and a second region are obtained after the target image is segmented, and the mask values of the first region and the second region on the reference mask are different. Of course, more regions can be obtained, such as a third region, a fourth region, etc. In the embodiment of the present invention, the first region can be, for example, a face image region, and the second region can be, for example, an image region other than a face. The reference mask with different mask values is used to make the animation degree of the face image region different from that of the image region other than a face. ③ Processing the image through the encoding network in the generator to obtain the encoding features of the target image. ④ Processing the encoding features determined by the encoding network, the reference mask and the determined animation style vector through the decoding network to obtain an animated image corresponding to the target image.
[0035] The encoding network and the decoding network can be symmetrical structures, that is, the spatial size of the features of the i-th layer of the encoding network and the features of the i-th layer from the end of the decoding network are the same. Figure 1b FIG. 1 is a schematic diagram of a structure of a generator provided by an embodiment of the present invention. Figure 1b In the generator, the encoding network and the decoding network are symmetrical structures, that is, the spatial dimensions of the features of the i-th layer of the encoding network and the features of the i-th layer from the end of the decoding network are the same. For example, in the image processing process, the dimension of the features extracted by the network is generally c*h*w, where c represents the number of channels of the feature, h represents the height of the feature, and w represents the width of the feature. Then, the features of the first layer of the encoding network are 64×128×128 features, and the features of the first-last layer of the decoding network are also 64×128×128 features. In the process of fusing the weighted coding features and the initial decoding features, the features of the first layer of 64×128×128 in the weighted coding features can be fused with the features of the first-last layer of 64×128×128 in the initial decoding features. The fusion processing of the weighted coding features and the initial decoding features is realized by correspondingly fusing the features of the i-th layer in the weighted coding features and the features of the i-th layer from the end of the initial decoding features through the decoding network. In the decoding network, the style of the generated image is controlled by inserting an anime style code, which is inserted through adaptive instance normalization. In the specific implementation, the decoding network also receives a reference mask obtained after segmenting the target image, which is used to control the fusion of features at different levels.
[0036] Among the above schemes, compared with the current schemes for image animation, this scheme has the following advantages: (1) Different regions in the image are animated to different degrees. Specifically, after the mask of the image is determined based on image segmentation technology, the mask is added to the model for calculation to achieve different animation processing for different regions in the image. (2) The training process is simple. In the process of preparing samples, this scheme does not require a pairing relationship between samples. The model training can be completed by inserting the corresponding training framework. (3) The animation processing result is highly similar to the original image. This scheme realizes a large degree of animation of some areas in the image based on the mask value, while some areas remain basically unchanged. Only some coordinated animation changes are made based on the animated area, so that the animation processing result is highly similar to the original image.
[0037] See also Figure 2, is a flow chart of an image animation process provided by an embodiment of the present invention. The method of the embodiment of the present invention can be executed by an intelligent device for image animation processing, the intelligent device can be a server or a terminal, the terminal includes a smart phone, a tablet computer, a digital audio and video player, an electronic reader, etc., a generator is deployed in the intelligent device, and the generator includes an encoding network and a decoding network. The intelligent device performs animation processing on the image based on the generator. The image animation processing method can include the following steps.
[0038] S201, obtaining a target image to be animated. In the embodiment of the present invention, the target image to be animated may be a real image, such as a photo, a video screenshot, etc. In a portrait animation scene, the target image may be a person image, i.e., a real image containing a person area, specifically a person photo, a person screenshot in a video, etc.
[0039] In one embodiment, the smart device may be a server, and the smart device receives a target image sent online by the terminal. The target image may be uploaded by the user in a designated page of the terminal and obtained by the terminal. The designated page may be an interactive page for image animation provided by the terminal, such as a website page for uploading images to be animated. The user may select an image from the terminal's gallery and upload it to the page. After the terminal detects the user's upload operation, it will send the image uploaded by the user to the smart device. The smart device collects the image sent by the terminal as the target image to be animated. It is understandable that if the terminal uploads only one image, the target image to be animated is the image uploaded by the user; if the terminal uploads multiple images, the target image to be animated is any one of the multiple images.
[0040] In one embodiment, the smart device collects the target image to be animated from the gallery offline, wherein the gallery stores at least one image to be animated, and the image in the gallery can be uploaded by the user through the terminal, or automatically captured from the video acquired by the shooting device. It should be noted that in the portrait animation scene, after the smart device acquires the image, it can identify the image and determine whether the image contains facial features based on the recognition result. If so, it is determined that the acquired image meets the conditions and the image is stored in the gallery. If not, feedback information can be sent to the terminal, and the feedback information is used to indicate that the image is incorrect, so that the user can re-upload the image through the terminal.
[0041] S202, segment the target image and determine a reference mask according to the segmentation result. In the embodiment of the present invention, the target image is segmented to obtain at least a first region and a second region, and the mask values of the first region and the second region on the reference mask are different. In a specific implementation, after the smart device obtains the target image, it can perform segmentation processing on the target image to obtain a segmentation processing result, wherein the segmentation processing result includes the first region and the second region obtained by the image segmentation processing, and the mask values corresponding to the first region and the second region respectively. The smart device determines the reference mask corresponding to the target image based on the mask values of each region, wherein the mask value ranges from 0 to 1, the mask value of the first region can be 1, and the mask value of the second region can be 0. The smart device can specifically call a segmentation model to perform segmentation processing on the target image. The segmentation model (segmentation) realizes fine-grained reasoning by densely predicting and inferring labels for each region in the image, thereby obtaining the category to which each region belongs and the mask value corresponding to the category. The mask value can be used to determine the degree of animation of the corresponding image region. A region can be a pixel point in the target image or a collection of multiple pixels in the image. After the target image is segmented, other regions except the first region and the second region can also be obtained. The embodiment of the present invention is specifically described with reference to the first region and the second region.
[0042] It should be noted that the specific training process of the segmentation model may include: obtaining a training image set, the training image set includes at least one training image, each training image is annotated with multiple sub-image areas and each sub-image area is annotated with a preset mask value, and each image area in the training image is an area containing a type of object; iteratively training the initial segmentation model based on the training image set to update the parameters in the initial segmentation model; when it is detected that the segmentation accuracy of the initial segmentation model after the parameter update for the image in the training image set is higher than the preset accuracy, the initial segmentation model after the parameter update is determined as the segmentation model; wherein, for any target training image in the training image set, the initial segmentation model after the parameter update is called to process the target training image to obtain multiple predicted image areas corresponding to the target training image and the preset mask value corresponding to each predicted image area, if the multiple predicted image areas match the multiple target sub-image areas annotated in the target training image, and the preset mask value corresponding to each predicted image area matches the preset mask value annotated with the corresponding target sub-image area, it is determined that the initial segmentation model after the parameter update is accurate for the segmentation of the target training image.
[0043] S203, determining the coding features of the target image through the coding network. In the embodiment of the present invention, after the intelligent device determines the reference mask according to the segmentation processing result, the coding features of the target image can be determined through the coding network, wherein the coding features of the target image include the first coding features of the first area in the target image and the second coding features of the second area in the target image. Specifically, the intelligent device can process the target image through the coding network to obtain the first coding features of the first area and the second coding features of the second area, and the coding network has the function of extracting image features.
[0044] In one embodiment, the specific method for the smart device to determine the coding features of the target image through the coding network can be that the smart device processes the target image through the coding network to obtain the initial coding features of the target image; determines the filtering range for the initial coding features based on the determined animation style vector, and performs filtering processing on the initial coding features according to the filtering range to obtain the coding features of the target image. The initial coding features include a first initial coding feature of a first area in the target image and a second initial coding feature of a second area in the target image. The coding features of the target image can be obtained by filtering and merging the first initial coding features and the second initial coding features.
[0045] In the above method, since the cartoon image is generally smoother, while the real character image is angular, the eigenvalue corresponding to the angular area is usually larger or smaller. By filtering the initial coding features of the intelligent device, the values in the initial coding features can be made similar, so that the animated image decoded based on the initial coding features is smoother, which solves the problem of image incoordination caused by different animation degrees of different image areas in the generated animated image. By determining the filtering range based on the determined animation style vector, different filters can be used for different styles to obtain different filtering effects. For example, when the animation image corresponding to the animation style vector is angular, the filter filtering range can be increased so that more encoder features are retained. When the animation image corresponding to the animation style vector is relatively smooth, the filter range can be reduced so that the encoder features after filtering are more similar. Through the above method, the adjustment of the smoothness of the image can be adjusted based on the animation style vector to improve the intelligence of image animation.
[0046] It should be noted that the animation style vector is used to determine the style of the animation image generated based on the target image, and may be a vector corresponding to the style feature of the image.
[0047] In one implementation, the determined animation style vector can be specifically selected by the user in a designated page in the terminal and sent to the smart device through the terminal. The designated page can be an interactive page for image animation provided by the terminal, for example, a website page for uploading an image to be animated. The user can select a style from multiple style options provided in the page and send it to the smart device through the terminal. The smart device obtains the animation style vector corresponding to the style selected by the user as the determined animation style vector. Alternatively, the user can also input a reference animation image in the interactive page, that is, the style of the reference animation image is the style that the user wants the image to be animated. The terminal sends the input information of the reference animation image to the smart device, and the smart device receives the input information for the reference animation image, and calls the style encoder to encode the reference animation image to obtain the reference style vector of the reference animation image; the smart device determines the target style vector that matches the reference style vector from the database as the determined animation style vector.
[0048] In one implementation, the determined animation style vector may specifically be a vector corresponding to an animation style that matches the style of the target image. Specifically, the smart device calls a style encoder to process the target image, obtains a first style vector, and determines a second style vector that matches the first style vector from a database as the determined animation style vector. Among them, the second style vector that matches the first style vector may be the same or similar to the first style vector. The above method may ensure that the style of the character image is not changed after the character image is animated. Optionally, a plurality of style vectors and matching relationships of the style vectors may be pre-stored in the database. After the smart device obtains the first style vector, it may find a matching second style vector from the database based on the matching relationship. Further, the smart device uses the above second style vector as the determined animation style vector. The above method may design different animation styles for target images of different styles, making the image animation process richer and more targeted. Among them, for the animation style vector corresponding to any target style stored in the database, the specific establishment method can be to obtain multiple animation images of the target style, and call the style encoder to process the multiple animation images of the target style to obtain multiple style vectors of the target style, and the smart device averages the multiple style vectors of the target style to obtain the animation style vector corresponding to the target style, and stores it in the database. Optionally, the database stores preset style vectors, and the smart device directly obtains the preset style vector from the database as the determined animation style vector.
[0049] S204, processing the coding features determined by the coding network, the reference mask and the determined animation style vector through the decoding network to obtain an animated image corresponding to the target image.
[0050] In an embodiment of the present invention, after the smart device obtains the coding features of the target image, it can process the coding features determined by the coding network, the reference mask and the determined animation style vector through the decoding network to obtain an animated image corresponding to the target image.
[0051] In the specific implementation, Figure 3 As shown, the process of the smart device processing the encoding features, reference masks and determined animation style vectors determined by the encoding network through the decoding network includes the following steps.
[0052] s31. Perform weighted processing on the coding feature based on each mask value in the reference mask through the decoding network to obtain a weighted coding feature. In a specific implementation, the reference mask includes a first mask value of a first area in the target image and a second mask value of a second area. The smart device performs weighted processing on the first coding feature based on the first mask value in the reference mask through the decoding network to obtain a first weighted coding feature, performs weighted processing on the second coding feature based on the second mask value in the reference mask through the decoding network to obtain a second weighted coding feature, and merges the first weighted coding feature and the second weighted coding feature to obtain a weighted coding feature.
[0053] s32. The determined animation style vector and the encoding feature are processed by a decoding network to obtain an initial decoding feature. In a specific implementation, the initial decoding feature may be a full-image animation feature corresponding to the target image, that is, after decoding the initial decoding feature, a fully animated target image may be obtained.
[0054] s33. The weighted coding features and the initial decoding features are fused through the decoding network to obtain the target decoding features of the target image. In a specific implementation, the coding network and the decoding network are pre-trained, the spatial dimensions between the features of the i-th layer of the coding network and the features of the i-th layer from the end of the decoding network are the same, the input of the coding network is the target image, the coding network can process the target image to obtain the multi-layer coding features corresponding to the target image, the decoding network can decode the multi-layer coding features to obtain the multi-layer decoding features, then the weighted coding features and the initial decoding features both include multi-layer features, and the method of fusing the weighted coding features and the initial decoding features through the decoding network includes fusing the i-th layer features in the weighted coding features and the i-th layer features from the end of the initial decoding features through the decoding network to achieve the fusion processing of the weighted coding features and the initial decoding features.
[0055] In one implementation scenario, this solution is used for portrait animation. The target image is a person image, the first area is a face image area, and the second area is a non-face image area. The smart device calls the trained segmentation model to segment the target image to obtain the face image area and the non-face image area in the target image, as well as the first mask value of the face image area and the second mask value of the non-face image area. The smart device determines the reference mask based on the first mask value of the face image area and the second mask value of the non-face image area. The mask value in the reference mask ranges from 0 to 1. The area with a mask value of 1 represents the face area, and the area with a mask value of 0 represents the non-face area. The portrait animation requires a greater degree of animation processing of the face area in the image. If the mask value of the image area is 1, it means that the area in the image needs to be animated.
[0056] After the smart device processes the target image through the encoding network, it can obtain the i-th layer feature E in the initial encoding feature E i , for the i-th layer E i The i-th layer coding feature H(E) of the coding feature H(E) of the target image after filtering i ), the i-th layer encoding feature H(E i ) is weighted to obtain the i-th layer feature (1-mask)*H(E) in the weighted coding feature (1-mask)*H(E) i ), that is, the coding features corresponding to the area with mask value 0 are retained, and the coding features corresponding to the area with mask value 1 are removed. The determined animation style vector and coding features are processed by the decoding network to obtain the penultimate i-th layer feature D in the initial decoding feature D i , then the i-th layer feature in the weighted coding feature and the i-th layer feature from the initial decoding feature are fused correspondingly through the decoding network to obtain the i-th layer feature M in the target decoding feature M i , M i It can be specifically expressed as M i =(1-mask)*H(E i )+D i In the above method, since the mask value of the coding feature corresponding to the face area in the target image is greater than 1, after weighting it with the weighting factor (1-mask), the face area feature of the target image in the target decoding feature will not be retained, but will be replaced by the initial decoding feature D i Determined, that is, the face area in the decoded animated image has a greater degree of animation. On the contrary, because the mask value of the encoding feature corresponding to the non-face area in the target image is greater than 0, the non-face area feature of the target image in the target decoding feature will be retained to the greatest extent, that is, the non-face area in the decoded animated image has a lower degree of animation.
[0057] The facial area of the target image is animated without changing the original style of the image, so that the animated image has a high similarity with the input target image.
[0058] s34. Decode the target decoder features through the decoding network to obtain an animated image corresponding to the target image.
[0059] In the above scheme, the initial decoding feature can be understood as the full-image animation feature corresponding to the target image. If the initial decoding feature is decoded, the fully animated target image can be obtained. After fusing it with the weighted coding feature, the animation degree of the target image can be reduced. The larger the weight of the weighted coding feature, the larger the proportion of the input target image feature in the target decoding feature, so that the generated animated image is closer to the input target image, that is, the animation degree is low. The smaller the weight in the weighted coding feature, the larger the proportion of the full-image animation feature in the target decoding feature, so that the generated animated image is closer to the animated image, that is, the animation degree is high. However, the corresponding weights of different regions in the image are determined by the mask value of the region, which realizes the adjustment of the animation degree of different regions in the image based on the mask value, so that the animation degree of different regions in the image is different, and the accuracy of the image animation result is improved. Among them, the determined animation style vector can be inserted into the decoding network through the variable instance normalization operation.
[0060] In one implementation, after the smart device obtains the animated image corresponding to the target image, it can send the animated image to the terminal. The user can view the animated image in the interactive page provided by the terminal. Furthermore, the interactive page also provides an animated image customization function. The user can select the area in the target image that needs to be animated, as well as the animation degree and animation style corresponding to each area in the interactive page. The terminal generates corresponding input information based on the user. The input information includes the target image, the area in the target image that needs to be animated, the animation degree and style corresponding to each area, etc. The terminal sends the input information to the smart device. The smart device determines the target mask for the target image based on the input information. The mask values corresponding to different areas in the target mask are not correct, and the division of the area is determined by the received input information. The smart device calls the encoding network and the decoding network to implement animation processing for the area that needs to be animated in the manner indicated by the input information, and does not process the area that does not need to be animated.
[0061] like Figure 4 FIG. 1 is a schematic diagram of an animation result obtained after animating a target image using an embodiment of the present invention. Figure 4In the example, the user inputs an image 40 in an interactive page provided by the terminal, and the terminal sends the image to the smart device. The smart device analyzes the first area 401 and the second area 402 in the image, wherein the first area is a face image area that needs to be animated, and the second area is a non-face area that does not need to be animated. The smart device completes the animation processing of the image based on the mask of the image to obtain an animated image 41, which includes the first area 403 after animation processing and the second area 404 that is not animated. The smart device sends the above animated image 41 to the terminal so that the terminal displays the animated image 41. Optionally, after the user inputs the image 40 in the interactive page provided by the terminal, he can select the first area 401 from the image according to his preference, and the unselected area in the image is the second area 402. The animation style selected by the user from the animation style options is cartoon. The terminal sends the above user selected content and the selected image to the smart device. The smart device generates a mask corresponding to the image based on the above information, and processes it to obtain an animated image 41, wherein the animated image 41 includes the first area 403 after animation processing and the second area 404 that is not animated. The smart device sends the above animated image 41 to the terminal so that the terminal displays the animated image 41 in the interactive page. It should be noted that Figure 4 Just for example, Figure 4 Two half-body images with different shapes are used to illustrate the original image and the animated image respectively. In actual use, the animation effect of the animated image will be more obvious.
[0062] It should be noted that the encoding network and the decoding network are different components of a generator. The encoding network is used to implement the encoding function of the generator, and the decoding network is used to implement the decoding function of the generator. The process of training the encoding network and the decoding network is the process of training the generator. The generator can be a deep learning model trained based on unsupervised learning. The specific training process can be seen in Figure 5 The embodiment shown.
[0063] In an embodiment of the present invention, the smart device obtains a target image to be animated, performs segmentation processing on the target image, determines a reference mask based on the segmentation processing result, determines the encoding features of the target image through an encoding network, and processes the encoding features, reference mask and determined animation style vector determined by the encoding network through a decoding network to obtain an animated image corresponding to the target image. After the segmentation processing of the target image, at least a first region and a second region are obtained, and the mask values of the first region and the second region on the reference mask are different; by implementing the above method, masks, image features and style features of different image regions can be combined to obtain a better animation effect.
[0064] See also Figure 5 , is a schematic diagram of a training process of a generator provided by an embodiment of the present invention. The training method of the generator may include the following steps:
[0065] S501, obtaining a sample image set and a sample comic set. In the embodiment of the present invention, the sample image set includes at least one sample image, and the sample comic set includes at least one sample comic image. The sample image may be a real image, such as a photo, a video screenshot, etc. In a specific implementation scenario, the sample image may be a person image, that is, a real image containing a person area, and may specifically be a person photo, a person screenshot in a video, etc.
[0066] S502: Train the initial generator based on the sample image set and the sample cartoon set to update the parameters in the initial generator.
[0067] In an embodiment of the present invention, after obtaining a sample image set and a sample comic set, the initial generator can be iteratively trained using the sample image set and the sample comic set to update the parameters in the initial generator, that is, to update the parameters of the initial encoding network and the parameters of the initial decoding network in the initial generator.
[0068] In the specific implementation, a round of training process includes Figure 6 The structure shown is expanded. In a round of training for the initial generator, any target sample image in the sample image set and any target sample comic image in the sample comic set are specifically used to train the initial generator. The specific training process can be to determine the target sample image features of the target sample image through the initial encoding network, and to process the target sample image features, the sample reference mask determined after segmenting the target sample image, and the sample comic style vector obtained after style confirmation of the target sample comic image through the initial decoding network in the initial generator to obtain a sample animated image, as shown in the following figure. Figure 6 As shown in the area 601 in the figure. The method of determining the sample reference mask after segmenting the target sample image includes segmenting the target sample image and determining the sample reference mask according to the segmentation result. The method of confirming the style of the target sample comic image includes processing the target sample comic image through a style encoder to obtain a sample comic style vector. Figure 6 As shown in the 603 area.
[0069] Furthermore, the sample animated image features of the sample animated image are determined by the initial encoding network, and the sample animated image features, the sample animated reference mask determined after segmenting the sample animated image, and the sample style vector obtained after style confirmation of the target sample image are processed by the initial decoding network in the initial generator to obtain the sample real image, as shown in the following figure. Figure 6 As shown in the area 602 in the figure. The method of determining the sample animation reference mask after segmenting the sample animation image includes segmenting the sample animation image and determining the sample animation reference mask according to the segmentation processing result. The method of confirming the style of the target sample image includes processing the target sample image through a style encoder to obtain a sample style vector. Figure 6 It should be noted that, since the area distribution of the target sample image and the sample animation image has not changed, the sample reference mask and the sample animation reference mask should be the same.
[0070] Furthermore, the loss value of each feature in the training process is calculated based on the loss function, and then it is determined whether the parameters in the initial generator need to be updated. The loss function includes at least one of the first loss function, the second loss function, the third loss function and the fourth loss function.
[0071] In one implementation, the process of training the initial generator based on the sample image set and the sample comic set includes training based on a first loss function, where the first loss function is used to calculate a first feature loss value between a target sample image in the sample image set and a sample real image determined based on the target sample image. Specifically, the target sample image can be encoded to obtain a first feature vector x, and the sample real image can be encoded to obtain a second feature vector x′, and the first feature vector x and the second feature vector x′ can be used as parameters in the first loss function to calculate the loss value. The first loss function can be specifically L rec(x,x′)=||xx′||2, i.e., the L2 norm between vector x and x′. Through the first loss function, the cycle consistency constraint in unsupervised training is realized, ensuring the consistency of the detailed features (such as the posture and expression of the character) in the target sample image and the sample real image. It should be noted that the sample real image is obtained according to the sample animated image corresponding to the target sample image; the sample animated image is processed by the initial encoding network in the initial generator to obtain the sample animated image features, and the sample animated image features, the sample animated reference mask determined after the sample animated image is segmented, and the sample style vector obtained after the target sample image is style confirmed is processed by the initial decoding network in the initial generator to obtain the sample real image; the target sample image is processed by the initial encoding network in the initial generator to obtain the target sample image features, and the target sample image features, the sample reference mask determined after the target sample image is segmented, and the sample comic style vector obtained after the target sample comic image in the sample comic set is style confirmed is processed by the initial decoding network in the initial generator to obtain the sample animated image.
[0072] In one implementation, the process of training the initial generator based on the sample image set and the sample comic set also includes training based on a second loss function, and the second loss function is used to calculate the second feature loss value between the target sample image and the sample animated image. Specifically, the second feature loss value between the target sample image and the sample animated image is determined by the difference between the first weighted feature vector corresponding to the target sample image and the second weighted feature vector corresponding to the sample animated image; the first weighted feature vector is obtained by weighting the first feature vector based on the sample reference mask after performing feature extraction processing on the target sample image to obtain the first feature vector; the second weighted feature vector is obtained by weighting the second feature vector based on the sample reference mask after performing feature extraction processing on the sample animated image to obtain the second feature vector. It should be noted that the specific method of performing feature extraction processing on the target sample image to obtain the first feature vector can be to call the trained feature extractor to extract the first initial feature vector x, filter the first initial feature vector to obtain the first feature vector H(x), and further, obtain the weighting coefficient (1-mask) based on the sample reference mask (mask), and use the weighting coefficient to weight the first feature vector to obtain the first weighted feature vector (1-mask)*H(x). Similarly, the sample animated image is processed to obtain the second weighted feature vector (1-mask)*H(y), and the second loss function is used to calculate the difference between the above feature vectors to obtain the second feature loss value between the target sample image and the sample animated image. Among them, the second loss function can be specifically L maskcontent(x, y) = (1-mask)*||H(x)-H(y)||1, i.e., the L1 norm between the vectors (1-mask)*H(x) and (1-mask)*H(y). The value range of mask is between 0 and 1. Different regions in the target sample image correspond to different mask values. Different vector values in the feature vector obtained by calculating the target sample image also correspond to different mask values. The mask value of the region with a high degree of animation is 1, and the mask value of the region with a low degree of animation is 0. The region with a high degree of animation can be the first region of the sample, and the region with a low degree of animation can be the second region of the sample. The first feature vector H(x) includes the feature vector H(x1) corresponding to the first region of the sample and the feature vector H(x2) corresponding to the second region of the sample. Since the mask value corresponding to the first area of the sample is greater than 1, after weighting it with (1-mask), it has no effect on the second loss value, while the mask value corresponding to the second area of the sample is greater than 0, after weighting it with (1-mask), the size of the second loss value is determined, that is, the second loss value is determined by the difference between the areas with low animation degree in the image. The method of calculating the loss value by the second loss function ensures that the areas with low animation degree (such as non-face areas in the target image) in the target sample image and the real sample image are consistent.
[0073] In one implementation, the process of training the initial generator based on the sample image set and the sample comic set also includes training based on a third loss function, where the third loss function is used to calculate a third feature loss value between a sample comic style vector s obtained after style confirmation of the target sample comic image and an animated style vector s′ corresponding to the sample animated image. The method of obtaining the sample comic style vector s by style confirmation of the target sample comic image includes processing the target sample comic image through a style encoder to obtain the sample comic style vector s, as shown in FIG. Figure 6 As shown in the area 603, the method of performing style confirmation on the sample animated image to obtain the animated style vector s′ includes processing the sample animated image through a style encoder to obtain the sample comic style vector s′, as shown in FIG. Figure 6 As shown in the area 605. The third loss function can be specifically L style (s, s′) = ||ss′||2, which is the L2 norm between vectors s and s′. By calculating the loss value through the third loss function, it can be ensured that the style vector of the generated result in the image animation process is consistent with the style vector of the input image.
[0074] In one implementation, the process of training the initial generator based on the sample image set and the sample comic set also includes training based on a fourth loss function, where the fourth loss function is used to calculate a fourth feature loss value between the target sample comic image and the sample animated image. The fourth loss function can be specifically L adv =||mm′||2, i.e., the feature loss value between the feature vector m of the target sample comic image and the feature vector m′ of the sample animated image, which can be specifically calculated using the L2 norm. The method of calculating the loss value by the fourth loss function can ensure that the content of the input sample comic image and the generated sample animated image during the image animation process is consistent.
[0075] In one implementation, in the process of training the initial generator based on the sample image set and the sample comic set, it is based on the above-mentioned first loss function, second loss function, third loss function and fourth loss function, that is, the above-mentioned loss functions are used together to optimize the generator.
[0076] It should be noted that Figure 6 In the training process shown, the third loss function L style (s,s′)=||ss′||2In addition to training the generator, it is also used to Figure 6 The style encoder used in the training process is trained, that is, when the third loss value calculated based on the third loss function is less than the third threshold, it is determined that the training of the style encoder is completed. The specific use of the style encoder is as follows Figure 6 As shown in regions 603, 604 and 605, the fourth loss function L adv =||mm′||2In addition to training the generator, it is also used to train Figure 6 The judgement device used in the training process, that is, when the fourth loss value calculated based on the fourth loss function is less than the fourth threshold, it is determined that the training of the judgement device is completed. The specific use of the judgement device is as follows: Figure 6 As shown in the middle area 606, the judger is used to judge whether the input image in the generator is a real image or a cartoon image.
[0077] It should be noted that in a round of training, the input order of images can also be changed to train the model, that is, input sample comic images and output the result of the real comic image. The specific training process can be as follows: Figure 7In the structure shown, in a round of training for the initial generator, any target sample image in the sample image set and any target sample comic image in the sample comic set are specifically used to train the initial generator. The specific training process can be to determine the target sample comic features of the target sample comic image through the initial encoding network, and to process the target sample comic features, the sample comic reference mask determined after segmenting the target sample comic image, and the target sample style vector obtained after style confirmation of the target sample image through the initial decoding network in the initial generator to obtain a realistic sample image. Specifically, Figure 7 As shown in the middle area 701, further, the realistic sample image features of the realistic sample image are determined by the initial encoding network, and the sample reference image is obtained by processing the realistic sample image features, the sample realistic sample reference mask determined after segmenting the realistic sample image, and the sample comic style vector obtained after style confirmation of the target sample comic image through the initial decoding network in the initial generator, as shown in FIG. Figure 7 As shown in the middle area 702, the variables of the parameters in the above training process are imported into the first loss function, the second loss function, the third loss function and the fourth loss function to obtain the first loss value, the second loss value, the third loss value and the fourth loss value respectively, and further optimize the generator.
[0078] S503. When it is detected that the initial generator after the parameter update meets the preset conditions, the initial generator after the parameter update is determined as the generator.
[0079] In the embodiment of the present invention, after a round of training is performed on the initial generator and the parameters in the initial generator are updated, it is detected whether the initial generator after the parameter update meets the preset conditions. When it is detected that the initial generator after the parameter update meets the preset conditions, the initial generator after the parameter update is determined as the generator. The parameters in the generator include the parameters in the encoding network and the parameters in the decoding network.
[0080] In one implementation, whether the initial generator after the parameter update meets the preset conditions is judged by one or more of the first loss value calculated based on the first loss function, the second loss value obtained by the second loss function, the third loss value obtained by the third loss function, and the fourth loss value obtained by the fourth loss function. For example, when the first loss value is less than the first threshold, it is determined that the initial generator after the parameter update meets the preset conditions, or, when the first loss value is less than the first threshold and the second loss value is less than the second threshold, it is determined that the initial generator after the parameter update meets the preset conditions.
[0081] In one implementation, when it is detected that the first loss value calculated based on the first loss function is less than the first threshold, the second loss value calculated based on the second loss function is less than the second threshold, the third loss value calculated based on the third loss function is less than the third threshold, and the fourth loss value calculated based on the fourth loss function is less than the fourth threshold, it is determined that the initial generator after the parameter update meets the preset conditions.
[0082] In the above scheme, the first loss function is used to implement the cycle consistency constraint in unsupervised training, ensuring the consistency of the detailed features (such as the posture and expression of the character) in the target sample image and the sample real image. The second loss function is used to calculate the loss value, ensuring the consistency of the target sample image and the sample real image in the area with a low degree of animation (such as the non-face area in the target image). The third loss function is used to calculate the loss value, ensuring that the style vector of the generated result and the style vector of the input image are consistent during the image animation process. The fourth loss function is used to calculate the loss value, ensuring that the content of the sample comic image input during the image animation process is consistent with the sample animated image generated.
[0083] In one implementation scenario, Figure 8 As shown, the smart device is a server, and the terminal can provide a display interface, in which the user can input an image. After receiving the image, the terminal uploads the image to the server, receives the animation processing result returned by the server, and displays it in the display page. In an optional embodiment, the server can process an image uploaded by a user in multiple styles to obtain multiple animated images, and feed the multiple animated images back to the terminal, each of which can correspond to an animation style. The terminal displays the multiple animated images in the display interface and stores the animated image selected by the user in the display interface.
[0084] The following will be combined with the attached Fig. 9 The image animation processing device provided by the embodiment of the present invention is introduced in detail. Fig. 9 The image animation processing device shown is used to implement the present invention Figure 1a-Figure 8 For the convenience of explanation, only the part related to the embodiment of the present invention is shown, and the specific technical details are not disclosed. Figure 1a-Figure 8 The embodiment shown.
[0085] See also Fig. 9 , is a structural schematic diagram of an image animation processing device provided by the present invention. The image animation processing device 90 may include: an acquisition module 901, a processing module 902 and a determination module 903.
[0086] An acquisition module 901 is used to acquire a target image to be processed into an animation;
[0087] A processing module 902 is used to perform segmentation processing on the target image;
[0088] The determination module 903 is used to determine a reference mask according to the segmentation processing result, wherein the target image is segmented to obtain at least a first area and a second area, and the mask values of the first area and the second area on the reference mask are different; determine the encoding features of the target image through an encoding network; and process the encoding features determined by the encoding network, the reference mask and the determined animation style vector through a decoding network to obtain an animated image corresponding to the target image.
[0089] In one implementation, the processing module 902 is specifically configured to:
[0090] Performing weighted processing on the coding feature based on each mask value in the reference mask through a decoding network to obtain a weighted coding feature;
[0091] Processing the determined animation style vector and the encoding feature through the decoding network to obtain an initial decoding feature;
[0092] The weighted coding feature and the initial decoding feature are fused through the decoding network to obtain a target decoding feature of the target image;
[0093] The target decoder features are decoded by the decoding network to obtain an animated image corresponding to the target image.
[0094] In one implementation, the processing module 902 is specifically configured to:
[0095] The target image is processed through the encoding network to obtain the initial encoding features of the target image;
[0096] Determining a filtering range for the initial encoding feature based on the determined animation style vector;
[0097] The initial coding feature is filtered according to the filtering range to obtain the coding feature of the target image.
[0098] In one implementation, the encoding network and the decoding network are pre-trained, and the spatial size between the features of the i-th layer of the encoding network and the features of the i-th layer from the end of the decoding network is the same. The processing module 902 is specifically configured to:
[0099] The i-th layer features in the weighted coding features and the i-th layer features from the bottom in the initial decoding features are correspondingly fused through the decoding network to achieve fusion processing of the weighted coding features and the initial decoding features.
[0100] In one implementation, the processing module 902 is further configured to:
[0101] Receiving input information for a reference animation image;
[0102] Calling a style encoder to encode the reference cartoon image to obtain a reference style vector of the reference cartoon image;
[0103] A target style vector matching the reference style vector is determined from a database as the determined animation style vector.
[0104] In one implementation, the processing module 902 is further configured to:
[0105] Get a sample image collection and a sample comic collection;
[0106] Training an initial generator based on the sample image set and the sample cartoon set to update parameters in the initial generator;
[0107] When it is detected that the initial generator after parameter update meets the preset conditions, the initial generator after parameter update is determined as the generator, wherein the parameters in the generator include parameters in the encoding network and parameters in the decoding network.
[0108] In one implementation, the process of training the initial generator based on the sample image set and the sample comic set includes: training based on a first loss function; the first loss function is used to calculate a first feature loss value between a target sample image in the sample image set and a sample realistic image determined according to the target sample image;
[0109] The sample realised image is obtained according to the sample animated image corresponding to the target sample image;
[0110] The sample animated image is passed through an initial encoding network in the initial generator to obtain a sample animated image feature, and the sample animated image feature, a sample animated reference mask determined after segmenting the sample animated image, and a sample style vector obtained after style confirmation of the target sample image are processed through an initial decoding network in the initial generator to obtain the sample real image;
[0111] The target sample image is passed through an initial encoding network in the initial generator to obtain target sample image features, and the target sample image features, a sample reference mask determined after segmenting the target sample image, and a sample comic style vector obtained after style confirmation of the target sample comic image in the sample comic set are processed through an initial decoding network in the initial generator to obtain the sample animated image.
[0112] In one implementation, the process of training the initial generator based on the sample image set and the sample comic set also includes: training based on a second loss function; the second loss function is used to calculate a second feature loss value between the target sample image and the sample animated image.
[0113] In one implementation, the second feature loss value between the target sample image and the sample animated image is determined by a difference between a first weighted feature vector corresponding to the target sample image and a second weighted feature vector corresponding to the sample animated image;
[0114] The first weighted feature vector is obtained by weighting the first feature vector obtained by performing feature extraction processing on the target sample image based on the sample reference mask; the second weighted feature vector is obtained by weighting the second feature vector obtained by performing feature extraction processing on the sample animated image based on the sample reference mask.
[0115] In one implementation, the process of training the initial generator based on the sample image set and the sample comic set also includes: training based on a third loss function; the third loss function is used to calculate a third feature loss value between a sample comic style vector obtained after style confirmation of a target sample comic image and an animated style vector corresponding to the sample animated image.
[0116] In one implementation, the process of training the initial generator based on the sample image set and the sample comic set also includes: training based on a fourth loss function; the fourth loss function is used to calculate a fourth feature loss value between the target sample comic image and the sample animated image.
[0117] In one implementation, when it is detected that the first loss value calculated based on the first loss function is less than a first threshold, the second loss value calculated based on the second loss function is less than a second threshold, the third loss value calculated based on the third loss function is less than a third threshold, and the fourth loss value calculated based on the fourth loss function is less than a fourth threshold, it is determined that the initial generator after the parameter update meets the preset conditions.
[0118] In the embodiment of the present invention, the acquisition module 901 acquires the target image to be animated, the processing module 902 performs segmentation processing on the target image, the determination module 903 determines the reference mask according to the segmentation processing result, determines the encoding features of the target image through the encoding network; and processes the encoding features determined by the encoding network, the reference mask and the determined animation style vector through the decoding network to obtain the animated image corresponding to the target image. Among them, after the target image is segmented, the first area and the second area are obtained, and the mask values of the first area and the second area on the reference mask are different; by implementing the above method, the animation degree of different areas in the image can be determined based on the mask in the image, so as to further complete the animation processing of each area in the image based on the trained model, and improve the accuracy of the image animation result.
[0119] See also Fig.10 , provides a schematic diagram of the structure of an intelligent device according to an embodiment of the present invention. As shown in the figure, the intelligent device includes: at least one processor 1001, an input device 1003, an output device 1004, a memory 1005, and at least one communication bus 1002. Of course, it can also include a power supply, a rack, and various other interfaces.
[0120] The memory 1005 may include a volatile memory (volatile memory), such as a random-access memory (RAM); the memory 1005 may also include a non-volatile memory (non-volatile memory), such as a flash memory, a solid-state drive (SSD), etc.; the memory 1005 may also include a combination of the above-mentioned types of memory.
[0121] The processor 1001 may be a central processing unit (CPU). The processor may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), etc. The PLD may be a field-programmable gate array (FPGA), a generic array logic (GAL), etc.
[0122] The communication bus 1002 is used to realize the connection and communication between these components. The input device 1003 can be a control panel or a microphone, etc., and the output device 1004 can be a display screen, etc. Among them, the communication bus 1002 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 1002 can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Fig.10 It is represented by only one thick line, but it does not mean that there is only one bus or one type of bus.
[0123] The processor 1001 may be combined with Fig. 9 In the described device, a set of program codes is stored in the memory 1005, and the processor 1001 calls the program codes stored in the memory 1005, for:
[0124] Get the target image to be animated;
[0125] Performing segmentation processing on the target image;
[0126] Determining a reference mask according to the segmentation processing result, wherein the target image is segmented to obtain at least a first region and a second region, and mask values of the first region and the second region on the reference mask are different;
[0127] Determining encoding features of the target image through an encoding network;
[0128] The encoding features determined by the encoding network, the reference mask and the determined animation style vector are processed by a decoding network to obtain an animated image corresponding to the target image.
[0129] In one implementation, the processor 1001 is specifically configured to:
[0130] Performing weighted processing on the coding feature based on each mask value in the reference mask through a decoding network to obtain a weighted coding feature;
[0131] Processing the determined animation style vector and the encoding feature through the decoding network to obtain an initial decoding feature;
[0132] The weighted coding feature and the initial decoding feature are fused through the decoding network to obtain a target decoding feature of the target image;
[0133] The target decoder features are decoded by the decoding network to obtain an animated image corresponding to the target image.
[0134] In one implementation, the processor 1001 is specifically configured to:
[0135] The target image is processed through the encoding network to obtain the initial encoding features of the target image;
[0136] Determining a filtering range for the initial encoding feature based on the determined animation style vector;
[0137] The initial coding feature is filtered according to the filtering range to obtain the coding feature of the target image.
[0138] In one implementation, the encoding network and the decoding network are pre-trained, the spatial size between the features of the i-th layer of the encoding network and the features of the i-th layer from the end of the decoding network are the same, the input of the encoding network is the target image, and the processor 1001 is specifically configured to:
[0139] The i-th layer features in the weighted coding features and the i-th layer features from the bottom in the initial decoding features are correspondingly fused through the decoding network to achieve fusion processing of the weighted coding features and the initial decoding features.
[0140] In one implementation, the processor 1001 is specifically configured to:
[0141] Receiving input information for a reference animation image;
[0142] Calling a style encoder to encode the reference cartoon image to obtain a reference style vector of the reference cartoon image;
[0143] A target style vector matching the reference style vector is determined from a database as the determined animation style vector.
[0144] In one implementation, the processor 1001 is specifically configured to:
[0145] Get a sample image collection and a sample comic collection;
[0146] Training an initial generator based on the sample image set and the sample cartoon set to update parameters in the initial generator;
[0147] When it is detected that the initial generator after parameter update meets the preset conditions, the initial generator after parameter update is determined as the generator, wherein the parameters in the generator include parameters in the encoding network and parameters in the decoding network.
[0148] In one implementation, the process of training the initial generator based on the sample image set and the sample comic set includes: training based on a first loss function; the first loss function is used to calculate a first feature loss value between a target sample image in the sample image set and a sample realistic image determined according to the target sample image;
[0149] The sample realised image is obtained according to the sample animated image corresponding to the target sample image;
[0150] The sample animated image is passed through an initial encoding network in the initial generator to obtain a sample animated image feature, and the sample animated image feature, a sample animated reference mask determined after segmenting the sample animated image, and a sample style vector obtained after style confirmation of the target sample image are processed through an initial decoding network in the initial generator to obtain the sample real image;
[0151] The target sample image is passed through an initial encoding network in the initial generator to obtain target sample image features, and the target sample image features, a sample reference mask determined after segmenting the target sample image, and a sample comic style vector obtained after style confirmation of the target sample comic image in the sample comic set are processed through an initial decoding network in the initial generator to obtain the sample animated image.
[0152] In one implementation, the process of training the initial generator based on the sample image set and the sample comic set also includes: training based on a second loss function; the second loss function is used to calculate a second feature loss value between the target sample image and the sample animated image.
[0153] In one implementation, the second feature loss value between the target sample image and the sample animated image is determined by a difference between a first weighted feature vector corresponding to the target sample image and a second weighted feature vector corresponding to the sample animated image;
[0154] The first weighted feature vector is obtained by weighting the first feature vector obtained by performing feature extraction processing on the target sample image based on the sample reference mask; the second weighted feature vector is obtained by weighting the second feature vector obtained by performing feature extraction processing on the sample animated image based on the sample reference mask.
[0155] In one implementation, the process of training the initial generator based on the sample image set and the sample comic set also includes: training based on a third loss function; the third loss function is used to calculate a third feature loss value between a sample comic style vector obtained after style confirmation of a target sample comic image and an animated style vector corresponding to the sample animated image.
[0156] In one implementation, the process of training the initial generator based on the sample image set and the sample comic set also includes: training based on a fourth loss function; the fourth loss function is used to calculate a fourth feature loss value between the target sample comic image and the sample animated image.
[0157] In one implementation, when it is detected that the first loss value calculated based on the first loss function is less than a first threshold, the second loss value calculated based on the second loss function is less than a second threshold, the third loss value calculated based on the third loss function is less than a third threshold, and the fourth loss value calculated based on the fourth loss function is less than a fourth threshold, it is determined that the initial generator after the parameter update meets the preset conditions.
[0158] In an embodiment of the present invention, the processor 1001 obtains a target image to be animated, performs segmentation processing on the target image, and determines a reference mask according to the segmentation processing result. The processor 1001 determines the encoding features of the target image through the encoding network; the processor 1001 processes the encoding features determined by the encoding network, the reference mask, and the determined animation style vector through the decoding network to obtain an animated image corresponding to the target image. Among them, after the target image is segmented, a first region and a second region are obtained, and the mask values of the first region and the second region on the reference mask are different; by implementing the above method, the animation degree of different regions in the image can be determined based on the mask in the image, so as to further complete the animation processing of each region in the image based on the trained model, thereby improving the accuracy of the image animation result.
[0159] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The computer storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.
[0160] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.
Claims
1. A method for processing image animation, characterized in that: The method comprises: Get the target image to be animated; Performing segmentation processing on the target image, and determining a reference mask according to the segmentation processing result, wherein the target image is segmented to obtain at least a first region and a second region, and mask values of the first region and the second region on the reference mask are different; Determining encoding features of the target image through an encoding network; Processing the coding features determined by the coding network, the reference mask and the determined animation style vector through a decoding network to obtain an animated image corresponding to the target image; Wherein, determining the coding features of the target image through a coding network includes: The target image is processed through the encoding network to obtain the initial encoding features of the target image; Determining a filtering range for the initial encoding feature based on the determined animation style vector; The initial coding feature is filtered according to the filtering range to obtain the coding feature of the target image.
2. The method according to claim 1, characterized in that The step of processing the coding features determined by the coding network, the reference mask and the determined animation style vector through the decoding network to obtain an animated image corresponding to the target image includes: Performing weighted processing on the coding feature based on each mask value in the reference mask through a decoding network to obtain a weighted coding feature; Processing the determined animation style vector and the encoding feature through the decoding network to obtain an initial decoding feature; The weighted coding feature and the initial decoding feature are fused through the decoding network to obtain a target decoding feature of the target image; The target decoder features are decoded by the decoding network to obtain an animated image corresponding to the target image.
3. The method according to claim 2, characterized in that The encoding network and the decoding network are pre-trained, the spatial size between the features of the i-th layer of the encoding network and the features of the i-th layer from the end of the decoding network are the same, and the input of the encoding network is the target image; The step of fusing the weighted coding feature and the initial decoding feature through the decoding network to obtain the target decoding feature of the target image includes: The i-th layer features in the weighted coding features and the i-th layer features from the bottom in the initial decoding features are correspondingly fused through the decoding network to achieve fusion processing of the weighted coding features and the initial decoding features.
4. The method according to claim 1, characterized in that Before the decoding network processes the encoding features determined by the encoding network, the reference mask and the determined animation style vector, the method further includes: Receiving input information for a reference animation image; Calling a style encoder to encode the reference cartoon image to obtain a reference style vector of the reference cartoon image; A target style vector matching the reference style vector is determined from a database as the determined animation style vector.
5. The method according to claim 1, characterized in that The encoding network and the decoding network are included in the generator, and the method further includes: Get a sample image collection and a sample comic collection; Training an initial generator based on the sample image set and the sample cartoon set to update parameters in the initial generator; When it is detected that the initial generator after parameter update meets the preset conditions, the initial generator after parameter update is determined as the generator, wherein the parameters in the generator include parameters in the encoding network and parameters in the decoding network.
6. The method according to claim 5, characterized in that The process of training the initial generator based on the sample image set and the sample comic set includes: training based on a first loss function; the first loss function is used to calculate a first feature loss value between a target sample image in the sample image set and a sample realization image determined according to the target sample image; The sample realised image is obtained according to the sample animated image corresponding to the target sample image; The sample animated image is passed through an initial encoding network in the initial generator to obtain a sample animated image feature, and the sample animated image feature, a sample animated reference mask determined after segmenting the sample animated image, and a sample style vector obtained after style confirmation of the target sample image are processed through an initial decoding network in the initial generator to obtain the sample real image; The target sample image is passed through an initial encoding network in the initial generator to obtain target sample image features, and the target sample image features, a sample reference mask determined after segmenting the target sample image, and a sample comic style vector obtained after style confirmation of the target sample comic image in the sample comic set are processed through an initial decoding network in the initial generator to obtain the sample animated image.
7. The method according to claim 6, characterized in that The process of training the initial generator based on the sample image set and the sample comic set also includes: training based on a second loss function; the second loss function is used to calculate a second feature loss value between the target sample image and the sample animated image.
8. The method according to claim 7, characterized in that The second feature loss value between the target sample image and the sample animated image is determined by the difference between the first weighted feature vector corresponding to the target sample image and the second weighted feature vector corresponding to the sample animated image; The first weighted feature vector is obtained by weighting the first feature vector obtained by performing feature extraction processing on the target sample image based on the sample reference mask; the second weighted feature vector is obtained by weighting the second feature vector obtained by performing feature extraction processing on the sample animated image based on the sample reference mask.
9. The method according to claim 7 or 8, characterized in that The process of training the initial generator based on the sample image set and the sample comic set also includes: training based on a third loss function; the third loss function is used to calculate a third feature loss value between a sample comic style vector obtained after style confirmation of a target sample comic image and an animated style vector corresponding to the sample animated image.
10. The method according to claim 9, characterized in that The process of training the initial generator based on the sample image set and the sample comic set also includes: training based on a fourth loss function; the fourth loss function is used to calculate the fourth feature loss value between the target sample comic image and the sample animated image.
11. The method according to claim 10, characterized in that When it is detected that the first loss value calculated based on the first loss function is less than the first threshold, the second loss value calculated based on the second loss function is less than the second threshold, the third loss value calculated based on the third loss function is less than the third threshold, and the fourth loss value calculated based on the fourth loss function is less than the fourth threshold, it is determined that the initial generator after the parameter update meets the preset conditions.
12. An image animation processing device, characterized in that: The device comprises: An acquisition module is used to acquire a target image to be processed into an animation; A processing module, used for segmenting the target image; A determination module, configured to determine a reference mask according to a segmentation processing result, wherein the target image is segmented to obtain at least a first region and a second region, and mask values of the first region and the second region on the reference mask are different; The determination module is further used to determine the coding features of the target image through a coding network; The determination module is further used to process the coding features determined by the coding network, the reference mask and the determined animation style vector through a decoding network to obtain an animated image corresponding to the target image; Wherein, the processing module is specifically used for: The target image is processed through the encoding network to obtain the initial encoding features of the target image; Determining a filtering range for the initial encoding feature based on the determined animation style vector; The initial coding feature is filtered according to the filtering range to obtain the coding feature of the target image.
13. A smart device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program, and the processor runs the computer program to execute the method according to any one of claims 1 to 11.
14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 11.
15. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 11 is implemented.
