Method and device for generating character interpolation image, computer-readable storage medium, and terminal

Abstract

A method and device for generating a person interpolation image, a computer-readable storage medium, and a terminal, the method comprising: obtaining a pixel matrix of a first image and a pixel matrix of a second image, and splicing them to obtain a first splicing matrix, wherein the first image and the second image contain the same target person; predicting the human body prediction features of the target person according to the first splicing matrix, and performing feature extraction on the first splicing matrix to obtain image features, wherein the human body prediction features contain the predicted human body information of the target person at the time when the person interpolation image belongs; performing feature fusion on the human body prediction features of the target person and the image features to obtain fusion features; and decoding the fusion features to obtain the person interpolation image. The above scheme is helpful to obtain a person interpolation image with higher definition and better quality.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and device for generating a character interpolation image, a computer-readable storage medium, and a terminal. Background Art

[0002] The image interpolation problem is a classic problem in the field of computer vision. It can be simply described as generating an image at the intermediate moment given two images at different moments. A common application scenario of image interpolation is video frame upscaling. By interpolating a video, a video with insufficient frame rate or stuck can be turned into a smoother video.

[0003] In the prior art, deep learning methods based on optical flow information or deep learning methods based on image pixel information are mainly used to obtain interpolated images, but these methods are not effective in obtaining interpolated images, especially for interpolated images containing people. Taking the deep learning method based on image pixel information as an example, the existing scheme mainly uses a pre-trained neural network model (for example, an image feature extraction network) to extract and fuse image features of two images at different times, and then generates an interpolated image based on the fused image features. The interpolated images containing people obtained by this method (referred to as interpolated images of people) are prone to blurring problems at the limbs of the human body (for example, arms, palms, etc.). Summary of the invention

[0004] The technical problem solved by the embodiment of the present invention is how to improve the blurring problem of the human body parts in the human body interpolation image, so as to obtain a human body interpolation image with higher definition and better quality.

[0005] To solve the above technical problems, an embodiment of the present invention provides a method for generating a person interpolation image, comprising the following steps: obtaining a pixel matrix of a first image and a pixel matrix of a second image, and splicing them to obtain a first splicing matrix, wherein the first image and the second image contain the same target person; predicting a human body prediction feature of the target person according to the first splicing matrix, and performing feature extraction on the first splicing matrix to obtain image features, wherein the human body prediction feature includes predicted human body information of the target person at the moment when the person interpolation image belongs; performing feature fusion on the human body prediction feature of the target person and the image feature to obtain fused features; and decoding the fused features to obtain the person interpolation image.

[0006] Optionally, the feature fusion is achieved by using a pre-trained feature fusion network, which includes a convolutional network and an attention mechanism network; the human body prediction features of the target person and the image features are fused to obtain fused features, including: splicing the human body prediction features of the target person and the image features to obtain a second splicing matrix; inputting the second splicing matrix into the convolutional network for convolution processing to obtain pre-fused features; inputting the pre-fused features into the attention mechanism network for multiple rounds of matrix item-by-item product operations to obtain the fused features.

[0007] Optionally, the attention mechanism network includes multiple attention sub-networks connected in sequence; the pre-fused features are input into the attention mechanism network for multiple rounds of matrix item-by-item product operations to obtain the fused features, including: using the pre-fused features as input data of the first attention sub-network, and starting from the first attention sub-network, each attention sub-network performs a round of matrix item-by-item product operations on its input data to obtain output data, and the output data is used as input data of the next attention sub-network, and the output data of the last attention sub-network is used as the fused features.

[0008] Optionally, for each attention sub-network, a matrix product operation is performed on its input data to obtain output data, including: performing dimensionality reduction processing on the input data to obtain a corresponding attention matrix; and performing a matrix product operation on the attention matrix and the input data to obtain the output data.

[0009] Optionally, the following formula is used to perform a matrix product operation on the attention matrix and the input data to obtain the output data:

[0010]

[0011] F∈R C×M×N ;

[0012] Where F represents the input data of the attention subnetwork, C represents the matrix dimension of the input data, M represents the number of matrix rows of the input data, W represents the number of matrix columns of the input data, N1(F) represents the attention matrix obtained by reducing the dimension of F, N1(F)∈R C×1×1 Or N1(F)∈R 1×M×N , Represents a matrix term-wise product operation between two matrices.

[0013] Optionally, the method further includes: determining a pre-trained interpolation image generation model, the interpolation image generation model including: a human feature prediction network and a human information prediction network, the human feature prediction network is used to predict the human prediction features of the target person, and the human information prediction network is used to convert the human prediction features into predicted human information; the pre-trained interpolation image generation model is trained in the following manner: obtaining a pixel matrix of a first sample image and a pixel matrix of a second sample image, and splicing them to obtain a sample splicing matrix; obtaining a third sample image located between the moments of the first sample image and the second sample image, and determining the real sample human body information based on the third sample image, wherein the The first sample image, the second sample image and the third sample image are derived from the same sample video and contain the same sample target person; at least based on the human body information loss function, the sample splicing matrix is ​​input into the interpolated image generation model for iterative training to optimize the parameters of the human body feature prediction network and the human body information prediction network; wherein each round of iterative training includes: inputting the sample splicing matrix into the human body feature prediction network to predict the sample human body prediction features of the sample target person, and then inputting the sample human body prediction features into the human body information prediction network to obtain predicted sample human body information; and calculating the function value of the human body information loss function according to the real sample human body information and the predicted sample human body information.

[0014] Optionally, the pre-trained interpolation image generation model also includes: an image feature extraction network, a feature fusion network and an image generation network, the image feature extraction network is used to perform feature extraction to obtain the image features, the feature fusion network is used to perform feature fusion to obtain the fusion features, and the image generation network is used to decode the fusion features to obtain the character interpolation image; at least based on the human body information loss function, the sample splicing matrix is ​​input into the interpolation image generation model for iterative training, including: performing weighted operations on the human body information loss function, the first image difference loss function and the second image difference loss function to obtain a fusion loss function; using the fusion loss function, the sample splicing matrix is ​​input into the interpolation image generation model type for iterative training; wherein each round of iterative training also includes: inputting the sample splicing matrix into the image feature extraction network for feature extraction to obtain sample image features; inputting the sample image features and the sample human body prediction features into the feature fusion network for feature fusion to obtain sample fusion features; inputting the sample fusion features into the image generation network for decoding to obtain sample interpolated images; calculating the function values ​​of the first image difference loss function and the second image difference loss function respectively according to the pixel values ​​of the third sample image and the pixel values ​​of the sample interpolated image, and performing weighted operation on the function values ​​of the human body information loss function, the first image difference loss function and the second image difference loss function to obtain the function value of the fusion loss function.

[0015] Optionally, one or more of the following is satisfied: the first image difference loss function is a perceptual loss function; the second image difference loss function is an L1 loss function; the weight of the human body information loss function is greater than the weights of the first image difference loss function and the second image difference loss function.

[0016] An embodiment of the present invention further provides a device for generating a person interpolation image, comprising: a pixel matrix splicing module, used to obtain a pixel matrix of a first image and a pixel matrix of a second image, and splice them to obtain a first splicing matrix, wherein the first image and the second image contain the same target person; a module for determining features to be fused, used to predict a human body prediction feature of the target person according to the first splicing matrix, and to extract features from the first splicing matrix to obtain image features, wherein the human body prediction feature includes predicted human body information of the target person at the time when the person interpolation image belongs; a feature fusion module, used to perform feature fusion on the human body prediction feature of the target person and the image feature to obtain a fused feature; and a module for determining a person interpolation image, used to decode the fused feature to obtain the person interpolation image.

[0017] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned method for generating a character interpolated image are executed.

[0018] An embodiment of the present invention further provides a terminal, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor executes the steps of the above-mentioned method for generating a character interpolated image when running the computer program.

[0019] Compared with the prior art, the technical solution of the embodiment of the present invention has the following beneficial effects:

[0020] Compared with the existing conventional technology that only relies on image feature extraction and feature fusion operations to obtain the interpolated image of the person, this embodiment additionally introduces the prediction operation of the human body prediction feature (correspondingly, in the interpolated image generation model, it is reflected in the addition of a human body feature prediction network and a human body information prediction network). Specifically: based on the concatenated matrix of the pixel matrix of the first image and the pixel matrix of the second image, the human body prediction features of the target person are predicted; then the human body prediction features and the image features are feature fused to obtain the fused features, and then decoded to obtain the interpolated image of the person.

[0021] Since the predicted human body prediction features include the predicted human body information of the target person at the moment when the person interpolation image belongs (for example, the position information of the key points of the limbs and / or the limbs three-dimensional modeling information), compared with generating the person interpolation image only based on the image features obtained by image feature extraction, the present implementation scheme adds the human body prediction features, which helps to solve the blur problem of the person interpolation images generated by the prior art at the limbs, and obtain clearer and higher quality person interpolation images.

[0022] Furthermore, in the model training stage, by integrating the human body information loss function, the first image difference loss function and the second image difference loss function, that is, using the fusion loss function obtained by weighted operation of the above three loss functions as the loss function for training the model, compared with using only a single loss function or two loss functions, using the training scheme provided by the embodiment of the present invention helps to improve the pertinence of model training and the accuracy of the optimized network parameters, and obtain a more accurate and reliable interpolation image generation model. Furthermore, by applying a more accurate model, it is helpful to obtain a better interpolation image of a person.

[0023] Furthermore, the human body information loss function is mainly used to optimize the parameters of the human body feature prediction network, and the network is used to predict the human body prediction features of the target person, which plays a very critical role in improving the clarity of the generated person interpolation image. Therefore, in the embodiment of the present invention, by setting a relatively larger weight for the human body information loss function, it is helpful to obtain a better network optimization effect, and then obtain a better quality person interpolation image.

[0024] Furthermore, in the feature fusion operation, the human body prediction features and the image features are first spliced ​​for preliminary fusion; then the splicing result (i.e., the second splicing matrix) is re-fused using a convolutional network to obtain pre-fused features; and the attention mechanism network is continued to perform one or more rounds of matrix item-by-item product operations on the pre-fused features. Thus, by comprehensively adopting a variety of different feature fusion methods and performing multiple rounds of fusion, it is helpful to achieve deep fusion of human body prediction features and image features, obtain more accurate fusion features, and generate a higher definition and better quality human interpolation image. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a flow chart of a method for generating a character interpolation image in an embodiment of the present invention;

[0026] Figure 2 yes Figure 1 A flowchart of a specific implementation of step S13;

[0027] Figure 3 is a structural schematic diagram of an interpolated image generation model in an embodiment of the present invention;

[0028] Figure 4 is a flowchart of a specific implementation method of training an interpolated image generation model in an embodiment of the present invention;

[0029] Figure 5 is a flowchart of another specific implementation method of training an interpolated image generation model in an embodiment of the present invention;

[0030] Figure 6 It is a schematic diagram of the reasoning process of generating a person interpolation image in a typical application scenario of an embodiment of the present invention;

[0031] Figure 7 It is a structural schematic diagram of a device for generating a character interpolation image in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to make the above-mentioned objects, features and beneficial effects of the present invention more obvious and understandable, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0033] Reference Figure 1 , Figure 1 The flowchart of a method for generating a character interpolation image in an embodiment of the present invention is shown in FIG. The method can be applied to a terminal with an image processing function, such as but not limited to: a computer, a mobile phone, a tablet computer, a vehicle-mounted terminal, a smart wearable terminal (e.g., a smart watch), a server, a cloud platform, etc.

[0034] The method for generating a character interpolation image may include steps S11 to S14:

[0035] Step S11: obtaining a pixel matrix of a first image and a pixel matrix of a second image, and stitching them together to obtain a first stitching matrix, wherein the first image and the second image contain the same target person;

[0036] Step S12: obtaining predicted human body features of the target person according to the first splicing matrix, and performing feature extraction on the first splicing matrix to obtain image features, wherein the predicted human body features include predicted human body information of the target person at the time when the person interpolated frame image belongs;

[0037] Step S13: performing feature fusion on the predicted human body features of the target person and the image features to obtain fused features;

[0038] Step S14: Decode the fusion feature to obtain the character interpolation image.

[0039] In the specific implementation of step S11, the first image and the second image (which may be referred to as two images to be inserted) may be obtained from different moments of the same video. For example, an image belonging to the first moment t1 may be selected as the first image before the moment of freeze or frame loss of the same video, and an image belonging to the second moment t2 may be selected as the second image after the moment of freeze or frame loss. In a specific implementation, the moment to which the interpolated image of the character to be generated belongs (for example, the moment of freeze or frame loss) is usually the middle moment t3 between the moment t1 of the first image and the moment t2 of the second image.

[0040] The first image and the second image contain the same target person, and the number of the target person may be one or more. The same target person may have the same or different body movements in the first image and the second image, for example, the palm of the target person is located at the waist or abdomen in the first image, and the palm of the target person is located at the chest in the second image.

[0041] The pixel matrix of the first image may specifically refer to a matrix containing channel values ​​of each pixel of the first image. Taking the first image as an RGB image with M×N pixels, each pixel having three color channels of red (R), green (G), and blue (B) as an example, the pixel matrix of the first image may be expressed as R1 3×M×N Specifically, the pixel matrix of the first image is composed of 3 sub-matrices with M rows and N columns (or N rows and M columns), and each element in the pixel matrix has 3 color channel values. The content of the pixel matrix of the second image and the determination method thereof refer to the aforementioned description of the pixel matrix of the first image, which will not be repeated here.

[0042] Specifically, the pixel matrix of the first image (referred to as the first pixel matrix) and the pixel matrix of the second image (referred to as the second pixel matrix) are spliced ​​to obtain the first spliced ​​matrix as follows: the number of rows and the number of columns of the first pixel matrix and the second pixel matrix are kept unchanged, and the color channel values ​​of each element of the first pixel matrix and the elements at the same position of the second pixel matrix are spliced ​​to obtain the first spliced ​​matrix. The number of color channel values ​​of each element of the first spliced ​​matrix is ​​the sum of the number of color channel values ​​of each element of the first pixel matrix and the number of color channel values ​​of each element of the second pixel matrix.

[0043] For example, the pixel matrix of the first image is R1 3×M×N ; The pixel matrix of the second image is R2 3×M×N , then the first splicing matrix obtained after splicing can be expressed as: R3 6×M×N , that is, each element of the first stitching matrix has 6 color channel values.

[0044] In the specific implementation of step S12, the human body prediction feature may include predicted human body information (or called limb information, limb movement information) of the target person in the person interpolation image. Specifically, the human body prediction feature includes the human body information (i.e., predicted human body information) of the target person at the moment of the person interpolation image (as mentioned above, this moment is usually the middle moment t3 between the moment t1 of the first image and the moment t2 of the second image).

[0045] Among them, the predicted human body information of the target person may be the position information of the key points of the limbs of the target person and / or the 3D modeling information of the limbs, the key points of the limbs may be selected from but not limited to the key points such as the finger joints, wrist joints, elbow joints, shoulder joints, neck joints, knee joints, hip joints, ankle joints, etc., and the 3D modeling information of the limbs may be selected from but not limited to: the rotation angle of the hand, the size of the palm / finger, the amplitude of the hand movement, the rotation angle of the leg, the angle between the thigh and the calf, the distance between the legs, the distance between the feet and the ground, etc. As an example, in the first image, the hand of the target person is on the abdomen, and in the second image, the hand of the target person moves to the shoulder, then the predicted human body prediction features include the position information of the hand of the target person between the abdomen and the shoulder and / or the 3D modeling information of the limbs.

[0046] The image features may include image pixel feature information of the first image and the second image, and the image pixel feature information may be selected from but not limited to: image background information, limb information of the target person, human body contour information, skin texture information, clothing color information, etc.

[0047] In the specific implementation of step S13, feature fusion is performed on the predicted human body features of the target person and the image features to obtain fused features.

[0048] In a first specific implementation, the matrix of the target person's predicted human body features and the matrix of the image features may be directly spliced, and the spliced ​​matrix is ​​used as the fusion feature.

[0049] In the second specific implementation, a feature fusion network may be used to implement the feature fusion operation. Figure 2 , Figure 2 yes Figure 1 A flowchart of a specific implementation of step S13 in the embodiment, in this implementation, the feature fusion is achieved by using a pre-trained feature fusion network, and the feature fusion network includes a convolutional network and an attention mechanism network; the human body prediction features of the target person and the image features are feature fused to obtain fused features, which may specifically include steps S21 to S23.

[0050] In step S21, the human body prediction features and the image features are spliced ​​to obtain a second splicing matrix.

[0051] Furthermore, the feature splicing in step S21 may include: splicing (or inserting) each row element (or code) of the human prediction feature as a whole to a preset position of the same row element (or code) of the image feature; or, splicing each row element (or code) of the image feature as a whole to a preset position of the same row element (or code) of the human prediction feature.

[0052] In a specific implementation, the preset position may be the position after the last encoding of each row of the image feature or the human body prediction feature.

[0053] In the embodiment of the present invention, by adopting an integrated splicing scheme, compared with the use of a distributed and random insertion method, it is helpful to reduce the probability of splicing errors, avoid the original information contained in the predicted human body features and image features from being changed, so that the second splicing matrix obtained subsequently contains accurate predicted human body information and image feature information. Further, by splicing each row of one feature matrix to the position after the last code of the corresponding row of another feature matrix, compared with inserting from the middle code position of the matrix, it is possible to avoid the code position and order in the original matrix from being disrupted, and completely and comprehensively retain the original feature information in the original matrix.

[0054] It should be pointed out that, in a specific implementation, the method and position of splicing between features are not limited to the above-described solution, and other appropriate methods can be adopted in combination with different scene requirements.

[0055] In step S22, the second splicing matrix is ​​input into the convolutional network for convolution processing to obtain pre-fusion features.

[0056] In step S23, the pre-fused features are input into the attention mechanism network to perform one or more rounds of matrix item-by-item product operations to obtain the fused features.

[0057] The matrix item-by-item product operation may also be called a Hadamard product operation or an element-by-element product operation, and each element of the matrix output by the operation is the product of the elements at the same position of the two input matrices.

[0058] In the embodiment of the present invention, the human body prediction features and the image features are first spliced ​​for preliminary fusion; then the splicing result (i.e., the second splicing matrix) is re-fused using a convolutional network to obtain pre-fused features; and the attention mechanism network is continued to perform one or more rounds of matrix item-by-item product operations on the pre-fused features. Thus, by comprehensively adopting a variety of different feature fusion methods and performing multiple rounds of fusion, it is helpful to achieve deep fusion of human body prediction features and image features, obtain more accurate fusion features, and make the generated human interpolation images have higher clarity and better quality.

[0059] Furthermore, in the above-mentioned embodiment, the attention mechanism network may include a plurality of sequentially connected attention sub-networks; the pre-fused features are input into the attention mechanism network for multiple rounds of matrix item-wise product operations to obtain the fused features, including: using the pre-fused features as the input data of the first attention sub-network, and starting from the first attention sub-network, each attention sub-network performs a round of matrix item-wise product operations on its input data to obtain output data, and the output data is used as the input data of the next attention sub-network, and the output data of the last attention sub-network is used as the fused features.

[0060] Furthermore, for each attention sub-network, a matrix product operation is performed on its input data to obtain output data, including: performing dimensionality reduction processing on the input data to obtain a corresponding attention matrix; and performing a matrix product operation on the attention matrix and the input data to obtain the output data.

[0061] Furthermore, the following formula may be used to perform a matrix product operation on the attention matrix and the input data to obtain the output data:

[0062]

[0063] F∈R C×M×N ;

[0064] Where F represents the input data of the attention sub-network, C represents the matrix dimension of the input data, M represents the number of matrix rows of the input data, and N represents the number of matrix columns of the input data. That is, the input data F is a matrix composed of C sub-matrices with M rows and N columns. Each element in the matrix F has C values. N1(F) represents the attention matrix obtained by reducing the dimension of F. N1(F)∈R C×1×1 Or N1(F)∈R 1×M×N , that is, N1(F) represents a single element with C values, N1(F) represents a single M-row, N-column matrix, each element of which has 1 value, Represents a matrix term-wise product operation between two matrices.

[0065] Specifically, for each attention sub-network, you can choose to use the following method 1 or method 2 to perform dimensionality reduction processing to obtain the attention matrix corresponding to the input data:

[0066] Method 1: Input data F∈R C×M×N , the attention matrix N1(F)∈R obtained after dimensionality reduction C×1×1 ;

[0067] Method 2: Input data F∈R C×M×N , the attention matrix N1(F)∈R obtained after dimensionality reduction1×M×N .

[0068] Compared with the use of a feature fusion network containing only a single attention network to iteratively perform multiple rounds of fusion processing, this embodiment uses a feature fusion network containing multiple sequentially connected attention sub-networks to perform linear matrix item-by-item product operations. Therefore, different parameters can be set for different attention sub-networks to perform different dimensionality reduction processing on the input data and obtain their corresponding attention matrices, so that the matrix item-by-item product operations output by different attention sub-networks can be different, so as to realize flexible control of the feature fusion process and achieve the expected deep fusion effect.

[0069] In the specific implementation, Figure 1 The method for generating a character interpolated image shown in the figure can be implemented by adopting a pre-trained interpolated image generation model, and the process of generating a character interpolated image by applying the pre-trained interpolated image generation model can be called a model application (or reasoning) process.

[0070] Combine the following Figures 3 to 5 The specific structure, training method and model reasoning process of the interpolation image generation model are described in detail.

[0071] Reference Figure 3 , Figure 3 It is a structural schematic diagram of an interpolated image generation model in an embodiment of the present invention.

[0072] The interpolated frame image generation model may include: a human feature prediction network 31 and a human information prediction network 32 .

[0073] The human feature prediction network 31 is used to predict the human predicted features of the target person according to the first splicing matrix, and the human information prediction network 32 is used to convert the human predicted features into predicted human information.

[0074] Combination Figure 4 , Figure 4 1 is a flowchart of a specific implementation method of training the interpolation image generation model in an embodiment of the present invention. In this implementation method, the human body information loss function is used as the training loss function for iterative training, and the training goal is to optimize the parameters of the human body feature prediction network 31 and the parameters of the human body information prediction network 32.

[0075] The specific process of training may include the following steps S41 to S43.

[0076] In step S41, a pixel matrix of a first sample image and a pixel matrix of a second sample image are obtained, and they are spliced ​​to obtain a sample splicing matrix.

[0077] For a specific method of performing matrix splicing to obtain the sample splicing matrix, see Figure 1 The description related to the splicing to obtain the first splicing matrix in step S11 is not repeated here.

[0078] In step S42, a third sample image (corresponding to Figure 3 ), and determine the real sample human body information based on the third sample image, wherein the first sample image, the second sample image and the third sample image are derived from the same sample video and contain the same sample target person.

[0079] The time to which the third sample image belongs may be an intermediate time between the time to which the first sample image belongs and the time to which the second sample image belongs.

[0080] The method of determining the real sample human body information based on the third sample image is as follows:

[0081] Method 1: multiple human body key points or limb key points can be selected from the sample target person in the third sample image, and the position information of each selected key point can be used as the real sample human body information. The limb key points can be selected from but not limited to key points such as finger joints, wrist joints, elbow joints, shoulder joints, neck joints, knee joints, hip joints, and ankle joints.

[0082] In a second method, limb 3D modeling information can be determined for the sample target person in the third sample image, and the limb 3D modeling information can be used as the real sample human body information. The limb 3D modeling information can be selected from, but not limited to: hand rotation angle, palm / finger size, hand movement amplitude, leg rotation angle, calf-to-leg angle, leg distance, and foot-to-ground distance.

[0083] In a specific implementation, the position information of the limb key points or the limb three-dimensional modeling information of the sample target person can be converted into a matrix form, and the converted matrix is ​​used as the real sample human body information.

[0084] In step S43, based at least on the human body information loss function, the sample splicing matrix is ​​input into the interpolation image generation model for iterative training to optimize the parameters of the human body feature prediction network and the human body information prediction network.

[0085] Specifically, the human body information loss function is constructed using the real sample human body information and the predicted sample human body information output by the human body information prediction network 32, that is, the human body information loss function is constructed using a preset loss function with the real sample human body information and the predicted sample human body information as function parameters. The human body information loss function can be expressed as: Among them, f() represents the human information loss function, represents the real sample human body information, and M represents the predicted sample human body information output by the human body information prediction network 32. The preset loss function may be selected from but not limited to L1 loss function, mean square error loss function, cross entropy loss function, and the like.

[0086] Specifically, in step S43, each round of iterative training includes the following steps S431 to S433:

[0087] S431, inputting the sample splicing matrix into the human body feature prediction network 31, and predicting the sample human body prediction features of the sample target person;

[0088] S432, inputting the sample human body prediction feature into the human body information prediction network 32 to obtain the predicted sample human body information;

[0089] S433, calculating the function value of the human body information loss function (corresponding to Figure 3 ), that is, the loss function value obtained in the current round of iteration.

[0090] Further, continue to refer to Figure 3 The interpolation image generation model may also include: an image feature extraction network 33, a feature fusion network 34 and an image generation network 35.

[0091] The image feature extraction network 33 is used to extract features from the first splicing matrix to obtain image features, the feature fusion network 34 is used to fuse the human body prediction features and the image features of the target person to obtain fused features, and the image generation network 35 is used to decode the fused features to obtain the person interpolated image.

[0092] Combination Figure 5 , Figure 5 is a flowchart of another specific implementation method of training the interpolation image generation model in an embodiment of the present invention. Figure 4The main difference between the training methods shown is that this training method uses a fusion loss function obtained by weighted operation of the human body information loss function, the first image difference loss function and the second image difference loss function as the training loss function for iterative training, and the training goal is to optimize not only the parameters of the human body feature prediction network 31 and the parameters of the human body information prediction network 32, but also the parameters of the image feature extraction network 33, the parameters of the feature fusion network 34 and the parameters of the image generation network 35.

[0093] The specific process of training may include Figure 4 Steps S41 to S42 in the embodiment may further include steps S51 to S52 as follows. Figure 4 The different contents are explained.

[0094] In step S51, a weighted operation is performed on the human body information loss function, the first image difference loss function, and the second image difference loss function to obtain a fusion loss function. In a non-limiting manner, the weighted operation may be a weighted summation operation.

[0095] The first image difference loss function and the second image difference loss function are constructed by using the pixel values ​​of the third sample image and the pixel values ​​of the sample interpolated image generated by the image generation network.

[0096] Furthermore, the first image difference loss function can adopt a perceptual loss function (PerceptualLoss), and be constructed with the pixel value of the third sample image and the pixel value of the sample interpolated image as function parameters; the second image difference loss function can adopt an expression of an L1 loss function, and be constructed with the pixel value of the third sample image and the pixel value of the sample interpolated image as parameters.

[0097] The first image difference loss function can be expressed as: V(I,Y), where V() represents the first image difference loss function, I represents the third sample image, and Y represents the sample interpolated image. The second image difference loss function can be expressed as: L1(I,Y), where L1() represents the first image difference loss function, I represents the third sample image, and Y represents the sample interpolated image.

[0098] The fusion loss function can be expressed as: Among them, w1, w2, w3 respectively represent the weight values ​​of the human body information loss function, the image difference first loss function and the image difference second loss function.

[0099] Furthermore, the weight of the human body information loss function is greater than the weights of the first image difference loss function and the second image difference loss function.

[0100] In the embodiment of the present invention, the human body information loss function is mainly used to optimize the parameters of the human body feature prediction network 31, and the network is used to predict the human body prediction features of the target person, which plays a very critical role in improving the clarity of the generated person interpolation image. Therefore, by setting a relatively larger weight for the human body information loss function, it is helpful to obtain a better network optimization effect, and then obtain a better quality person interpolation image.

[0101] In step S52, the fusion loss function is used to input the sample splicing matrix into the interpolation image generation model for iterative training, thereby optimizing the parameters of the networks 31 to 35.

[0102] Specifically, in step S52, each round of iterative training may include the above steps S431 to S433, and may also include steps S521 to S524:

[0103] S521, inputting the sample stitching matrix into the image feature extraction network to perform feature extraction to obtain sample image features;

[0104] S522, inputting the sample image feature and the sample human body prediction feature into the feature fusion network for feature fusion to obtain a sample fusion feature;

[0105] S523, inputting the sample fusion feature into the image generation network for decoding to obtain a sample interpolated frame image;

[0106] S524, respectively calculating the function values ​​of the first image difference loss function and the second image difference loss function according to the pixel values ​​of the third sample image and the pixel values ​​of the sample interpolated frame image (corresponding to Figure 3 The first image difference loss and the second image difference loss in the image difference are obtained by weighting the function values ​​of the human body information loss function, the first image difference loss function and the second image difference loss function to obtain the function value of the fusion loss function. That is, the loss function value obtained in the current round of iteration.

[0107] There may be no particular order in which steps S431 and S521 are performed, and there may be no particular order in which steps S432 and S522 are performed.

[0108] Combine the following Figure 6 The process of generating a person interpolated image using a pre-trained interpolated image generation model is described.

[0109] Reference Figure 6 , Figure 6 It is a schematic diagram of the reasoning process of generating a person interpolation image in a typical application scenario of an embodiment of the present invention.

[0110] Among them, the network structure of the interpolated image generation model used in the inference process includes Figure 3 The human feature prediction network 31, image feature extraction network 33, feature fusion network 34 and image generation network 35 are shown. Each network can be a human feature prediction network 31, image feature extraction network 33, feature fusion network 34 and image generation network 35. Figure 4 or Figure 5 The network after optimization using the training method shown.

[0111] Specifically, the process of applying the interpolation image generation model to generate the interpolation image of the character may include:

[0112] Step 1: obtaining a pixel matrix of a first image and a pixel matrix of a second image, and stitching them together to obtain a first stitching matrix, wherein the first image and the second image contain the same target person;

[0113] Step 2: input the first splicing matrix into the human feature prediction network 31 to predict the human predicted features of the target person, and input the first splicing matrix into the image feature extraction network 33 to extract features to obtain image features, wherein the human predicted features include the predicted human body information of the target person at the time when the person interpolated image belongs;

[0114] The third step is to input the predicted human body features of the target person and the image features into the feature fusion network 34 for feature fusion to obtain fusion features;

[0115] The fourth step is to input the fusion feature into the image generation network 35 for decoding to obtain the character interpolation image.

[0116] In the specific implementation, for more details about the above reasoning process, refer to the previous article and Figures 1 to 5 The relevant description is executed and will not be repeated here.

[0117] It should be pointed out that, in the specific implementation, in addition to using the above-mentioned networks to predict the target person's body prediction features, extracting features to obtain image features, fusing features to obtain fused features, and decoding the fused features to obtain the person's interpolated image, other existing neural networks or algorithms that can achieve the same or similar functions can also be used.

[0118] In addition, the method of training the interpolation image generation model to optimize each network parameter is not limited to Figure 4 and Figure 5The provided solution may also be implemented using other conventional training methods in a specific implementation. For example, when optimizing the parameters of networks 31 to 35, only a single loss function or a loss function obtained by weighted summation of two loss functions may be used as a loss function for training.

[0119] Compared with the existing conventional technology that only relies on image feature extraction and feature fusion operations to obtain the interpolated image of the character, this embodiment additionally introduces the prediction operation of the human body prediction feature (correspondingly, in the interpolated image generation model, it is reflected in the addition of a human body feature prediction network 31 and a human body information prediction network 32). Specifically: based on the splicing matrix of the pixel matrix of the first image and the pixel matrix of the second image, the human body prediction feature of the target person is predicted, and the human body prediction feature contains the predicted human body information of the target person at the moment of the interpolated image of the character (for example, the position information of the key points of the limbs and / or the three-dimensional modeling information of the limbs). Therefore, compared with generating the interpolated image of the character only based on the image features obtained by image feature extraction, this embodiment adds the human body prediction feature, which helps to solve the blur problem of the interpolated image of the character generated by the prior art at the limbs, and obtains a clearer and higher quality interpolated image of the character.

[0120] Furthermore, in the model training stage, by integrating the human body information loss function, the first image difference loss function and the second image difference loss function, that is, using the fusion loss function obtained by weighted operation of the above three loss functions as the loss function for training the model, compared with using only a single loss function or two loss functions, using the training scheme provided by the embodiment of the present invention helps to improve the pertinence of model training and the accuracy of the optimized network parameters, and obtain a more accurate and reliable interpolation image generation model. Furthermore, by applying a more accurate model, it is helpful to obtain a better interpolation image of a person.

[0121] Reference Figure 7 , Figure 7 : is a schematic diagram of the structure of a device for generating a character interpolation image in an embodiment of the present invention. The device may include:

[0122] A pixel matrix stitching module 71 is used to obtain a pixel matrix of a first image and a pixel matrix of a second image, and stitch them together to obtain a first stitching matrix, wherein the first image and the second image contain the same target person;

[0123] The module 72 for determining features to be fused is used to predict the predicted human body features of the target person according to the first splicing matrix, and to extract features from the first splicing matrix to obtain image features, wherein the predicted human body features include predicted human body information of the target person at the time when the person interpolated image belongs.

[0124] A feature fusion module 73 is used to fuse the predicted human body features of the target person and the image features to obtain fused features;

[0125] The character interpolation image determination module 74 is used to decode the fusion feature to obtain the character interpolation image.

[0126] For the principle, specific implementation and beneficial effects of the character interpolation image generation device, please refer to the previous article and Figures 1 to 6 The related description about the method for generating the character interpolation image shown is not repeated here.

[0127] The embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to execute the above Figures 1 to 6 The steps of the method for generating a character interpolation image are shown. The computer-readable storage medium may include a non-volatile memory or a non-transitory memory, and may also include an optical disk, a mechanical hard disk, a solid-state hard disk, etc.

[0128] Specifically, in the embodiment of the present invention, the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0129] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM) and direct rambus RAM (DR RAM).

[0130] The embodiment of the present invention further provides a terminal, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor executes the above-mentioned Figures 1 to 6 The steps of the method for generating a character interpolation image are shown.

[0131] It should be understood that the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article indicates that the associated objects before and after are in an "or" relationship.

[0132] The "plurality" appearing in the embodiments of the present application refers to two or more.

[0133] The first, second, etc. descriptions appearing in the embodiments of the present application are only used for illustration and distinction of the description objects. There is no order, nor do they indicate any special limitation on the number of devices in the embodiments of the present application, and cannot constitute any limitation on the embodiments of the present application.

[0134] It should be pointed out that the serial numbers of the steps in this embodiment do not limit the execution order of the steps.

[0135] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope defined by the claims.

Claims

1. A method for generating a character interpolation image, characterized in that: include: Acquire a pixel matrix of a first image and a pixel matrix of a second image, and stitch them together to obtain a first stitching matrix, wherein the first image and the second image include the same target person; Predicting a human body prediction feature of the target person according to the first splicing matrix, and extracting features from the first splicing matrix to obtain image features, wherein the human body prediction feature includes predicted human body information of the target person at the time when the person interpolated frame image belongs; Performing feature fusion on the human body prediction feature of the target person and the image feature to obtain a fusion feature; Decoding the fusion feature to obtain the interpolated image of the character; Wherein, the predicted human body information is the position information of the limb key points and / or the limb three-dimensional modeling information of the target person; The image features include image pixel feature information of the first image and the second image, and the image pixel feature information is selected from one or more of the following: image background information, body contour information, skin texture information, and clothing color information of the target person.

2. The method according to claim 1, characterized in that The feature fusion is achieved by using a pre-trained feature fusion network, which includes a convolutional network and an attention mechanism network; The predicted human body features of the target person and the image features are subjected to feature fusion to obtain fused features, including: Splicing the predicted human body features of the target person and the image features to obtain a second splicing matrix; Inputting the second splicing matrix into the convolutional network for convolution processing to obtain pre-fusion features; The pre-fused features are input into the attention mechanism network to perform one or more rounds of matrix item-by-item product operations to obtain the fused features.

3. The method according to claim 2, characterized in that The attention mechanism network includes a plurality of attention sub-networks connected in sequence; The pre-fused features are input into the attention mechanism network to perform multiple rounds of matrix item-by-item product operations to obtain the fused features, including: The pre-fused features are used as the input data of the first attention sub-network. Starting from the first attention sub-network, each attention sub-network performs a round of matrix item-by-item product operation on its input data to obtain output data, which is used as the input data of the next attention sub-network. The output data of the last attention sub-network is used as the fused features.

4. The method according to claim 3, characterized in that For each attention sub-network, the matrix product operation is performed on its input data to obtain the output data, including: Performing dimensionality reduction processing on the input data to obtain a corresponding attention matrix; Perform a matrix product operation on the attention matrix and the input data to obtain the output data.

5. The method according to claim 4, characterized in that The following formula is used to perform a matrix product operation on the attention matrix and the input data to obtain the output data: ； ； in, represents the input data of the attention subnetwork, C represents the matrix dimension of the input data, M represents the number of matrix rows of the input data, and N represents the number of matrix columns of the input data. represents the attention matrix obtained by reducing the dimension of F, or , Represents a matrix term-wise product operation between two matrices.

6. The method according to claim 1, characterized in that The method further comprises: Determining a pre-trained interpolation image generation model, the interpolation image generation model comprising: a human feature prediction network and a human information prediction network, the human feature prediction network being used to predict human predicted features of the target person, and the human information prediction network being used to convert the human predicted features into predicted human information; The pre-trained interpolation image generation model is trained in the following manner: Obtaining a pixel matrix of a first sample image and a pixel matrix of a second sample image, and splicing them to obtain a sample splicing matrix; Acquire a third sample image located between the time points of the first sample image and the second sample image, and determine the real sample human body information based on the third sample image, wherein the first sample image, the second sample image, and the third sample image are from the same sample video and contain the same sample target person; At least based on the human body information loss function, inputting the sample splicing matrix into the interpolation image generation model for iterative training to optimize the parameters of the human body feature prediction network and the human body information prediction network; Each round of iterative training includes: Inputting the sample splicing matrix into the human body feature prediction network to predict sample human body prediction features of the sample target person, and then inputting the sample human body prediction features into the human body information prediction network to obtain predicted sample human body information; The function value of the human body information loss function is calculated according to the real sample human body information and the predicted sample human body information.

7. The method according to claim 6, characterized in that The pre-trained interpolation image generation model further includes: an image feature extraction network, a feature fusion network, and an image generation network, wherein the image feature extraction network is used to perform feature extraction to obtain the image features, the feature fusion network is used to perform feature fusion to obtain the fusion features, and the image generation network is used to decode the fusion features to obtain the character interpolation image; At least based on the human body information loss function, the sample splicing matrix is ​​input into the interpolation image generation model for iterative training, including: Performing a weighted operation on the human body information loss function, the first image difference loss function, and the second image difference loss function to obtain a fusion loss function; Using the fusion loss function, the sample splicing matrix is ​​input into the interpolation image generation model for iterative training; Each round of iterative training also includes: Inputting the sample splicing matrix into the image feature extraction network to extract features, thereby obtaining sample image features; Inputting the sample image features and the sample human body prediction features into the feature fusion network for feature fusion to obtain sample fusion features; Inputting the sample fusion features into the image generation network for decoding to obtain a sample interpolated frame image; The function values ​​of the first image difference loss function and the second image difference loss function are respectively calculated according to the pixel value of the third sample image and the pixel value of the sample interpolated image, and the function values ​​of the human body information loss function, the first image difference loss function and the second image difference loss function are weightedly operated to obtain the function value of the fusion loss function.

8. The method according to claim 7, characterized in that Meet one or more of the following: The first image difference loss function is a perceptual loss function; The second image difference loss function is an L1 loss function; The weight of the human body information loss function is greater than the weights of the first image difference loss function and the second image difference loss function.

9. A device for generating a character interpolation image, characterized in that: include: A pixel matrix stitching module, used to obtain a pixel matrix of a first image and a pixel matrix of a second image, and stitch them together to obtain a first stitching matrix, wherein the first image and the second image contain the same target person; a module for determining features to be fused, configured to predict predicted human body features of the target person according to the first splicing matrix, and extract features from the first splicing matrix to obtain image features, wherein the predicted human body features include predicted human body information of the target person at the time when the person interpolated image belongs; A feature fusion module, used for fusing the predicted human features of the target person and the image features to obtain fused features; A character interpolation image determination module, used for decoding the fusion feature to obtain the character interpolation image; Wherein, the predicted human body information is the position information of the limb key points and / or the limb three-dimensional modeling information of the target person; The image features include image pixel feature information of the first image and the second image, and the image pixel feature information is selected from one or more of the following: image background information, body contour information, skin texture information, and clothing color information of the target person.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for generating a person interpolated image according to any one of claims 1 to 8 are executed.

11. A terminal comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor runs the computer program, the steps of the method for generating a character interpolated image according to any one of claims 1 to 8 are performed.

Images