Image generation method, device, electronic device and storage medium

Abstract

The present disclosure provides an image generation method, device, electronic device and storage medium, and relates to the field of image processing technology. The method includes: determining the first depth information of a first image area and the second depth information of a second image area in a first image, the first image area is the image area where the target object is located, and the second image area is the area where the background image other than the target object is located; according to the image data of the second image area, replacing the image data of the first image area to obtain the second image; performing depth filling on the third image area in the second image to obtain the third depth information of the third image area; according to the first depth information and the third depth information, fusing the image data in the first image area into the second image after depth filling to obtain the third image. In this way, when the perspective changes, the background holes can be filled, the distortion or missing at the boundary of the target object can be prevented, and the image effect of the generated three-dimensional image can be optimized.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image generating method, device, electronic device and storage medium. Background Art

[0002] With the development of image processing technology, users can take photos anytime and anywhere. However, when viewing the photos, users can only view flat static scenes and portraits. If users want to view three-dimensional dynamic images, they need to rely on post-production or shoot videos to achieve this.

[0003] In the related art, in order to enable users to view images with three-dimensional effects, a depth estimation method based on binocular vision is often used. Images of the same scene at different angles are taken by dual cameras, and the differences between the images at different angles are determined. The differences between the images at different angles are converted into depth information of the target object in the image (the distance between the target object and the imaging point). The three-dimensional mesh structure of the image is reconstructed based on the depth information. By selecting a fixed angle, the viewing angle corresponding to the image of the three-dimensional mesh structure is rotated, translated or scaled, and the calculated three-dimensional mesh structure is mapped to the plane image to obtain a three-dimensional image display effect.

[0004] In the above-mentioned related technologies, only images at different angles are fused, and the background area is not processed, resulting in distortion or loss of the background area when the angle of the portrait in the image changes, resulting in obvious artificial traces in the image. Summary of the invention

[0005] The embodiments of the present disclosure provide an image generation method, device, electronic device and storage medium, which can optimize the image effect of the generated three-dimensional image. The technical solution is as follows:

[0006] According to one aspect of an embodiment of the present disclosure, there is provided an image generation method, the method comprising:

[0007] Determine first depth information of a first image area and second depth information of a second image area in a first image, wherein the first image area is an image area where a target object is located, and the second image area is an area where a background image other than the target object is located in the first image;

[0008] replacing the image data of the first image area in the first image according to the image data of the second image area to obtain a second image;

[0009] performing depth filling on a third image area in the second image according to the second depth information to obtain third depth information of the third image area, where the third image area is an image area corresponding to the first image area;

[0010] According to the first depth information and the third depth information, the image data in the first image area is fused into the second image after depth filling to obtain a third image.

[0011] In some embodiments, the step of fusing the image data in the first image area into the depth-filled second image to obtain the third image according to the first depth information and the third depth information includes:

[0012] Creating a first three-dimensional model based on the image data of the first image area, where the first three-dimensional model is a three-dimensional model corresponding to the target object;

[0013] Creating a second three-dimensional model based on the second image after depth filling, where the second three-dimensional model is a three-dimensional model corresponding to the background image;

[0014] Based on the first depth information and the third depth information, the pixel information corresponding to the first three-dimensional model and the second three-dimensional model are fused to obtain the third image, wherein the depth information mapped by the pixel points corresponding to the first three-dimensional model in the third image is the first depth information, and the depth information mapped by the pixel points corresponding to the second three-dimensional model in the third image is the third depth information.

[0015] In some embodiments, the fusing pixel information of the first three-dimensional model and the second three-dimensional model based on the first depth information and the third depth information to obtain the third image includes:

[0016] Determine depth information of each pixel of the target object from the first three-dimensional model, wherein the depth information of each pixel is based on depth information of a target key point of the target object, and the target key point is any key point of the target object;

[0017] Determine a target pixel point corresponding to a target key point of the target object in the second three-dimensional model;

[0018] Assigning pixel information of the target key point and first depth information corresponding to the target key point to the target pixel point;

[0019] Determining pixel positions corresponding to other pixel points in the target object in the second three-dimensional model according to a positional relationship between the target key point and other pixel points in the target object;

[0020] The pixel information of other pixel points in the first three-dimensional model and the third depth information corresponding to the other pixel points are assigned to the pixel points at the corresponding pixel positions in the second three-dimensional model to obtain a third image.

[0021] In some embodiments, replacing the image data of the first image area in the first image according to the image data of the second image area to obtain the second image includes:

[0022] Performing image segmentation on the first image using an image segmentation model to determine a region contour corresponding to a region of the first image;

[0023] removing image data corresponding to the first image region within the region outline in the first image;

[0024] The image data of the second image area is input into an image completion model to obtain the second image, and the image completion model is used to perform background filling on the pixel values ​​of the first image area.

[0025] In some embodiments, the method further comprises:

[0026] Acquire multiple first image sets, each first image set corresponds to an image category;

[0027] For each first image set, determining a sampling weight of the first image set according to a first number and a second number, wherein the first number is the number of sample images included in the first image set, the second number is the total number of sample images included in the plurality of first image sets, the sampling weight is positively correlated with the second number, and the sampling weight is negatively correlated with the first number;

[0028] Based on the sampling weight of the first image set, sampling the first image set to obtain a second image set;

[0029] According to a plurality of second image sets, a second depth determination model is trained to obtain the first depth determination model.

[0030] In some embodiments, the first depth determination model includes a feature extraction layer, a feature map generation layer, a feature fusion layer, and a depth determination layer;

[0031] The determining first depth information of a first image area and second depth information of a second image area in the first image includes:

[0032] Inputting the first image to the feature extraction layer, performing multi-layer feature extraction on the first image through the feature extraction layer to obtain multiple image features of the first image;

[0033] The plurality of image features are sampled respectively by the feature map generation layer to obtain a plurality of feature maps of different scales;

[0034] Performing feature fusion on the multiple feature maps through the feature fusion layer to obtain a fused feature map;

[0035] The fused feature map is convolved by the depth determination layer to obtain the first depth information and the second depth information.

[0036] In some embodiments, after fusing the image data in the first image area into the second image after depth filling according to the first depth information and the third depth information to obtain the third image, the method further includes:

[0037] Determine the position coordinates of the special effect element in the image coordinate system of the third image; and determine the depth coordinates of the special effect element in the camera coordinate system of the third image, where the depth coordinates are the coordinate positions corresponding to the depth information of the special effect element in the image in the camera coordinate system;

[0038] The special effect element is integrated into the position coordinates in the image coordinate system and the depth coordinates in the camera coordinate system in the third image to obtain the fourth image.

[0039] In some embodiments, after fusing the image data in the first image area into the second image after depth filling according to the first depth information and the third depth information to obtain the third image, the method further includes:

[0040] Setting the position coordinates corresponding to the target key point of the target object to the coordinate origin of the camera coordinate system corresponding to the third image;

[0041] Determine a rotation angle for rotating in a direction corresponding to each coordinate axis of the camera coordinate system;

[0042] Based on the rotation angle, pixels in the third image are moved to generate an image video.

[0043] In some embodiments, determining a rotation angle for rotating in a direction corresponding to each coordinate axis of the coordinate system includes:

[0044] Obtain a first preset display angle, a preset movement speed, and a preset display frame number of the target pixel point in any direction; determine a display angle weight according to the preset movement speed and the preset display frame number; determine a rotation angle in the direction according to the display angle weight and the first preset display angle.

[0045] According to another aspect of an embodiment of the present disclosure, there is provided an image generating device, the device comprising:

[0046] A first determining unit is configured to determine first depth information of a first image area and second depth information of a second image area in a first image, wherein the first image area is an image area where a target object is located, and the second image area is an area where a background image other than the target object is located in the first image;

[0047] a replacing unit, configured to replace the image data of the first image area in the first image according to the image data of the second image area, so as to obtain a second image;

[0048] a filling unit configured to perform depth filling on a third image area in the second image according to the second depth information to obtain third depth information of the third image area, where the third image area is an image area corresponding to the first image area;

[0049] The first fusion unit is configured to fuse the image data in the first image area into the depth-filled second image according to the first depth information and the third depth information to obtain a third image.

[0050] In some embodiments, the first fusion unit comprises:

[0051] A first creating subunit is configured to create a first three-dimensional model based on the image data of the first image area, wherein the first three-dimensional model is a three-dimensional model corresponding to the target object;

[0052] A second creating subunit is configured to create a second three-dimensional model based on the second image after depth filling, where the second three-dimensional model is a three-dimensional model corresponding to the background image;

[0053] The fusion subunit is configured to perform fusion of pixel information corresponding to the first three-dimensional model and the second three-dimensional model based on the first depth information and the third depth information to obtain the third image, wherein the depth information mapped by the pixel points corresponding to the first three-dimensional model in the third image is the first depth information, and the depth information mapped by the pixel points corresponding to the second three-dimensional model in the third image is the third depth information.

[0054] In some embodiments, the fusion subunit is configured to determine the depth information of each pixel of the target object from the first three-dimensional model, wherein the depth information of each pixel is based on the depth information of the target key point of the target object, and the target key point is any key point of the target object; determine the target pixel corresponding to the target key point of the target object in the second three-dimensional model; assign the pixel information of the target key point and the first depth information corresponding to the target key point to the target pixel; determine the pixel position corresponding to other pixels in the target object in the second three-dimensional model according to the positional relationship between the target key point and other pixels in the target object; assign the pixel information of other pixels in the first three-dimensional model and the third depth information corresponding to the other pixels to the pixel at the corresponding pixel position in the second three-dimensional model to obtain a third image.

[0055] In some embodiments, the replacement unit comprises:

[0056] a segmentation subunit, configured to perform image segmentation on the first image by using an image segmentation model, and determine a region contour corresponding to a region of the first image;

[0057] a removal subunit, configured to remove image data corresponding to the first image region within the region outline in the first image;

[0058] The completion subunit is configured to input the image data of the second image area into an image completion model to obtain the second image, and the image completion model is used to perform background filling on the pixel values ​​of the first image area.

[0059] In some embodiments, the apparatus further comprises:

[0060] An acquisition unit is configured to acquire a plurality of first image sets, each of which corresponds to an image category;

[0061] A second determining unit is configured to determine, for each first image set, a sampling weight of the first image set according to a first number and a second number, wherein the first number is the number of sample images included in the first image set, the second number is the total number of sample images included in the plurality of first image sets, the sampling weight is positively correlated with the second number, and the sampling weight is negatively correlated with the first number;

[0062] A first sampling unit is configured to perform sampling on the first image set based on a sampling weight of the first image set to obtain a second image set;

[0063] The model training unit is configured to train a second depth determination model according to multiple second image sets to obtain the first depth determination model.

[0064] In some embodiments, the first depth determination model includes a feature extraction layer, a feature map generation layer, a feature fusion layer, and a depth determination layer;

[0065] The first determining unit includes:

[0066] a feature extraction subunit, configured to input the first image into the feature extraction layer, perform multi-layer feature extraction on the first image through the feature extraction layer, and obtain a plurality of image features of the first image;

[0067] A sampling subunit is configured to perform sampling processing on the plurality of image features respectively through the feature map generation layer to obtain a plurality of feature maps of different scales;

[0068] A feature fusion subunit is configured to perform feature fusion on the multiple feature maps through the feature fusion layer to obtain a fused feature map;

[0069] The convolution subunit is configured to perform convolution processing on the fused feature map through the depth determination layer to obtain the first depth information and the second depth information.

[0070] In some embodiments, the apparatus further comprises:

[0071] A third determining unit is configured to determine the position coordinates of the special effect element in the image coordinate system of the third image; and determine the depth coordinates of the special effect element in the camera coordinate system of the third image, where the depth coordinates are the coordinate positions corresponding to the depth information of the special effect element in the image in the camera coordinate system;

[0072] The second fusion unit is configured to fuse the special effect element into the position coordinates in the image coordinate system and the depth coordinates in the camera coordinate system in the third image to obtain the fourth image.

[0073] In some embodiments, the apparatus further comprises:

[0074] A coordinate setting unit, configured to set the position coordinates corresponding to the target key point of the target object to the coordinate origin of the camera coordinate system corresponding to the third image;

[0075] A fourth determining unit is configured to determine a rotation angle for rotating in a direction corresponding to each coordinate axis of the camera coordinate system;

[0076] The generating unit is configured to move the pixels in the third image based on the rotation angle to generate an image video.

[0077] In some embodiments, the fourth determining unit includes:

[0078] The acquisition subunit is configured to execute acquisition of a first preset display angle, a preset movement speed, and a preset display frame number of the target pixel point in any direction; determine a display angle weight according to the preset movement speed and the preset display frame number; and determine a rotation angle in the direction according to the display angle weight and the first preset display angle.

[0079] According to another aspect of an embodiment of the present disclosure, an electronic device is provided, wherein the terminal includes a processor and a memory, wherein at least one program code is stored in the memory, and the at least one program code is loaded and executed by the processor to implement the image generation method described in any possible implementation manner mentioned above.

[0080] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, in which at least one program code is stored. The at least one program code is loaded and executed by a processor to implement the image generation method described in any possible implementation manner mentioned above.

[0081] According to another aspect of an embodiment of the present disclosure, a computer program product or a computer program is provided, wherein the computer program product or the computer program comprises a computer program code, wherein the computer program code is stored in a computer-readable storage medium, wherein a processor of a computer device reads the computer program code from the computer-readable storage medium, and the processor executes the computer program code, so that the computer device performs the operations performed in the above-mentioned image generation method.

[0082] In the disclosed embodiment, image filling and depth filling are performed on a first image area in a first image to obtain a second image. Since the second image is obtained after background filling and depth filling, the second image and the first image area where the target object is located in the first image are fused to obtain a third image. When the viewing angle of the third image changes, it can fill the background holes and prevent distortion or missing at the boundary of the target object, thereby optimizing the image effect of the generated image. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0084] Figure 1 is a flow chart of an image generating method provided according to an exemplary embodiment;

[0085] Figure 2 is a flow chart of an image generating method provided according to an exemplary embodiment;

[0086] Figure 3 is a schematic diagram of image processing provided according to an exemplary embodiment;

[0087] Figure 4 is a schematic diagram of image processing provided according to an exemplary embodiment;

[0088] Figure 5 is a schematic diagram of image processing provided according to an exemplary embodiment;

[0089] Figure 6 is a flow chart of an image generating method provided according to an exemplary embodiment;

[0090] Figure 7 is a flow chart of an image generating method provided according to an exemplary embodiment;

[0091] Figure 8 is a schematic diagram of image processing provided according to an exemplary embodiment;

[0092] Fig. 9 is a flow chart of an image generating method provided according to an exemplary embodiment;

[0093] Fig.10 is a schematic diagram of image processing provided according to an exemplary embodiment;

[0094] Fig.11 is a block diagram of an image generating device provided according to an exemplary embodiment;

[0095] Fig.12 It is a schematic structural diagram of an electronic device provided according to an exemplary embodiment. DETAILED DESCRIPTION

[0096] In order to make the objectives, technical solutions and advantages of the present disclosure more clear, the embodiments of the present disclosure will be further described in detail below with reference to the accompanying drawings.

[0097] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0098] In the process of image acquisition, in order to display the acquired image in the form of a three-dimensional image, the electronic device performs image processing on the acquired picture to generate a three-dimensional image for display to the user. The solution provided in the embodiment of the present disclosure is applied in an electronic device, and the electronic device is an electronic device with an image acquisition function. For example, the electronic device is a camera, or the electronic device is a mobile phone, a tablet computer or a wearable device with a camera. In the embodiment of the present disclosure, the electronic device is not specifically limited.

[0099] Figure 1 FIG. 1 is a flow chart of an image generation method provided according to an exemplary embodiment. Figure 1 As shown, the method comprises the following steps:

[0100] Step 101: Determine first depth information of a first image area and second depth information of a second image area in a first image, wherein the first image area is an image area where a target object is located, and the second image area is an area where a background image other than the target object is located in the first image.

[0101] Step 102: replacing the image data of the first image area in the first image according to the image data of the second image area to obtain a second image.

[0102] Step 103: performing depth filling on a third image area in the second image according to the second depth information to obtain third depth information of the third image area, where the third image area is an image area corresponding to the first image area.

[0103] Step 104: According to the first depth information and the third depth information, the image data in the first image area is merged into the second image after depth filling to obtain a third image.

[0104] In some embodiments, the step of fusing the image data in the first image area into the depth-filled second image to obtain the third image according to the first depth information and the third depth information includes:

[0105] Creating a first three-dimensional model based on the image data of the first image area, the first three-dimensional model being a three-dimensional model corresponding to the target object;

[0106] Creating a second three-dimensional model based on the second image after depth filling, where the second three-dimensional model is a three-dimensional model corresponding to the background image;

[0107] Based on the first depth information and the third depth information, the pixel information corresponding to the first three-dimensional model and the second three-dimensional model are fused to obtain the third image, wherein the depth information mapped in the third image by the pixel points corresponding to the first three-dimensional model is the first depth information, and the depth information mapped in the third image by the pixel points corresponding to the second three-dimensional model is the third depth information.

[0108] In some embodiments, based on the first depth information and the third depth information, fusing the pixel information of the first three-dimensional model and the second three-dimensional model to obtain the third image includes:

[0109] Determine depth information of each pixel of the target object from the first three-dimensional model, wherein the depth information of each pixel is based on depth information of a target key point of the target object, and the target key point is any key point of the target object;

[0110] Determine a target pixel point corresponding to a target key point of the target object in the second three-dimensional model;

[0111] Assigning pixel information of the target key point and first depth information corresponding to the target key point to the target pixel point;

[0112] Determine pixel positions corresponding to other pixel points in the target object in the second three-dimensional model according to a positional relationship between the target key point and other pixel points in the target object;

[0113] The pixel information of other pixel points in the first three-dimensional model and the third depth information corresponding to the other pixel points are assigned to the pixel points at the corresponding pixel positions in the second three-dimensional model to obtain a third image.

[0114] In some embodiments, the step of replacing the image data of the first image area in the first image according to the image data of the second image area to obtain the second image includes:

[0115] Performing image segmentation on the first image by using an image segmentation model to determine a region contour corresponding to the first image region;

[0116] Removing image data corresponding to the first image region within the region outline in the first image;

[0117] The image data of the second image area is input into an image completion model to obtain the second image. The image completion model is used to perform background filling on the pixel values ​​of the first image area.

[0118] In some embodiments, the method further comprises:

[0119] Acquire multiple first image sets, each first image set corresponds to an image category;

[0120] For each first image set, determining a sampling weight of the first image set according to a first number and a second number, the first number being the number of sample images included in the first image set, the second number being the total number of sample images included in the plurality of first image sets, the sampling weight being positively correlated with the second number, and the sampling weight being negatively correlated with the first number;

[0121] Based on the sampling weight of the first image set, sampling the first image set to obtain a second image set;

[0122] According to the plurality of second image sets, a second depth determination model is trained to obtain the first depth determination model.

[0123] In some embodiments, the first depth determination model includes a feature extraction layer, a feature map generation layer, a feature fusion layer, and a depth determination layer;

[0124] The determining first depth information of a first image area and second depth information of a second image area in the first image includes:

[0125] The first image is input into the feature extraction layer, and multi-layer feature extraction is performed on the first image through the feature extraction layer to obtain multiple image features of the first image;

[0126] The plurality of image features are sampled and processed respectively by the feature map generation layer to obtain a plurality of feature maps of different scales;

[0127] The feature fusion layer performs feature fusion on the multiple feature maps to obtain a fused feature map;

[0128] The fused feature map is convolved through the depth determination layer to obtain the first depth information and the second depth information.

[0129] In some embodiments, after fusing the image data in the first image area into the second image after depth filling according to the first depth information and the third depth information to obtain the third image, the method further includes:

[0130] Determine the position coordinates of the special effect element in the image coordinate system of the third image; and determine the depth coordinates of the special effect element in the camera coordinate system of the third image, where the depth coordinates are the coordinate positions corresponding to the depth information of the special effect element in the image in the camera coordinate system;

[0131] The special effect element is integrated into the position coordinates of the third image in the image coordinate system and the depth coordinates of the third image in the camera coordinate system to obtain the fourth image.

[0132] In some embodiments, after fusing the image data in the first image area into the second image after depth filling according to the first depth information and the third depth information to obtain the third image, the method further includes:

[0133] Setting the position coordinates corresponding to the target key point of the target object to the coordinate origin of the camera coordinate system corresponding to the third image;

[0134] Determine the rotation angle of each coordinate axis of the camera coordinate system.

[0135] Based on the rotation angle, the pixels in the third image are moved to generate an image video.

[0136] In some embodiments, determining the rotation angle of each coordinate axis of the coordinate system includes:

[0137] Obtain a first preset display angle, a preset movement speed, and a preset display frame number of the target pixel point in any direction; determine a display angle weight according to the preset movement speed and the preset display frame number; determine a rotation angle in the direction according to the display angle weight and the first preset display angle.

[0138] In the disclosed embodiment, image filling and depth filling are performed on a first image area in a first image to obtain a second image. Since the second image is obtained after background filling and depth filling, the second image and the first image area where the target object is located in the first image are fused to obtain a third image. When the viewing angle of the third image changes, it can fill the background holes and prevent distortion or missing at the boundary of the target object, thereby optimizing the image effect of the generated image.

[0139] Figure 2 FIG. 1 is a flow chart of an image generation method according to an exemplary embodiment. In the embodiment disclosed in the present invention, training of a first depth determination model is used as an example for explanation. Figure 2 As shown, the method comprises the following steps:

[0140] Step 201: The electronic device obtains a plurality of first image sets, each of which corresponds to an image category.

[0141] The electronic device acquires multiple images, marks the depth of the pixels in the multiple images, and obtains multiple sample images. The multiple sample images are divided into training data and test data. The training data is used to train the model, and the test data is used to determine whether the trained model meets the requirements.

[0142] In some embodiments, the electronic device directly divides the acquired sample images into training data and test data. In some embodiments, the electronic device classifies the acquired sample images, and then determines the training data and test data corresponding to each image category. For example, the sample images are divided into indoor scene types and outdoor scene types according to the shooting scene of the image; the sample images are divided into human scene types and unmanned scene types according to the image content of the image, etc.

[0143] In this implementation, by classifying sample images, the model can be trained according to the difference in depths under different image categories, thereby improving the accuracy of the trained first depth determination model.

[0144] Step 202: For each first image set, the electronic device determines a sampling weight of the first image set according to the first quantity and the second quantity.

[0145] The first number is the number of sample images included in the first image set, the second number is the total number of sample images included in the multiple first image sets, the sampling weight is positively correlated with the second number, and the sampling weight is negatively correlated with the first number. The electronic device determines the sampling weights of images of different image categories according to the number of sample images in different image collections, and performs model training according to the sampling weights.

[0146] In some embodiments, the electronic device uses the ratio of the second number to the first number as the sampling weight of the image set corresponding to the first number. For example, the second number is K and the first number is k_i, then the sampling weight of the first image set corresponding to the first number is K / k_i. Wherein, i represents the label of the image set. In this implementation, the ratio of the total number of sample images to the number of sample images in each image category is used as the sampling weight of the image category, so that the sampling weight of the image category with a larger number of samples is smaller, and the sampling weight of the image category with a smaller number of samples is larger. This ensures that the data of each sample category is balanced during model training, and prevents deviations in model training.

[0147] Step 203: The electronic device samples the first image set based on the sampling weight of the first image set to obtain a second image set.

[0148] In this step, the electronic device obtains sample images from the plurality of first image sets respectively according to the sampling weight of each first image set, and composes the sample images sampled from each first image set into a second image set.

[0149] In some embodiments, the electronic device determines a third number, which is the total number of expected second image sets. For each first image collection, the electronic device determines the fourth number of images that need to be collected from the first image set based on the sampling weight of the first image set and the third number, and collects the fourth number of images from the first image set. The fourth number of images are continuous adjacent images in the first image set, or the fourth number of images are images randomly sampled from the first image set, or the fourth number of images are images uniformly sampled from the first image set, etc. In the embodiments of the present disclosure, the manner in which the electronic device samples from the first image set is not specifically limited.

[0150] Step 204: The electronic device trains a second depth determination model according to the plurality of second image sets to obtain the first depth determination model.

[0151] The electronic device adjusts the model parameters of the second depth determination model according to the second image set and the vector loss function to obtain a trained first depth determination model. The process is implemented by the following steps (1)-(3), including:

[0152] (1) The electronic device determines a loss value of the second depth determination model based on the second image set and the vector loss function.

[0153] The electronic device performs model training on the second depth determination model. The electronic device inputs the sample images in the second image set into the second depth determination model, outputs the depth information corresponding to the training data, inputs the depth information output by the second depth determination model and the depth information annotated in the sample images in the second image set into the loss function, and obtains the loss value of the second depth determination model.

[0154] In some embodiments, the loss function includes at least one of a depth x-direction loss function, a depth y-direction loss function, a normal vector loss function, and a reverse robust loss function (Reversed HuBer).

[0155] In addition, before this step, the electronic device constructs a second depth determination model. In some embodiments, the electronic device constructs the second depth determination model through a convolutional neural network.

[0156] The structure of the second depth determination model is the same as that of the first depth determination model. Figure 3, the second depth determination model includes a feature extraction layer, a feature map generation layer, a feature fusion layer and a depth determination layer. Each layer in the second depth determination model is composed of a convolutional layer, and each convolutional layer is a convolutional layer of the same structure or a convolutional layer of a different structure. For example, the convolutional layer in the second depth determination model is at least one of Depthwise Convolution (depth convolution structure), Pointwise Convolution (point-by-point convolution structure) or Depthwise-Pointwise Convolution (depth point-by-point convolution structure). In the embodiment of the present disclosure, the structure of the convolutional layer is not specifically limited.

[0157] Among them, the feature extraction layer is composed of four convolution layers. The feature extraction layer is used to perform multi-layer feature extraction on the input image to obtain multiple image features of the image. For example, the image is an ordinary 3-channel image. Correspondingly, the electronic device inputs the 3-channel image into the first convolution layer, and outputs the 3-channel image as a 16-channel image through the first convolution layer; then inputs the 16-channel image into the second convolution layer, and outputs the 16-channel image as a 32-channel image through the second convolution layer; then inputs the 32-channel image into the third convolution layer, and outputs the 32-channel image as a 64-channel image through the third convolution layer; then inputs the 64-channel image into the fourth convolution layer, and outputs the 64-channel image as a 128-channel image through the fourth convolution layer. For images with different numbers of channels, the image features of the image are extracted respectively, so that different image features corresponding to different convolution layers can be obtained.

[0158] The feature map generation layer is used to sample and process multiple image features to obtain multiple feature maps of different scales. The image features of different convolution degrees output by the feature extraction layer are used to determine the features of the local image and the global image in the image, and the relative relationship between the position of each pixel in the image and the global image is recorded, so as to provide local feature information and global feature information to the feature fusion layer and the depth determination layer.

[0159] The feature map generation layer is composed of five convolutional layers. The first to fourth convolutional layers are used to convolve the 128-channel image to obtain the feature sampling information of the 128-channel image; the first to fourth convolutional layers are respectively connected to the fifth convolutional layer, and the convolved image is input to the fifth convolutional layer. The fifth convolutional layer fuses the four received images to obtain a 512-channel image, downsamples the 512-channel image to obtain a 128-channel image, and inputs the 128-channel image to the feature fusion layer.

[0160] The feature fusion layer is used to fuse the multiple feature maps to obtain a fused feature map. The feature fusion layer gradually restores the image resolution and reduces the number of channels, and fuses the features of the feature extraction layer, taking into account the features of different depths in the image.

[0161] Among them, the feature fusion layer includes three convolution layers. The first convolution layer downsamples the 128-channel image to obtain a 64-channel image; the second convolution layer is used to downsample the 64-channel image to obtain a 32-channel image; the third convolution layer is used to sample the 32-channel image to obtain a 16-channel image.

[0162] The depth determination layer is used to determine the depth of the image after convolution processing to obtain the depth information of each pixel in the original image.

[0163] In some embodiments, the electronic device first acquires a sample image and then constructs a second depth determination model; or, the electronic device first constructs a second depth determination model and then acquires a sample image; or, the electronic device simultaneously acquires a sample image and constructs a second depth determination model. That is, the electronic device first executes step 201 and then executes step 202; or, the electronic device first executes step 202 and then executes step 201; or, the electronic device simultaneously executes step 201 and step 202. In the embodiments of the present disclosure, the execution order of step 201 and step 202 is not specifically limited.

[0164] (2) The electronic device updates the model parameters of the second depth determination model through the loss value and the model optimizer to obtain a third depth determination model.

[0165] The optimizer is used to update the model parameters using a stochastic gradient descent method. In this step, the electronic device updates the gradient of the stochastic gradient descent method according to the optimizer. In some embodiments, the gradient value is set as needed, and in the disclosed embodiment, the gradient value is not specifically limited.

[0166] (3) The electronic device determines a loss value of the third depth determination model according to the training data and the vector loss function until the loss value is less than a preset loss value, thereby completing the model training and obtaining the first depth determination model.

[0167] After the electronic device adjusts the model parameters of the second depth determination model, it continues to perform model training on the obtained third depth determination model. The process is similar to steps (1)-(2) and will not be repeated here. After each execution of step (2), the electronic device determines whether the model training is completed based on the loss value of the model. In response to the loss value being not less than the preset loss value, steps (1)-(2) are continued. In response to the loss value being less than the preset loss value, the model training is completed to obtain the first depth determination model.

[0168] In some embodiments, after the electronic device completes model training, it evaluates the prediction result of the first depth determination model. Correspondingly, the electronic device tests the third depth determination model through the test data to obtain the test result of the third depth test determination model. The electronic device uses at least one of the mean relative error algorithm or the root mean squared error algorithm to evaluate the prediction result of the first determination model. The electronic device determines the first depth determination model that has been evaluated as qualified as an available depth determination model. See Figure 4 and Figure 5 , Figure 4 and Figure 5 is a scene depth prediction effect diagram provided according to an exemplary embodiment. Figure 4 and Figure 5 Pixels with the same depth information are marked with the same label, and the more similar the depth information is, the more similar the labeled labels are. For example, different depth information is distinguished by different colors, and the more similar the depth information is, the closer the colors are.

[0169] In some embodiments, the process of training the first depth determination model is performed by other electronic devices, and accordingly, the process of the electronic device acquiring the first depth determination model is: the electronic device sends an acquisition request to other electronic devices, and the acquisition request is used to request to acquire the first depth determination model; the other electronic device acquires the first depth determination model according to the acquisition request, and sends the first depth determination model to the electronic device; the electronic device receives the first depth determination model. Among them, the process of other electronic devices training the first depth determination model is similar to the process of the electronic device training the first depth determination model, which will not be repeated here.

[0170] In an embodiment of the present disclosure, sample data is classified to obtain multiple first image sets, and the sampling weight of each first image set is determined by the number of sample images in each image set and the total number of sample images in the multiple first image sets. Therefore, when sampling, the number of images in each category can be controlled by the number of sample images, ensuring that the weight of the first image set with a larger number of sample images is smaller, and the weight of the first image set with a smaller number of sample images is larger. In this way, the sample images in each category of data are balanced during training to prevent deviations in model training.

[0171] Figure 6 FIG. 1 is a flow chart of an image generation method according to an exemplary embodiment. In the embodiment disclosed in the present invention, an example of processing an image to generate a three-dimensional dynamic image is used for explanation. Figure 6 As shown, the method comprises the following steps:

[0172] Step 601: The electronic device determines first depth information of a first image area and second depth information of a second image area in a first image.

[0173] The first image area is the image area where the target object is located, and the second image area is the area where the background image other than the target object is located in the first image. In some embodiments, the target object is a designated object, a human or other animal face, etc.

[0174] In some embodiments, the electronic device obtains the first depth information and the second depth information through the first depth determination model. The process is: the electronic device inputs the first image into the first depth determination model to obtain the first depth information and the second depth information. The structure of the first depth determination model is the same as that of the second depth determination model. Accordingly, the first depth determination model includes a feature extraction layer, a feature map generation layer, a feature fusion layer and a depth determination layer. This step is implemented by the following steps (1)-(4), including:

[0175] (1) The electronic device inputs the first image into the feature extraction layer, and performs multi-layer feature extraction on the first image through the feature extraction layer to obtain multiple image features of the first image.

[0176] This step is similar to step (1) of step 204, in which the electronic device extracts image features through the feature extraction layer in the second depth determination model, and will not be repeated here.

[0177] (2) The electronic device samples the multiple image features respectively through the feature map generation layer to obtain multiple feature maps of different scales.

[0178] This step is similar to step (1) of step 204, in which the electronic device generates a feature map layer in the feature map generation model through the second depth determination, and the process of generating the feature map is similar, and will not be repeated here.

[0179] (3) The electronic device performs feature fusion on the multiple feature maps through the feature fusion layer to obtain a fused feature map.

[0180] This step is similar to step (1) of step 204, in which the electronic device performs feature fusion through the feature fusion layer in the second depth determination model, and will not be repeated here.

[0181] (4) The electronic device performs convolution processing on the fused feature map through the depth determination layer to obtain the first depth information and the second depth information.

[0182] This step is similar to the process in step (1) of step 204 where the electronic device determines the depth information of the image through the depth determination layer in the second depth determination model, and will not be described in detail here.

[0183] In this implementation, the first depth information and the second depth information of the first image are determined by a pre-trained first depth determination model, thereby shortening the time for determining the first depth information and further improving the imaging speed, so that this solution can be applied to instant imaging scenarios.

[0184] In addition, it should be noted that before this step, the electronic device detects whether there is a target object in the first image. In response to the presence of the target object in the first image, the electronic device executes step 601; in response to the absence of the target object in the first image, step 605 is directly executed to model the first image.

[0185] Another point that needs to be explained is that in response to the presence of a target object in the first image, the electronic device also detects the area ratio of the first image area corresponding to the target object to the first image. In response to the area ratio of the first image area to the first image being greater than a preset threshold, step 601 is executed. In response to the area ratio of the first image area to the first image being not greater than the preset threshold, step 605 is directly executed to model the first image.

[0186] In some embodiments, the first image is any type of image, for example, the first image is an RGB (Red, Green, Blue) three-channel image.

[0187] Step 602: The electronic device replaces the image data of the first image area in the first image according to the image data of the second image area to obtain a second image.

[0188] Image data refers to information such as the position and pixel value of pixels in an image. In this step, the electronic device removes the image data in the first image area through a mask, and then fills the background of the first image area with the second image area in the first image to obtain a second image. This step is implemented by the following steps (1)-(3), including:

[0189] (1) The electronic device performs image segmentation on the first image using an image segmentation model to determine a region contour corresponding to the first image region.

[0190] The image segmentation model is an image segmentation model previously acquired by the electronic device. In some embodiments, the image segmentation model is a mask segmentation model. In this step, the electronic device determines the first image area in the first image through the image segmentation model and marks the area contour corresponding to the first image area.

[0191] (2) The electronic device removes the image data corresponding to the first image area within the outline of the area in the first image.

[0192] The electronic device processes the pixels in the contour area, removes the pixel values ​​of the pixels in the first image area, and obtains an image mask of the first image area. Figure 7 and Figure 8 , Figure 7 and Figure 8 The first image area is the area where the person is located. After masking the first image, we get Figure 7 and Figure 8 The mask image shown on the left.

[0193] (3) The electronic device inputs the image data of the second image area into an image completion model to obtain the second image method, and the image completion model is used to perform background filling on the pixel values ​​of the first image area.

[0194] The electronic device inputs the pixel values ​​of the second image area into the image completion model, so that the image completion model performs background filling on the first image area from which the pixel values ​​have been removed according to the pixel values ​​of the second image area, thereby obtaining a complete background image. Figure 7 and 8 , Figure 7 and Figure 8 The image on the right is a complete background image obtained after completion. The image completion model fills the missing part of the second image with background according to the image data in the second image area, and the electronic device determines the image features of the second image area through the image completion model, and fills the background with the pixel values ​​of the first image area according to the image features of the second image area.

[0195] In this implementation, by performing background filling on the first image area in the first image, a new second image is obtained as the background image, thereby preventing the appearance of a hole area at the boundary of the target object when the viewing angle changes, and optimizing the image effect of the generated three-dimensional image.

[0196] Step 603: the electronic device performs depth filling on a third image area in the second image according to the second depth information to obtain third depth information of the third image area, where the third image area is an image area corresponding to the first image area.

[0197] In this step, the electronic device fills the depth information of the image filled in the second image. Among them, the electronic device diffuses the depth information to the third image area in the second image according to the second depth information to obtain the third depth information. In some embodiments, the diffusion method for diffusing the depth information is a Poisson diffusion method. For example, the electronic device determines the variation law of the depth information between adjacent pixels in the second image area in the second image area, and determines the depth information of each pixel in the filled image area according to the variation law of the depth information; or, the electronic device determines the depth information of the pixel in the contour area, determines the depth information corresponding to the contour area as the depth information of the filled area, and fills the image area according to the depth information.

[0198] In this implementation, the electronic device fills the filled background with depth information so that the filled background image matches the depth of the original second image area, thereby making the generated background more harmonious and the generated three-dimensional image more realistic.

[0199] Step 604: The electronic device creates a first three-dimensional model based on the image data of the first image area, where the first three-dimensional model is a three-dimensional model corresponding to the target object.

[0200] The first three-dimensional model is a three-dimensional model generated based on the image data of the first image area. In some embodiments, the electronic device establishes a three-dimensional model of the first image area according to at least one key point of the target object in the first area. For example, the electronic device identifies at least one key point of the target object in the first area, and generates a first three-dimensional model corresponding to the target object through a three-dimensional model generation algorithm according to the at least one key point. Fig. 9 , Fig. 9 The middle right image is a first three-dimensional model built based on the face image in the left image. For example, if the target object is a face, the at least one key point is a face key point. In some embodiments, the three-dimensional model generation algorithm is a 3DMM (3D Morphable Model, 3D deformation model; 3D, 3 Dimensional, three-dimensional) algorithm. Then the first three-dimensional model is a mesh image model.

[0201] Step 605: The electronic device creates a second three-dimensional model based on the second image after depth filling, and the second three-dimensional model is a three-dimensional model corresponding to the background image.

[0202] This step is similar to step 604 and will not be described again here.

[0203] Step 606: The electronic device fuses the pixel information corresponding to the first three-dimensional model and the second three-dimensional model based on the first depth information and the third depth information to obtain the third image.

[0204] The depth information mapped in the third image by the pixel points corresponding to the first three-dimensional model is the first depth information, and the depth information mapped in the third image by the pixel points corresponding to the second three-dimensional model is the third depth information.

[0205] In this implementation, the third image is generated by fusing the first three-dimensional model and the second three-dimensional model, so that the third image contains a three-dimensional target image and a three-dimensional background image, thereby ensuring that when the perspective changes, the background holes can be filled while preventing distortion or missing at the boundaries of the target object, thereby optimizing the image effect of the generated three-dimensional image.

[0206] In some embodiments, the electronic device determines an empty coordinate system, fuses the first three-dimensional model and the second three-dimensional model into the coordinate system, so that the depth information of the pixel points corresponding to the first three-dimensional model and the second three-dimensional model is the depth information based on the standard of an image coordinate system, and copies the pixel information corresponding to the first three-dimensional model and the second three-dimensional model to the corresponding pixel positions respectively to obtain a third image. In some embodiments, the electronic device establishes a coordinate system according to the first three-dimensional model or the second three-dimensional model, maps the second three-dimensional model or the first three-dimensional model to the coordinate system, and copies the pixel information corresponding to the first three-dimensional model and the second three-dimensional model to the corresponding pixel positions respectively to obtain a third image. In the mapping process, the electronic device determines the positional relationship between other pixel points and the target key point according to the position of any key point in the first three-dimensional model or the second three-dimensional model in the second image and the parameter information between each key point in the first three-dimensional model and the second three-dimensional model, and fuses the first three-dimensional model and the second three-dimensional model according to the positional relationship to obtain a third image. This step is implemented by the following steps (A1)-(A5), including:

[0207] (A1) The electronic device determines the depth information of each pixel of the target object from the first three-dimensional model, where the depth information of each pixel is based on the depth information of a target key point of the target object, where the target key point is any key point of the target object.

[0208] Among them, the depth information of each pixel point is based on the depth information of the target key point of the target object, and the target key point is any key point among the at least one key point. For example, the target key point is the pixel point corresponding to the nose tip in the face image, or the target key point is the center point of the first three-dimensional model. The electronic device selects a target key point from at least one key point of the target object, and determines the depth information of the target key point as the first depth information. The electronic device determines the depth information of each pixel point in the first three-dimensional model relative to the target key point based on the model parameters of the target first three-dimensional model, and obtains the depth information of each pixel point in the first three-dimensional model based on the first depth information of the target key point and the depth information of each pixel point in the first three-dimensional model relative to the target key point. For example, if the first three-dimensional model is a mesh image determined by the 3DMM algorithm, the depth information of each pixel point of the target object is determined according to the parameter information of each point in the mesh image.

[0209] (A2) The electronic device determines a target pixel point corresponding to a target key point of the target object in the second three-dimensional model.

[0210] In this step, the first three-dimensional model and the second three-dimensional model are three-dimensional models corresponding to the target object and the background image in the same first image, so the first three-dimensional model and the second three-dimensional model can be mapped to the same image coordinate system. In this step, the electronic device maps the first three-dimensional model to the second three-dimensional model. In some embodiments, the electronic device selects the center point of the second three-dimensional model as the target pixel point, or the electronic device determines the mapping relationship between the first three-dimensional model and the second three-dimensional model based on the mapping relationship between the first three-dimensional model and the first image, and the mapping relationship between the second three-dimensional model and the first image, and determines the target pixel point corresponding to the target key point from the second three-dimensional model based on the mapping relationship between the first three-dimensional model and the second three-dimensional model.

[0211] (A3) The electronic device assigns the pixel information of the target key point and the first depth information corresponding to the target key point to the target pixel point.

[0212] The pixel information includes information such as the pixel value of the pixel point. The depth information of the target pixel point in the second three-dimensional model of the electronic device is modified to the first depth information of the target key point. For example, the electronic device determines the position of the nose tip in the three-dimensional face as the target key point, and determines the depth information of the pixel point corresponding to the target key point as the first depth information.

[0213] In some embodiments, the electronic device directly assigns the pixel value of the target key point and the first depth information corresponding to the target key point to the target pixel point. In some embodiments, the electronic device sets a new layer on the second three-dimensional model and assigns the position corresponding to the target pixel point in the layer to the pixel value and the first depth information.

[0214] In this implementation, by adding a new layer, the first three-dimensional model and the second three-dimensional model are not affected by each other and an integrated molding effect is achieved, thereby optimizing the image effect of the generated three-dimensional image.

[0215] (A4) The electronic device determines the pixel positions corresponding to other pixel points in the target object in the second three-dimensional model according to the positional relationship between the target key point and other pixel points in the target object.

[0216] In this step, after determining the target pixel point, the electronic device determines the pixel positions of other pixel points in the second three-dimensional model based on the correspondence between the target key point and other pixel points in the first three-dimensional model.

[0217] For example, the electronic device sets the target key point of the first 3D model at the origin of the coordinate system corresponding to the second 3D model, and sets the origin of the coordinate system corresponding to the first 3D model and the second 3D model to the position of the pixel point corresponding to the target key point in the second image.

[0218] (A5) Assigning pixel values ​​of other pixel points in the first three-dimensional model and third depth information corresponding to the other pixel points to pixel points at corresponding pixel positions in the second three-dimensional model to obtain a third image.

[0219] This step is similar to step (A3) and will not be described again here.

[0220] In this implementation, the electronic device fuses the first three-dimensional model and the second three-dimensional model according to the correspondence between different pixel points in the same image, so that when the perspective changes, it can fill the background holes and prevent distortion or missing at the boundary of the target object, thereby optimizing the image effect of the generated three-dimensional image.

[0221] In addition, the electronic device can also add special effects elements to the third image to obtain a fourth image with special effects elements. The process is: the electronic device determines the position coordinates of the special effects element in the image coordinate system of the third image; and determines the depth coordinates of the special effects element in the camera coordinate system of the third image, and the depth coordinates are the coordinate positions corresponding to the depth information of the special effects element in the image in the camera coordinate system; and merges the special effects element into the position coordinates in the image coordinate system and the depth coordinates in the camera coordinate system in the third image to obtain the fourth image.

[0222] Among them, the electronic device converts the pixel position to a coordinate system according to the camera imaging principle. The coordinates in the coordinate system are homogeneous coordinates (X, Y, 1), and the depth of the pixel point in the coordinate system is the distance estimated by the depth map. The depth coordinate 1 of the homogeneous coordinate and the depth Z are multiplied to form the real depth coordinate (X, Y, Z), which is the reconstructed 3D model.

[0223] In this step, the electronic device selects different positions and depths in the three-dimensional image, places different dynamic effects, and obtains a fourth image. Fig.10 , butterfly elements are placed around the face at depths of 1, 2, and 3.5 respectively. This process is similar to (A1)-(A5) in step 606, and will not be repeated here.

[0224] In this implementation, the electronic device adds special effect elements to the third image according to the depth information, so that the added special effect elements are more closely matched and vivid with the third image, thereby optimizing the image effect of the generated three-dimensional image.

[0225] After the electronic device generates the three-dimensional third image, it can also set the motion trajectory of the target object in the three-dimensional image and generate an image video according to the motion trajectory. The process is implemented by the following steps (B1)-(B3), including:

[0226] (B1) The electronic device sets the position coordinates corresponding to the target key point of the target object to the coordinate origin of the camera coordinate system corresponding to the third image.

[0227] In this step, the electronic device sets the position corresponding to the target key point in the generated third image as the coordinate origin of the camera coordinate system.

[0228] (B2) The electronic device determines a rotation angle in a direction corresponding to each coordinate axis of the camera coordinate system.

[0229] In this step, the electronic device determines the rotation angles in different directions respectively. In some embodiments, the rotation angle is a preset rotation angle, or the rotation angle is a rotation angle generated according to a rotation instruction.

[0230] Correspondingly, in some embodiments, the electronic device obtains a first preset display angle, a preset movement speed, and a preset display frame number of the target pixel in any direction; determines a display angle weight based on the preset movement speed and the preset display frame number; and determines a rotation angle in the direction based on the display angle weight and the first preset display angle.

[0231] For example, the first preset display angle in the preset X (or Y) direction is AmpX (or AmpY), t is the preset display frame number, in some embodiments, the preset display frame number is also marked by time, s is the preset movement speed, then each rotation around the X axis is AmpX*sin(s*t) angle (or around the Y axis is AmpY*sin(s*t) angle). Wherein, sin(s*t) is the display angle.

[0232] In this implementation, the display trajectory of the third image is determined by a pre-set motion trajectory, so that the third image can be displayed in rotation along a specified route, thereby preventing the problem of trajectory confusion when the third image generates a video.

[0233] In some embodiments, the electronic device obtains a rotation instruction, and according to the rotation instruction, selects a rotation angle in the direction from a rotation angle corresponding to the rotation instruction and a second preset display angle.

[0234] Wherein, the rotation instruction is an instruction input by the user through the screen and received by the electronic device, or the rotation instruction is an instruction generated by an angle sensor in the electronic device. Accordingly, in some embodiments, the electronic device receives a gesture operation input by the user and determines the rotation angle according to the gesture operation. In other embodiments, the electronic device determines the tilt angle of the current electronic device through an angle sensor, and determines the tilt angle as the rotation angle. For example, if the angle sensor is a gyroscope, the electronic device obtains the gyroscope quaternion attitude according to the electronic device posture, calculates the tilt angles x_anlge and y_angle of the X-axis and Y-axis, rotates min(x_anlge, AmpX) angle around the X-axis, and then rotates min(y_anlge, AmpY) angle around the Y-axis.

[0235] In this implementation, the electronic device determines the motion trajectory of the third image according to the received rotation instruction, so that the motion trajectory of the third image is more flexible.

[0236] (B3) The electronic device moves the pixels in the third image based on the rotation angle to generate an image video.

[0237] In this step, the electronic device translates the coordinate system to the target pixel point, rotates the pixel points in the third image based on the target pixel point and the rotation angle, and obtains an image video. The target key point moves along the above motion trajectory and eventually returns to the initial position, and the above (B2)-(B3) are repeated to obtain a three-dimensional motion video.

[0238] In this implementation, the third image generates a three-dimensional motion video according to the running trajectory, which enriches the image display method.

[0239] In the disclosed embodiment, image filling and depth filling are performed on a first image area in a first image to obtain a second image. Since the second image is obtained after background filling and depth filling, the second image and the first image area where the target object is located in the first image are fused to obtain a third image. When the viewing angle of the third image changes, it can fill the background holes and prevent distortion or missing at the boundary of the target object, thereby optimizing the image effect of the generated image.

[0240] Fig.11 A block diagram of image generation according to an exemplary embodiment. Fig.11 , the device comprises:

[0241] A first determining unit 1101 is configured to determine first depth information of a first image area and second depth information of a second image area in a first image, wherein the first image area is an image area where a target object is located, and the second image area is an area where a background image other than the target object is located in the first image;

[0242] A replacing unit 1102 is configured to replace the image data of the first image area in the first image according to the image data of the second image area to obtain a second image;

[0243] A filling unit 1103 is configured to perform depth filling on a third image area in the second image according to the second depth information to obtain third depth information of the third image area, where the third image area is an image area corresponding to the first image area;

[0244] The first fusion unit 1104 is configured to fuse the image data in the first image area into the second image after depth filling according to the first depth information and the third depth information to obtain a third image.

[0245] In some embodiments, the first fusion unit 1104 includes:

[0246] A first creating subunit is configured to create a first three-dimensional model based on the image data of the first image area, wherein the first three-dimensional model is a three-dimensional model corresponding to the target object;

[0247] A second creating subunit is configured to create a second three-dimensional model based on the second image after depth filling, where the second three-dimensional model is a three-dimensional model corresponding to the background image;

[0248] The fusion subunit is configured to perform fusion of pixel information corresponding to the first three-dimensional model and the second three-dimensional model based on the first depth information and the third depth information to obtain the third image, wherein the depth information mapped by the pixel points corresponding to the first three-dimensional model in the third image is the first depth information, and the depth information mapped by the pixel points corresponding to the second three-dimensional model in the third image is the third depth information.

[0249] In some embodiments, the fusion subunit is configured to determine the depth information of each pixel of the target object from the first three-dimensional model, wherein the depth information of each pixel is based on the depth information of the target key point of the target object, and the target key point is any key point of the target object; determine the target pixel corresponding to the target key point of the target object in the second three-dimensional model; assign the pixel information of the target key point and the first depth information corresponding to the target key point to the target pixel; determine the pixel position corresponding to other pixels in the target object in the second three-dimensional model according to the positional relationship between the target key point and other pixels in the target object; assign the pixel information of other pixels in the first three-dimensional model and the third depth information corresponding to the other pixels to the pixel at the corresponding pixel position in the second three-dimensional model to obtain a third image.

[0250] In some embodiments, the replacement unit 1102 includes:

[0251] a segmentation subunit, configured to perform image segmentation on the first image by using an image segmentation model, and determine a region contour corresponding to a region of the first image;

[0252] a removal subunit, configured to remove image data corresponding to the first image region within the region outline in the first image;

[0253] The completion subunit is configured to input the image data of the second image area into an image completion model to obtain the second image, and the image completion model is used to perform background filling on the pixel values ​​of the first image area.

[0254] In some embodiments, the apparatus further comprises:

[0255] An acquisition unit is configured to acquire a plurality of first image sets, each of which corresponds to an image category;

[0256] A second determining unit is configured to determine, for each first image set, a sampling weight of the first image set according to a first number and a second number, wherein the first number is the number of sample images included in the first image set, the second number is the total number of sample images included in the plurality of first image sets, the sampling weight is positively correlated with the second number, and the sampling weight is negatively correlated with the first number;

[0257] A first sampling unit is configured to perform sampling on the first image set based on a sampling weight of the first image set to obtain a second image set;

[0258] The model training unit is configured to train a second depth determination model according to multiple second image sets to obtain the first depth determination model.

[0259] In some embodiments, the first depth determination model includes a feature extraction layer, a feature map generation layer, a feature fusion layer, and a depth determination layer;

[0260] The first determining unit 1101 includes:

[0261] a feature extraction subunit, configured to input the first image into the feature extraction layer, perform multi-layer feature extraction on the first image through the feature extraction layer, and obtain a plurality of image features of the first image;

[0262] A sampling subunit is configured to perform sampling processing on the plurality of image features respectively through the feature map generation layer to obtain a plurality of feature maps of different scales;

[0263] A feature fusion subunit is configured to perform feature fusion on the multiple feature maps through the feature fusion layer to obtain a fused feature map;

[0264] The convolution subunit is configured to perform convolution processing on the fused feature map through the depth determination layer to obtain the first depth information and the second depth information.

[0265] In some embodiments, the apparatus further comprises:

[0266] A third determining unit is configured to determine the position coordinates of the special effect element in the image coordinate system of the third image; and determine the depth coordinates of the special effect element in the camera coordinate system of the third image, where the depth coordinates are the coordinate positions corresponding to the depth information of the special effect element in the image in the camera coordinate system;

[0267] The second fusion unit is configured to fuse the special effect element into the position coordinates in the image coordinate system and the depth coordinates in the camera coordinate system in the third image to obtain the fourth image.

[0268] In some embodiments, the apparatus further comprises:

[0269] A coordinate setting unit, configured to set the position coordinates corresponding to the target key point of the target object to the coordinate origin of the camera coordinate system corresponding to the third image;

[0270] A fourth determining unit is configured to determine a rotation angle for rotating in a direction corresponding to each coordinate axis of the camera coordinate system;

[0271] The generating unit is configured to move the pixels in the third image based on the rotation angle to generate an image video.

[0272] In some embodiments, the fourth determining unit includes:

[0273] The acquisition subunit is configured to execute acquisition of a first preset display angle, a preset movement speed, and a preset display frame number of the target pixel point in any direction; determine a display angle weight according to the preset movement speed and the preset display frame number; and determine a rotation angle in the direction according to the display angle weight and the first preset display angle.

[0274] In the disclosed embodiment, image filling and depth filling are performed on a first image area in a first image to obtain a second image. Since the second image is obtained after background filling and depth filling, the second image and the first image area where the target object is located in the first image are fused to obtain a third image. When the viewing angle of the third image changes, it can fill the background holes and prevent distortion or missing at the boundary of the target object, thereby optimizing the image effect of the generated image.

[0275] It should be noted that: the image generation device provided in the above embodiment only uses the division of the above functional modules as an example to illustrate the image generation. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the image generation device provided in the above embodiment and the image generation method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0276] Fig.12The block diagram of the structure of an electronic device 1200 provided by an exemplary embodiment of the present disclosure is shown. In some embodiments, the electronic device 1200 is a portable mobile terminal, such as a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer or a desktop computer. The electronic device 1200 may also be called a user device, a portable terminal, a laptop terminal, a desktop terminal, or other names.

[0277] Typically, the electronic device 1200 includes a processor 1201 and a memory 1202 .

[0278] In some embodiments, the processor 1201 includes one or more processing cores, such as a 4-core processor, an 8-core processor, etc. In some embodiments, the processor 1201 is implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). In some embodiments, the processor 1201 also includes a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1201 is integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1201 also includes an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0279] In some embodiments, the memory 1202 includes one or more computer-readable storage media, which are non-transitory. In some embodiments, the memory 1202 also includes a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1202 is used to store at least one instruction, which is used to be executed by the processor 1201 to implement the image generation method provided by the method embodiment of the present disclosure.

[0280] In some embodiments, the electronic device 1200 may further optionally include: a peripheral device interface 1203 and at least one peripheral device. In some embodiments, the processor 1201, the memory 1202 and the peripheral device interface 1203 are connected via a bus or a signal line. In some embodiments, each peripheral device is connected to the peripheral device interface 1203 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1204, a display screen 1205, a camera assembly 1206, an audio circuit 1207, a positioning assembly 1208 and a power supply 1209.

[0281] The peripheral device interface 1203 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1201 and the memory 1202. In some embodiments, the processor 1201, the memory 1202, and the peripheral device interface 1203 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1201, the memory 1202, and the peripheral device interface 1203 are implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0282] The radio frequency circuit 1204 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1204 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1204 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. In some embodiments, the radio frequency circuit 1204 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. In some embodiments, the radio frequency circuit 1204 communicates with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1204 also includes circuits related to NFC (Near Field Communication), which is not limited in the present disclosure.

[0283] The display screen 1205 is used to display a UI (User Interface). In some embodiments, the UI includes graphics, text, icons, videos, and any combination thereof. When the display screen 1205 is a touch display screen, the display screen 1205 also has the ability to collect touch signals on the surface or above the surface of the display screen 1205. In some embodiments, the touch signal is input as a control signal to the processor 1201 for processing. At this time, the display screen 1205 is also used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 1205 is one, which is arranged on the front panel of the electronic device 1200; in other embodiments, the display screen 1205 is at least two, which are arranged on different surfaces of the electronic device 1200 or are folded; in other embodiments, the display screen 1205 is a flexible display screen, which is arranged on a curved surface or a folded surface of the electronic device 1200. Even, the display screen 1205 is also arranged in a non-rectangular irregular shape, that is, a special-shaped screen. In some embodiments, the display screen 1205 is made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0284] The camera assembly 1206 is used to capture images or videos. In some embodiments, the camera assembly 1206 includes a front camera and a rear camera. Typically, the front camera is disposed on the front panel of the terminal, and the rear camera is disposed on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and the VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1206 also includes a flash. In some embodiments, the flash is a monochrome temperature flash, and in some embodiments, the flash is a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which is used for light compensation at different color temperatures.

[0285] In some embodiments, the audio circuit 1207 includes a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 1201 for processing, or input them into the radio frequency circuit 1204 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, in some embodiments, there are multiple microphones, which are respectively arranged at different parts of the electronic device 1200. In some embodiments, the microphone is an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 1201 or the radio frequency circuit 1204 into sound waves. In some embodiments, the speaker is a traditional film speaker, and in some embodiments, the speaker is a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 1207 also includes a headphone jack.

[0286] The positioning component 1208 is used to locate the current geographic location of the electronic device 1200 to implement navigation or LBS (Location Based Service). In some embodiments, the positioning component 1208 is a positioning component based on the US GPS (Global Positioning System), China's Beidou system, or Russia's Galileo system.

[0287] The power supply 1209 is used to power various components in the electronic device 1200. In some embodiments, the power supply 1209 is an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 1209 includes a rechargeable battery, the rechargeable battery is a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged through a wired line, and a wireless rechargeable battery is a battery that is charged through a wireless coil. The rechargeable battery is also used to support fast charging technology.

[0288] In some embodiments, the electronic device 1200 further includes one or more sensors 1210 , including but not limited to: an acceleration sensor 1211 , a gyroscope sensor 1212 , a pressure sensor 1213 , a fingerprint sensor 1214 , an optical sensor 1215 , and a proximity sensor 1216 .

[0289] In some embodiments, the acceleration sensor 1211 detects the magnitude of acceleration on the three coordinate axes of the coordinate system established by the electronic device 1200. For example, the acceleration sensor 1211 is used to detect the components of gravity acceleration on the three coordinate axes. In some embodiments, the processor 1201 controls the display screen 1205 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 1211. In some embodiments, the acceleration sensor 1211 is also used to collect motion data of games or users.

[0290] In some embodiments, the gyro sensor 1212 detects the body direction and rotation angle of the electronic device 1200, and the gyro sensor 1212 cooperates with the acceleration sensor 1211 to collect the user's 3D actions on the electronic device 1200. The processor 1201 can implement the following functions based on the data collected by the gyro sensor 1212: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0291] In some embodiments, the pressure sensor 1213 is disposed on the side frame of the electronic device 1200 and / or the lower layer of the display screen 1205. When the pressure sensor 1213 is disposed on the side frame of the electronic device 1200, it is capable of detecting the user's holding signal of the electronic device 1200, and the processor 1201 performs left and right hand recognition or shortcut operations according to the holding signal collected by the pressure sensor 1213. When the pressure sensor 1213 is disposed on the lower layer of the display screen 1205, the processor 1201 controls the operability controls on the UI interface according to the user's pressure operation on the display screen 1205. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0292] The fingerprint sensor 1214 is used to collect the user's fingerprint, and the processor 1201 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 1214, or the fingerprint sensor 1214 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as a trusted identity, the processor 1201 authorizes the user to perform relevant sensitive operations, which include unlocking the screen, viewing encrypted information, downloading software, paying, and changing settings. In some embodiments, the fingerprint sensor 1214 is set on the front, back, or side of the electronic device 1200. When a physical button or a manufacturer logo is set on the electronic device 1200, the fingerprint sensor 1214 is integrated with the physical button or the manufacturer logo.

[0293] The optical sensor 1215 is used to collect the ambient light intensity. In one embodiment, the processor 1201 controls the display brightness of the display screen 1205 according to the ambient light intensity collected by the optical sensor 1215. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1205 is increased; when the ambient light intensity is low, the display brightness of the display screen 1205 is decreased. In another embodiment, the processor 1201 also dynamically adjusts the shooting parameters of the camera assembly 1206 according to the ambient light intensity collected by the optical sensor 1215.

[0294] The proximity sensor 1216, also called a distance sensor, is usually disposed on the front panel of the electronic device 1200. The proximity sensor 1216 is used to collect the distance between the user and the front of the electronic device 1200. In one embodiment, when the proximity sensor 1216 detects that the distance between the user and the front of the electronic device 1200 is gradually decreasing, the processor 1201 controls the display screen 1205 to switch from the screen-on state to the screen-off state; when the proximity sensor 1216 detects that the distance between the user and the front of the electronic device 1200 is gradually increasing, the processor 1201 controls the display screen 1205 to switch from the screen-off state to the screen-on state.

[0295] Those skilled in the art will understand that Fig.12 The structure shown in the figure does not constitute a limitation on the electronic device 1200, and it can include more or less components than those shown in the figure, or combine certain components, or adopt a different component arrangement.

[0296] In an exemplary embodiment, a computer-readable storage medium is also provided, in which at least one program code is stored, and the at least one program code is loaded and executed by a server to implement the image generation method in the above embodiment. In some embodiments, the computer-readable storage medium is a memory. For example, the computer-readable storage medium is a ROM (Read-Only Memory), a RAM (Random Access Memory), a CD-ROM (Compact Disc Read-Only Memory), a magnetic tape, a floppy disk, and an optical data storage device.

[0297] In an exemplary embodiment, a computer program product or a computer program is also provided, which includes a computer program code, and the computer program code is stored in a computer-readable storage medium. A processor of a computer device reads the computer program code from the computer-readable storage medium, and the processor executes the computer program code, so that the computer device performs the operations performed in the above-mentioned image generation method.

[0298] A person of ordinary skill in the art will understand that all or part of the steps for implementing the above embodiments can be accomplished through hardware, or through hardware related to program code, where the program is stored in a computer-readable storage medium, and the above-mentioned storage medium is a read-only memory, a disk or an optical disk, etc.

[0299] The above are merely optional embodiments of the present disclosure and are not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An image generation method, characterized in that: The method comprises: Determine first depth information of a first image area and second depth information of a second image area in a first image, wherein the first image area is an image area where a target object is located, and the second image area is an area where a background image other than the target object is located in the first image; replacing the image data of the first image area in the first image according to the image data of the second image area to obtain a second image; performing depth filling on a third image area in the second image according to the second depth information to obtain third depth information of the third image area, where the third image area is an image area corresponding to the first image area; Creating a first three-dimensional model based on the image data of the first image area, where the first three-dimensional model is a three-dimensional model corresponding to the target object; Creating a second three-dimensional model based on the second image after depth filling, where the second three-dimensional model is a three-dimensional model corresponding to the background image; Based on the first depth information and the third depth information, the pixel information corresponding to the first three-dimensional model and the second three-dimensional model are fused to obtain a third image, wherein the depth information mapped by the pixel points corresponding to the first three-dimensional model in the third image is the first depth information, and the depth information mapped by the pixel points corresponding to the second three-dimensional model in the third image is the third depth information.

2. The method according to claim 1, characterized in that The fusing pixel information corresponding to the first three-dimensional model and the second three-dimensional model based on the first depth information and the third depth information to obtain a third image includes: Determine depth information of each pixel of the target object from the first three-dimensional model, wherein the depth information of each pixel is based on depth information of a target key point of the target object, and the target key point is any key point of the target object; Determine a target pixel point corresponding to a target key point of the target object in the second three-dimensional model; Assigning pixel information of the target key point and first depth information corresponding to the target key point to the target pixel point; Determining pixel positions corresponding to other pixel points in the target object in the second three-dimensional model according to a positional relationship between the target key point and other pixel points in the target object; Assign pixel information of other pixel points in the first three-dimensional model and third depth information corresponding to the other pixel points to pixel points at corresponding pixel positions in the second three-dimensional model to obtain a third image.

3. The method according to claim 1, characterized in that The step of replacing the image data of the first image area in the first image according to the image data of the second image area to obtain the second image comprises: Performing image segmentation on the first image using an image segmentation model to determine a region contour corresponding to a region of the first image; removing image data corresponding to the first image region within the region outline in the first image; The image data of the second image area is input into an image completion model to obtain the second image, and the image completion model is used to perform background filling on the pixel values ​​of the first image area.

4. The method according to claim 1, characterized in that: The method further comprises: Acquire multiple first image sets, each first image set corresponds to an image category; For each first image set, determining a sampling weight of the first image set according to a first number and a second number, wherein the first number is the number of sample images included in the first image set, the second number is the total number of sample images included in the plurality of first image sets, the sampling weight is positively correlated with the second number, and the sampling weight is negatively correlated with the first number; Based on the sampling weight of the first image set, sampling the first image set to obtain a second image set; According to a plurality of second image sets, a second depth determination model is trained to obtain the first depth determination model.

5. The method according to claim 4, characterized in that The first depth determination model includes a feature extraction layer, a feature map generation layer, a feature fusion layer and a depth determination layer; The determining first depth information of a first image area and second depth information of a second image area in the first image includes: Inputting the first image to the feature extraction layer, performing multi-layer feature extraction on the first image through the feature extraction layer to obtain multiple image features of the first image; The plurality of image features are sampled respectively by the feature map generation layer to obtain a plurality of feature maps of different scales; Performing feature fusion on the multiple feature maps through the feature fusion layer to obtain a fused feature map; The fused feature map is convolved by the depth determination layer to obtain the first depth information and the second depth information.

6. The method according to claim 1, characterized in that After obtaining the third image, the method further includes: Determine the position coordinates of the special effect element in the image coordinate system of the third image; and determine the depth coordinates of the special effect element in the camera coordinate system of the third image, where the depth coordinates are the coordinate positions corresponding to the depth information of the special effect element in the image in the camera coordinate system; The special effect element is integrated into the position coordinates in the image coordinate system and the depth coordinates in the camera coordinate system in the third image to obtain a fourth image.

7. The method according to claim 1, characterized in that After obtaining the third image, the method further includes: Setting the position coordinates corresponding to the target key point of the target object to the coordinate origin of the camera coordinate system corresponding to the third image; Determine a rotation angle for rotating in a direction corresponding to each coordinate axis of the camera coordinate system; Based on the rotation angle, pixels in the third image are moved to generate an image video.

8. The method according to claim 7, characterized in that The determining of the rotation angle for rotating in the direction corresponding to each coordinate axis of the camera coordinate system includes: Obtain a first preset display angle, a preset movement speed, and a preset display frame number of a target pixel point in any direction; determine a display angle weight according to the preset movement speed and the preset display frame number; determine a rotation angle in the direction according to the display angle weight and the first preset display angle.

9. An image generating device, characterized in that: The device comprises: A first determining unit is configured to determine first depth information of a first image area and second depth information of a second image area in a first image, wherein the first image area is an image area where a target object is located, and the second image area is an area where a background image other than the target object is located in the first image; a replacing unit, configured to replace the image data of the first image area in the first image according to the image data of the second image area, so as to obtain a second image; a filling unit configured to perform depth filling on a third image area in the second image according to the second depth information to obtain third depth information of the third image area, where the third image area is an image area corresponding to the first image area; The first fusion unit includes: a first creation subunit, configured to create a first three-dimensional model based on the image data of the first image area, the first three-dimensional model is a three-dimensional model corresponding to the target object; a second creation subunit, configured to create a second three-dimensional model based on the second image after depth filling, the second three-dimensional model is a three-dimensional model corresponding to the background image; a fusion subunit, configured to perform fusion of pixel information corresponding to the first three-dimensional model and the second three-dimensional model based on the first depth information and the third depth information to obtain a third image, wherein the depth information mapped by the pixel points corresponding to the first three-dimensional model in the third image is the first depth information, and the depth information mapped by the pixel points corresponding to the second three-dimensional model in the third image is the third depth information.

10. The device according to claim 9, characterized in that The fusion subunit is configured to determine the depth information of each pixel of the target object from the first three-dimensional model, wherein the depth information of each pixel is based on the depth information of a target key point of the target object, and the target key point is any key point of the target object; Determine a target pixel point corresponding to a target key point of the target object in the second three-dimensional model; Assigning pixel information of the target key point and first depth information corresponding to the target key point to the target pixel point; According to the positional relationship between the target key point and other pixel points in the target object, the pixel positions corresponding to the other pixel points in the target object in the second three-dimensional model are determined; the pixel information of other pixel points in the first three-dimensional model and the third depth information corresponding to the other pixel points are assigned to the pixel points at the corresponding pixel positions in the second three-dimensional model to obtain a third image.

11. The device according to claim 9, characterized in that The replacement unit comprises: a segmentation subunit, configured to perform image segmentation on the first image by using an image segmentation model, and determine a region contour corresponding to a region of the first image; a removal subunit, configured to remove image data corresponding to the first image region within the region outline in the first image; The completion subunit is configured to input the image data of the second image area into an image completion model to obtain the second image, and the image completion model is used to perform background filling on the pixel values ​​of the first image area.

12. The device according to claim 9, characterized in that The device also includes: An acquisition unit is configured to acquire a plurality of first image sets, each of which corresponds to an image category; A second determining unit is configured to determine, for each first image set, a sampling weight of the first image set according to a first number and a second number, wherein the first number is the number of sample images included in the first image set, the second number is the total number of sample images included in the plurality of first image sets, the sampling weight is positively correlated with the second number, and the sampling weight is negatively correlated with the first number; A first sampling unit is configured to perform sampling on the first image set based on a sampling weight of the first image set to obtain a second image set; The model training unit is configured to train a second depth determination model according to multiple second image sets to obtain the first depth determination model.

13. The device according to claim 12, characterized in that The first depth determination model includes a feature extraction layer, a feature map generation layer, a feature fusion layer and a depth determination layer; The first determining unit includes: a feature extraction subunit, configured to input the first image into the feature extraction layer, perform multi-layer feature extraction on the first image through the feature extraction layer, and obtain a plurality of image features of the first image; A sampling subunit is configured to perform sampling processing on the plurality of image features respectively through the feature map generation layer to obtain a plurality of feature maps of different scales; A feature fusion subunit is configured to perform feature fusion on the multiple feature maps through the feature fusion layer to obtain a fused feature map; The convolution subunit is configured to perform convolution processing on the fused feature map through the depth determination layer to obtain the first depth information and the second depth information.

14. The device according to claim 9, characterized in that The device also includes: A third determining unit is configured to determine the position coordinates of the special effect element in the image coordinate system of the third image; and determine the depth coordinates of the special effect element in the camera coordinate system of the third image, where the depth coordinates are the coordinate positions corresponding to the depth information of the special effect element in the image in the camera coordinate system; The second fusion unit is configured to fuse the special effect element into the position coordinates in the image coordinate system and the depth coordinates in the camera coordinate system in the third image to obtain a fourth image.

15. The device according to claim 9, characterized in that The device also includes: A coordinate setting unit, configured to set the position coordinates corresponding to the target key point of the target object to the coordinate origin of the camera coordinate system corresponding to the third image; A fourth determining unit is configured to determine a rotation angle for rotating in a direction corresponding to each coordinate axis of the camera coordinate system; The generating unit is configured to move the pixels in the third image based on the rotation angle to generate an image video.

16. The device according to claim 15, characterized in that The fourth determining unit includes: The acquisition subunit is configured to execute the acquisition of a first preset display angle, a preset movement speed and a preset display frame number of a target pixel point in any direction; determine a display angle weight according to the preset movement speed and the preset display frame number; and determine a rotation angle in the direction according to the display angle weight and the first preset display angle.

17. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one program code, and the at least one program code is loaded and executed by the processor to implement the image generation method according to any one of claims 1 to 8.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one program code, and the at least one program code is loaded and executed by a processor to implement the image generation method according to any one of claims 1 to 8.

19. A computer program product, characterized in that The computer program product includes a computer program code, which is stored in a computer-readable storage medium. A processor of a computer device reads the computer program code from the computer-readable storage medium, and the processor executes the computer program code, so that the computer device implements the image generation method according to any one of claims 1 to 8.

Images