Method for generating a new perspective image, electronic device

Abstract

Embodiments of the present application disclose a method for generating a new-view image and an electronic device. The electronic device can obtain a foreground image and a background image according to a single-view image, repair the background image, and then generate a new-view image according to the foreground image and the repaired background image. The method repairs the missing part in the background and ensures the quality of the background part in the new-view image.

Description

Technical Field

[0001] This application relates to the fields of terminals and image processing, and in particular to methods and electronic devices for generating images from new perspectives. Background Technology

[0002] Novel view synthesis (NVS) is an important application area of ​​computer vision, aiming to synthesize images from unknown perspectives from images with known perspectives. Neural radiance field (NeRF) is a technique in NVS that utilizes large-scale multi-view images for learning, generating near-photographic quality novel view images; however, its dependence on multi-view images limits its usability. Some novel view synthesis methods based on sparse perspectives can use only a single-view image as input, but the generated novel view images do not achieve photographic quality. Summary of the Invention

[0003] This application provides a method and electronic device for generating new perspective images, which can repair missing parts in the background and ensure the quality of the background part in the new perspective image.

[0004] In a first aspect, a method for generating a new perspective image is provided, applied to an electronic device. The method may include: acquiring a foreground image and a background image of a first image, wherein the perspective of the first image is a first perspective; repairing the background image; and generating a second image with a second perspective, which is different from the first perspective, based on the foreground image and the repaired background image.

[0005] By implementing the first approach, electronic devices can repair missing parts in the background, ensuring the quality of the background portion in the new perspective image.

[0006] In conjunction with the first aspect, in some embodiments, before generating a second image from a second perspective based on the foreground image and the repaired background image, the method further includes: determining the MPI information of each of N sub-layers of the repaired background image, where each sub-layer corresponds to a depth, and the MPI information of a sub-layer includes: the depth of the sub-layer in the first perspective, the color and volume density of each pixel in the sub-layer, and N is a preset value greater than 1; determining the MPI information of a sub-layer of the foreground image, where the MPI information of a sub-layer of the foreground image includes: the depth of the sub-layer in the first perspective, the color and volume density of each pixel in the sub-layer;

[0007] Based on the foreground image and the repaired background image, a second image from a first-person perspective is generated. Specifically, based on the principle of volume rendering, the second image is generated according to the MPI information of each of the N sub-layers of the repaired background image, the MPI information of one sub-layer of the foreground image, and the second-person perspective.

[0008] In some implementations, determining the MPI information of each of the N sub-layers of the restored background image specifically includes: determining the disparity value of each of the N sub-layers of the restored background image, wherein the disparity value of a sub-layer is equal to the reciprocal of the depth of the sub-layer; and determining the color and volume density of pixels in each of the N sub-layers based on the disparity values ​​of the N sub-layers and the restored background image.

[0009] In some implementations, electronic devices can use a trained network model to determine disparity values, color, and volume density.

[0010] In some implementations, determining the disparity values ​​of each of the N sub-layers of the restored background image specifically includes: inputting the restored background image into a sub-layer disparity prediction network to obtain the disparity values ​​of each of the N sub-layers of the restored background image.

[0011] In some implementations, determining the MPI information of a sublayer of the foreground image specifically includes one or more of the following:

[0012] 1. Determine the color of the foreground pixel in a sublayer of the foreground image as the color of the foreground pixel in the first image.

[0013] 2. Determine the minimum depth among the N sub-layers of the repaired background image, and determine the depth of a sub-layer of the foreground image as the depth of any depth less than the minimum depth among the N sub-layers.

[0014] Alternatively, determine the MPI information of each of the Y sub-layers of the first image. Each sub-layer corresponds to a depth. The MPI information of a sub-layer includes: the depth of the sub-layer, the color of each pixel in the sub-layer, and the volume density. Y is a preset value greater than 1. Based on the depth and volume density in the MPI information of each of the Y sub-layers of the first image, determine the weight map of each of the Y sub-layers of the first image. The weight map of a sub-layer includes the contribution of the color of each pixel in the corresponding sub-layer to the color of the first image. Match the weight maps of each of the Y sub-layers with the foreground segmentation map of the first image to determine the first depth. The first depth is the depth of the sub-layer whose weight map is most similar to the foreground segmentation map of the first image. The first depth is determined as the depth of a sub-layer of the foreground image.

[0015] In some implementations, the weight maps of the Y sub-layers of the first image are determined according to the following formula:

[0016] ,

[0017] in, This represents the weight of a pixel in the i-th sublayer. This represents the volume density of pixels in the i-th sublayer. This represents the distance between the i-th sublayer and the (i+1)-th sublayer.

[0018] 3. If the depth of a sublayer of the foreground image is the minimum depth among N+1 sublayers, then the volume density of pixels in a sublayer of the foreground image is determined to be 10. 3 The N+1 sub-layers include: one sub-layer of the foreground image and N sub-layers of the background image;

[0019] If the depth of a sublayer of the foreground image is not the minimum depth among the N+1 sublayers, then the volume density of the pixels in a sublayer of the foreground image is determined as the first value, the product of the first value and the second value is the preset value, and the second value is the distance between a sublayer of the foreground image and the nearest sublayer among the N sublayers of the background image.

[0020] In conjunction with the first aspect, in some embodiments, before acquiring the foreground and background images of the first image, the method further includes: displaying the first image; and receiving a user operation for generating a new perspective image of the first image.

[0021] In conjunction with the first aspect, in some implementations, the first image may be an image captured by an electronic device, an image downloaded by the electronic device from a network, or an image received by the electronic device from another device.

[0022] In a second aspect, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method provided as in the first aspect or any embodiment of the first aspect.

[0023] Thirdly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method provided as in the first aspect or any embodiment of the first aspect.

[0024] Fourthly, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method provided as in the first aspect or any embodiment of the first aspect.

[0025] Fifthly, a chip system is provided, the chip system including a processor and a memory, the memory storing a computer program, which, when executed by the processor, implements the method provided as in the first aspect or any embodiment of the first aspect. Attached Figure Description

[0026] Figure 1A flowchart illustrating the method for generating new perspective images provided in embodiments of this application;

[0027] Figure 2 An example is shown, comprising a first image, a segmented foreground image and background image, and a repaired background image;

[0028] Figure 3 The process for determining the MPI information of the repaired background image provided in the embodiments of this application;

[0029] Figure 4 A flowchart for determining the depth of a sub-layer of a foreground image based on the principle of volume rendering, provided for embodiments of this application;

[0030] Figure 5 An example is shown for Figure 2 A second image, generated from a new perspective, is shown as the first image.

[0031] Figure 6 A hardware structure block diagram of the electronic device provided in the embodiments of this application;

[0032] Figure 7 A software architecture block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0033] The technical solutions in the embodiments of this application will be clearly and thoroughly described below with reference to the accompanying drawings.

[0034] Multi-plane images (MPI) are a method for generating new perspective images based on a single-view image. MPI references a single-view image and uses a discrete approach to represent limited information, which can alleviate the overfitting problem of NeRF and allow the generated new perspective images to achieve photorealistic quality.

[0035] When generating a new perspective image using MPI, the electronic device first uses a neural network (such as a convolutional neural network) to predict the MPI information of the single-view image. This MPI information includes the depth and MPI information of N sub-layers. The MPI information of a sub-layer includes the color value (RGB value) and opacity value (σ) of each pixel. When it is necessary to synthesize an image from a new perspective, the position of each pixel in the scene is first calculated based on the new camera parameters (such as position and orientation). Then, the RGB values ​​and σ are extracted from the sub-layers based on the pixel positions, interpolated, and blended to generate the pixel values ​​under the new perspective. Finally, all pixel values ​​are combined to form a complete new perspective image.

[0036] However, the new perspective images generated by MPI may also have the following problems:

[0037] 1. Voiding phenomenon.

[0038] During the generation of new perspectives, the change in viewpoint causes previously occluded areas in the background to be observed again. However, the geometric and color data of these areas are missing during the synthesis of the new perspective image. MPI-based new perspective image generation methods fit the source image very well, but their ability to generate occluded content is poor, resulting in holes or artifacts in the new perspective image.

[0039] 2. Structural misalignment.

[0040] The limited capabilities of deep neural networks may result in insufficient accuracy in depth estimation when using them to predict the depth of input images. This can lead to problems such as structural misalignment and repetition in images from new perspectives, which can degrade foreground quality and cause confusion in the relative positions of foreground and background.

[0041] The foreground refers to the main object or area of ​​interest to the user in an image. It could be a person, animal, vehicle, building, or other object with a specific shape, color, or texture. Foregrounds typically have visual features that differ from the background, such as color, brightness, and texture, which allows them to be distinguished from the background.

[0042] The background is everything in an image except the foreground. It typically provides information about the image's environment, such as the scene, location, and time. A background can contain various elements, such as sky, trees, grass, and buildings.

[0043] To address the aforementioned technical problems, embodiments of this application provide a method for generating images from new perspectives. This method can be applied to smart electronic devices, such as mobile phones, tablets, desktop computers, desktop computers with touch-sensitive surfaces or touch panels, laptops, handheld computers, smart screens, wearable devices (such as smartwatches, smart bracelets, etc.), augmented reality (AR) devices, virtual reality (VR) devices, artificial intelligence (AI) devices, in-vehicle systems, game consoles, and so on.

[0044] The method provided in this application is based on volumetric rendering technology. Volumetric rendering describes the process of synthesizing a final image by accumulating volume density and color information along the light path.

[0045] Volume rendering formula reference formula 1:

[0046] , Formula 1

[0047] in,

[0048] This represents the color rendered on the image by a ray r that originates from the camera's optical center and passes through the 3D scene;

[0049] N is the number of sampling points, representing the total number of samples taken along the light path;

[0050] This represents the transmittance of the medium from the optical center of the camera to the i-th sampling point, which is the probability that light does not encounter any particles before reaching that point;

[0051] This represents the probability that the light ray encounters a particle at the i-th sampling point, i.e., the volume density;

[0052] It is the distance between the (i+1)th sampling point and the ith sampling point;

[0053] It is the color of the i-th sampling point.

[0054] The color of each pixel in the image can be calculated using Formula 1. A sampling point refers to the point in the segment through which the ray r passes.

[0055] When the position of the camera's optical center is different , It also changes depending on the position of the camera's optical center.

[0056] This formula takes into account the attenuation and color mixing effects of light as it passes through a 3D data volume due to the density and color of the voxels. Specifically, a certain proportion of light is absorbed at each sampling point (by...). (This indicates that the remaining light continues to propagate forward and mixes with the color of the next sample point.) Ultimately, the contributions of the light at all sample points are summed up to form the final rendered color.

[0057] As can be seen, the volume rendering formula synthesizes the final color by accumulating the contribution of each sampling point along the ray path. The contribution of each sampling point is determined by its transmittance and opacity (from...). (Calculated) and color together determine it.

[0058] Figure 1 A flowchart illustrating a method for generating new perspective images provided in an embodiment of this application. Figure 1 As shown, the method may include the following steps:

[0059] S101, the electronic device receives a user operation to generate a new perspective image or a new perspective video of the first image.

[0060] The first image is a single-view image, which can be taken by an electronic device, downloaded by the electronic device from the network, or received by the electronic device from other devices.

[0061] The first image contains multiple pixels, each with a corresponding color (RGB value). That is, the first image is an RGB image.

[0062] The resolution of the first image can be called the first resolution.

[0063] The first image can be seen as the view of the 3D scene from a first-person perspective. The first-person perspective corresponds to the optical center position of a camera. By taking a picture of the 3D scene from this optical center position, the first image can be obtained.

[0064] In some implementations, the electronic device first displays a first image, then receives user operations on that first image and responds to those operations to generate a new perspective image. For example, when the electronic device displays the first image in a gallery, it can simultaneously provide several operation controls, such as zoom, drag left, drag right, and rotate functions. The electronic device can receive user operations applied to these functions and respond to those operations to generate an image or video with the corresponding new perspective. As another example, when the electronic device displays the first image in a gallery, it can receive drag operations (such as zoom, drag left, drag right, rotate, etc.) applied to that first image and respond to those operations to generate an image or video with the corresponding new perspective.

[0065] In some implementations, the electronic device first displays a first image, then receives a user operation on that first image and responds to the user operation to generate a video from a new perspective. This implementation differs from the previous one in that the user operation triggers the electronic device to generate video rather than an image.

[0066] The following steps will explain how electronic devices can generate images or videos from new perspectives.

[0067] S102, the first image is segmented into a foreground image and a background image.

[0068] Electronic devices can employ image segmentation methods to divide a first image into a foreground image and a background image. Image segmentation methods may include, but are not limited to, thresholding, edge detection, region growing, graph cut algorithms, and deep learning-based segmentation methods.

[0069] In practical implementation, the electronic device can first acquire the foreground segmentation map of the first image, then acquire the mask map based on the foreground segmentation map, and finally obtain the foreground image and the segmented image based on the mask map and the first image. The foreground segmentation map includes the probability that each pixel in the first image is foreground, with a probability range of (0, 1). The mask map is obtained from the foreground segmentation map. When the probability that a pixel in the foreground segmentation map is foreground is greater than a certain value, the value of that pixel in the mask map is set to 1; when the probability that a pixel in the foreground segmentation map is foreground is less than that value, the value of that pixel in the mask map is set to 0. Therefore, the mask map is a 0-1 distribution map.

[0070] The resolution of both the foreground and background images is the same as that of the first image, so that the foreground and background images can represent the first image without loss, thus ensuring the quality of the new perspective images generated subsequently.

[0071] The foreground image includes all pixels from the foreground portion of the first image, and the positions of these pixels in the foreground image are the same as their positions in the first image. The RGB values ​​of all other pixels in the foreground image are 0.

[0072] The background image includes all the pixels of the background portion of the first image, and the positions of these pixels in the background image are the same as their positions in the first image. The RGB values ​​of all other pixels in the background image are 0.

[0073] refer to Figure 2 , Figure 2 An example is shown, including the first image and the segmented foreground and background images.

[0074] S103, repair the missing background portion in the background image to obtain the repaired background image.

[0075] The missing background portion in the background image is also the part of the background of the first image that is obscured by the foreground, referring to the area in the background image where RGB is 0.

[0076] Electronic devices can use image inpainting methods to repair missing background portions in a background image, generating geometric and color features of the missing background portions. Image inpainting methods may include, but are not limited to, image interpolation (inpainting) inpainting, deep learning-based inpainting methods, and generative adversarial network-based inpainting methods. S103 can improve the generation quality of occluded portions in the background of the original first image.

[0077] refer to Figure 2 , Figure 2 The restored background image is also shown as an example.

[0078] S104, Determine the MPI information of the repaired background image.

[0079] An image can include multiple sublayers, and its MPI information includes the MPI information of these sublayers. The MPI information of a sublayer includes: the depth (D) of the sublayer as viewed from the camera's optical center position, the RGB value of each pixel in the sublayer, and σ. Constructing a 3D scene and viewing it from a certain camera's optical center position yields a 2D image. Based on this, σ represents the volume density of a pixel, indicating the probability that a ray of light will terminate at that pixel as it passes through the corresponding 3D scene from the camera's optical center position.

[0080] In other words, each sublayer is an image containing RGBσ four-channel information.

[0081] The MPI information of the repaired background image includes MPI information from N sub-layers, where N is a preset value greater than 1. S104 specifically includes the following steps:

[0082] S1041, The electronic device acquires the disparity values ​​corresponding to N sub-layers of the repaired background image. .

[0083] in, This represents the disparity value of the i-th sublayer, where i ranges from (1, N). The disparity value is the reciprocal of the depth value. S1041 is equivalent to dividing the repaired background image into multiple sublayers of different depths. Each sublayer corresponds to one depth.

[0084] In some implementations, the electronic device can input the repaired background image into a sublayer disparity prediction network to obtain disparity values ​​corresponding to N sublayers. The sublayer disparity prediction network can be trained in advance. The training data can include multiple sets of image data. Each set of image data includes a single-view RGB image and the disparity values ​​of N sublayers corresponding to that single-view RGB image.

[0085] S1042, the electronic device will predict the disparity value The feature maps of the N sub-layers are stitched together with the repaired background image.

[0086] Each sub-layer feature map has a first-level resolution. Each pixel in each sub-layer feature map contains a high-dimensional representation of RGB values ​​and σ.

[0087] In some implementations, the electronic device can transmit parallax values. The restored background image is input into a feature extraction network to obtain N sub-layer feature maps. The feature extraction network can be trained in advance, and the training data can include multiple sets of data. Each set of data includes the disparity value of an image, the image, and the corresponding N sub-layer feature maps.

[0088] S1043, the electronic device converts N sub-layer feature maps into N sub-layers.

[0089] Each sublayer has the same resolution as the first resolution. A sublayer contains multiple pixels, each with its own RGB value and σ.

[0090] In some implementations, the electronic device can input N sublayer feature maps into a sublayer prediction network to obtain N sublayers. The sublayer prediction network can be trained in advance, and the training data can include multiple sets of data, where each set of data can include N sublayer feature maps and the corresponding N sublayers.

[0091] Through S1041-S1043, the electronic device can obtain the MPI information of the N sub-layers of the first image.

[0092] Figure 3 The execution process of S104 is illustrated exemplarily. Figure 3 As shown, This represents the RGB values ​​of multiple pixels contained in the i-th sublayer. σ represents the multiple pixels contained in the i-th sublayer. This represents the disparity value corresponding to the i-th sublayer.

[0093] S105, determine the MPI information of the foreground image.

[0094] The MPI information of the foreground image includes the MPI information of a sublayer. Referring to the previous introduction, the MPI information of a sublayer includes: the depth (D) of the sublayer in the image as viewed from the camera's optical center position in the first-person perspective, the RGB value and σ of each pixel in the sublayer.

[0095] The foreground image corresponds to a sub-layer, and the resolution of this sub-layer is the first resolution.

[0096] The following describes how to determine the MPI information of a sublayer of the foreground image.

[0097] Determine RGB values

[0098] The resolution of the foreground image sublayer is the first resolution. The foreground image sublayer includes all pixels from the foreground portion of the first image. These pixels are positioned in the same location as they are in the first image, and their RGB values ​​are the same in the first image. For all pixels in the foreground image sublayer other than those in the foreground portion of the first image, their RGB values ​​are set to 0.

[0099] Determine the depth (D)

[0100] For the depth (D) of this sub-layer, the embodiments of this application provide the following two calculation methods:

[0101] Method 1. Determine the depth of the foreground image's sublayers based on the minimum depth among the multiple sublayers of the repaired background image.

[0102] In some implementations, since the depth of the foreground is always less than the depth of the background, the electronic device can first find the minimum depth among multiple sublayers of the repaired background image, and then set the depth of the sublayer of the foreground image to a depth less than that minimum depth.

[0103] Method 2. Determine the depth of the sub-layer of the foreground image based on the principle of volume rendering.

[0104] refer to Figure 4 Method 2 may include the following steps:

[0105] S1051, the electronic device determines the MPI information of the first image.

[0106] The first image comprises Y sub-layers, and the MPI information of the first image includes the MPI information of these Y sub-layers. Y is a preset value greater than 1.

[0107] This step can be referred to in the specific implementation of S104. It only requires replacing the repaired background image in S104 with the first image, which will not be elaborated here. The MPI information of the first image determined in S1051 can be used as a coarse expression of the first image. This expression only includes the visible part of the first image and does not include the geometric and texture information of the background part of the first image that is occluded by the foreground.

[0108] Y can be set to be the same as N in S104, so that the electronic device can use the same networks in S1041-S1043 to calculate the MPI information of the first image.

[0109] S1052, determine the weight map of each sub-layer in the Y sub-layers of the first image.

[0110] Each sublayer corresponds to a weight map. The resolution of this weight map is the first resolution. This weight map includes the weight of each pixel in the corresponding sublayer. This weight represents the degree to which the RGB color of the pixel contributes to the RGB values ​​of the first image. The larger the weight, the greater the contribution of the corresponding pixel's RGB color to the first image.

[0111] , Formula 2

[0112] Here, the image with the first resolution is defined to contain M pixels. This represents the weight of a pixel in the i-th sublayer. This represents the volume density of the pixel in the i-th sublayer. This represents the distance between the pixel in the i-th sublayer and the pixel in the (i+1)-th sublayer (i.e., the distance between the i-th sublayer and the (i+1)-th sublayer). The distance between the i-th sublayer and the (i+1)-th sublayer is equal to the difference in depth (D) between these two sublayers.

[0113] In Formula 2, the range of values ​​for i is (1, Y).

[0114] Thus, by using Formula 2, the weight of each pixel in each sub-layer of the first image can be calculated, and the weight map of each sub-layer in the first image can be obtained.

[0115] S1053, Obtain the foreground segmentation map of the first image.

[0116] The foreground segmentation map includes the probability that each pixel in the first image is foreground, with the probability range being (0, 1).

[0117] The electronic device can use thresholding, region segmentation, edge segmentation, or other methods to obtain a foreground segmentation map of the first image. The foreground segmentation map can be obtained during process S102.

[0118] S1054, compare the weight map of each sub-layer of the first image with the foreground segmentation map, and determine the depth (D) of the sub-layer that is most similar to the foreground segmentation map among the Y sub-layers as the depth (D) of the sub-layer of the foreground image.

[0119] The method for comparing the weight map of the sublayer with the foreground segmentation map is as follows: the weights of pixels in the weight map with weights greater than a weight threshold are reset to 1, and the weights of pixels with weights less than a weight threshold are reset to 0. Then, the reset weight map is compared with the corresponding mask map of the foreground segmentation map. The similarity between the sublayer and the mask map represents the similarity between the distribution of 0s and 1s in the weight map of the sublayer and the distribution of 0s and 1s in the mask map.

[0120] Therefore, the depth (D) of the sublayer of the foreground image can be determined in S1054.

[0121] Method 2 allows for the accurate determination of the depth (D) of the sublayer of the foreground image using volume rendering principles. This ensures both the quality of the foreground in the new perspective image and the accuracy of the relative positions of the foreground and background in the new perspective image.

[0122] Determine σ

[0123] The sublayer of the foreground image includes all pixels in the foreground portion of the first image. The σ value for each pixel is set to a specific value such that the product of σ and δ is a preset fixed value (e.g., 10). 3 Here, σ can be called the first value, and δ can be called the second value.

[0124] Wherein, if the depth of a sublayer of the foreground image is not the smallest among all sublayers, then δ is the distance between the sublayer of the foreground image and the sublayer of the background image closest to that sublayer in the direction closest to the camera's optical center, when the sublayers are arranged from the camera's optical center according to their depths, based on the depths of the foreground image's sublayer and the N sublayers of the background image. This distance can be the difference in depth between the two sublayers.

[0125] If the depth of a sublayer of the foreground image is the smallest among all sublayers, then the σ of these pixels is set to 10. 3 .

[0126] For all pixels in the sublayer of the foreground image other than those in the foreground portion of the first image, σ is set to 0.

[0127] Through the above steps S103-S105, the electronic device determines the MPI information of the background image and the foreground image from the first perspective.

[0128] S106, the electronic device outputs a second image from a second perspective based on the MPI information of the restored background image and the MPI information of the foreground image, or outputs a video including multiple new perspective images, the second perspective being different from the first perspective.

[0129] The second perspective depends on the user's input in S101. For example, if the user inputs a leftward view in S101, the camera optical center position corresponding to the second perspective is located to the left of the original camera optical center position of the first perspective; if the user inputs a rightward view in S101, the camera optical center position corresponding to the second perspective is located to the right of the original camera optical center position of the first perspective.

[0130] Whether the electronic device outputs an image or video depends on the user's operation in S101. A video with a new perspective can include multiple consecutive images from different new perspectives. Generating a video with a new perspective can be viewed as generating images from different new perspectives multiple times.

[0131] In this embodiment of the application, the electronic device can output a second image or video of a new perspective based on the MPI information of the repaired background image, the MPI information of the foreground image, and the camera optical center position corresponding to the new perspective, according to the volume rendering method.

[0132] The resolution of the image from the new perspective is also the first resolution.

[0133] Taking the generation of a second-view image as an example, electronic devices can output a second image from a new perspective based on the principle of volume rendering.

[0134] Specifically, S106 may include the following steps:

[0135] S1061, the electronic device, based on the principle of affine transformation, obtains the MPI information of the background image and the foreground image from the second perspective according to the MPI information of the background image and the foreground image from the first perspective.

[0136] From a second-view perspective, the background image comprises N sub-layers, and the foreground image comprises one sub-layer. The MPI information of the background image includes the MPI information of the N sub-layers. The MPI information of one sub-layer includes: the depth (D) of the 3D scene viewed from the camera's optical center position in the second-view perspective, the RGB value of each pixel in that sub-layer, and σ. The MPI information of the foreground image also includes the MPI information of one sub-layer, which includes: the depth (D) of the 3D scene viewed from the camera's optical center position in the second-view perspective, the RGB value of each pixel in that sub-layer, and σ.

[0137] Electronic devices can obtain the MPI information of the background image and the foreground image from a second-view perspective based on the principle of affine transformation. Specifically, the electronic device can use the following formula 3 to calculate the MPI information of the background image and the foreground image from a second-view perspective:

[0138] Formula 3

[0139] in,( This represents the coordinates of a pixel in the i-th sublayer from a second-view perspective. This represents the coordinates of the pixel corresponding to the aforementioned pixel in the i-th sublayer from the first-person perspective, where i ranges from (1 to N+1). This indicates the camera parameters from a first-person perspective. This indicates the camera parameters from a second-person perspective. This indicates the rotation of the second-view perspective relative to the first-view perspective. This represents the translation of the second perspective relative to the first perspective, n = , This represents the depth of the i-th sublayer in the first-person perspective.

[0140] According to Formula 3, the electronic device can calculate the coordinates of a pixel in the i-th sub-layer under the second view, corresponding to the pixel in the same layer under the first view. The RGB value and σ of the corresponding pixel in the first viewpoint are used as the RGB value and σ of a certain pixel in the i-th sublayer in the second viewpoint. The depth of each sublayer in the second viewpoint is the same as the depth of each sublayer in the first viewpoint. By analogy, the electronic device can calculate the MPI information of the background image and the foreground image in the second viewpoint.

[0141] In the second-person perspective, the resolution of each background and foreground layer is the first resolution.

[0142] S1062, the electronic device generates a second image based on the principle of volume rendering, according to the MPI information of the background image and the MPI information of the foreground image from the second perspective.

[0143] Specifically, electronic devices can output a second image using volume rendering formulas:

[0144] , Formula 1

[0145] Where r represents a ray of light that originates from the optical center of the camera in the second-person view and passes through the 3D scene;

[0146] This indicates the color of a specific pixel in the second image;

[0147] N+1 is the number of sublayers and also the number of sampling points. N+1 sublayers are N sublayers of the background image arranged according to depth and one sublayer of the foreground image.

[0148] Let be the transmittance of the medium from the camera's optical center position in the second viewpoint to the i-th sampling point, which is the probability that light does not encounter any particles before reaching that point;

[0149] This represents the volume density of the sampling point when light ray r passes through the i-th sublayer from a second perspective;

[0150] It is the spacing between the (i+1)th sublayer and the ith sublayer;

[0151] It is the color of the sampled point in the i-th sublayer.

[0152] The pixel where the ray r intersects with the i-th sublayer is the sampling point in the i-th sublayer. This sampling point can be calculated using the direction of the ray r and the depth of the sublayer.

[0153] Therefore, the color of each pixel in the second image can be calculated using Formula 1. Then, the electronic device can generate a second image from a new perspective based on the color of each pixel.

[0154] Figure 5 An example is shown for Figure 2 A second image, generated from a new perspective, is shown as the first image. Figure 5 As shown, compared to Figure 2 The first image shown, Figure 5 The viewpoint of the image shown has shifted to the right and is closer to the object in the image. Figure 2 In the second image shown from the new perspective, the background is clear, without any voids or artifacts; the foreground and background structures are accurate, with the main subject, the person, in the foreground and buildings in the background, and the foreground quality is excellent.

[0155] S107, the electronic device displays a second image from a second perspective, or plays a video from a new perspective.

[0156] The method provided in the embodiments of this application, as described above, repairs missing parts in the background, ensuring the quality of the background portion in the new perspective image. The electronic device also accurately determines the depth of the foreground and fuses the foreground and background into a new perspective image based on the foreground depth, ensuring the accuracy of the foreground and background structure and the quality of the foreground in the new perspective image. Therefore, the method provided in the embodiments of this application can synthesize high-quality new perspective images based on images from a single perspective, thereby meeting user needs.

[0157] The electronic device provided in the embodiments of this application is described below.

[0158] Figure 6 A hardware structure block diagram of an electronic device 100 provided in an embodiment of this application is shown. This electronic device 100 is used to perform the method for generating new perspective images provided in the preceding method embodiments.

[0159] Electronic device 100 may include a processor 101, a memory 102, a wireless communication module 103, a mobile communication module 104, an antenna 103A, an antenna 104A, a power switch 105, a sensor module 106, a focusing motor 107, a camera 108, a display screen 109, etc. The sensor module 106 may include a gyroscope sensor 106A, an accelerometer sensor 106B, an ambient light sensor 106C, an image sensor 106D, a proximity sensor 106E, etc. The wireless communication module 103 may include a WLAN communication module, a Bluetooth communication module, etc. All of the above components can transmit data via a bus.

[0160] Processor 101 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0161] Memory 102 can be used to store computer executable program code, which may include instructions. Processor 101 executes various functional applications and data processing of electronic device 100 by running the instructions stored in memory 102. Memory 102 may include a program storage area and a data storage area. In specific implementations, memory 102 may include high-speed random access memory, and may also include non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices.

[0162] The wireless communication function of the electronic device 100 can be implemented through antenna 103A, antenna 104A, mobile communication module 104, wireless communication module 103, modem processor, and baseband processor.

[0163] Antennas 103A and 104A can be used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization.

[0164] The mobile communication module 104 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 104 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 104 can receive electromagnetic waves via antenna 104A, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 104 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 104A.

[0165] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs a sound signal through an audio device or displays an image or video through the display screen 109.

[0166] The wireless communication module 103 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLAN), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR). The wireless communication module 103 can be one or more devices integrating at least one communication processing module. The wireless communication module 103 receives electromagnetic waves via antenna 103A, performs frequency modulation and filtering of the electromagnetic wave signal, and sends the processed signal to processor 101. The wireless communication module 103 can also receive signals to be transmitted from processor 101, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 103A.

[0167] The power switch 105 can be used to control the power supply to the electronic device 100.

[0168] The gyroscope sensor 106A can be used to determine the motion attitude of the electronic device 100. In some embodiments, the gyroscope sensor 106A can determine the angular velocity of the electronic device 100 about three axes (i.e., the x, y, and z axes). The gyroscope sensor 106A can be used for image stabilization. For example, when the shutter is pressed, the gyroscope sensor 106A detects the angle of the electronic device 100's shake, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to counteract the shake of the electronic device 100 through reverse movement, thus achieving image stabilization. The gyroscope sensor 106A can also be used in navigation and motion-sensing game scenarios.

[0169] Accelerometer 106B can detect the magnitude of acceleration of electronic device 100 in various directions (generally three axes). When electronic device 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of electronic device; for example, accelerometer 106B can be applied to applications such as screen orientation switching and pedometers.

[0170] The ambient light sensor 106C is used to sense the brightness of ambient light. The electronic device 100 can adaptively adjust the brightness of the display screen 109 according to the sensed ambient light brightness. The ambient light sensor 106C can also be used to automatically adjust the white balance when taking pictures.

[0171] The 106D image sensor, also known as a photosensitive element, uses the photoelectric conversion function of an optoelectronic device to convert a light image on a photosensitive surface into an electrical signal proportional to the light image. Image sensors can be either charge-coupled device (CCD) sensors or complementary metal-oxide-semiconductor (CMOS) sensors.

[0172] The distance sensor 106E can be used to measure distance. The electronic device 100 can measure distance via infrared or laser. In some shooting scenarios, the electronic device 100 can use the distance sensor 106E to measure distance for fast focusing.

[0173] The focusing motor 107 can be used for rapid focusing. The electronic device 100 can control the movement of the lens via the focusing motor 107 to achieve autofocus.

[0174] Electronic device 100 can perform shooting functions through ISP, camera 108, video codec, GPU, display screen 109 and application processor.

[0175] The ISP is used to process data fed back from the camera 108. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, converting it into an image visible to the naked eye. The ISP can also perform algorithmic optimization on image noise and brightness. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 108.

[0176] Camera 108 can be used to capture still images or videos. An object is projected onto an image sensor through the lens, generating an optical image. The image sensor converts the light signal into an electrical signal, which is then passed to an ISP (Internet Service Provider) for conversion into a digital image signal. The ISP can output the digital image signal to a DSP (Digital Signal Processor) for processing. The DSP converts the digital image signal into image signals in standard formats such as RGB and YUV. In some embodiments, the electronic device 100 may include one or N cameras 108, where N is a positive integer greater than 1.

[0177] Video codecs are used to compress or decompress digital images. Electronic device 100 may support one or more image codecs. Thus, electronic device 100 can open or save images or videos in various encoding formats.

[0178] Electronic device 100 can implement display functions through a GPU, display screen 109, and application processor. The GPU is a microprocessor for image processing, connected to the display screen 109 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 101 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0179] The display screen 109 is used to display images, videos, etc. The display screen 109 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N displays 109, where N is a positive integer greater than 1.

[0180] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0181] In this embodiment, memory 102 stores a computer program for implementing a method for generating images from new perspectives, and processor 101 executes the computer program to implement the method for generating images from new perspectives as provided in any of the above embodiments. Figure 1 In the method embodiment shown, S101 can be executed by an information input module such as a display screen 109, S102-S106 can be executed by a processor 101 such as an AP, GPU, ISP, NPU or other processor with image processing capabilities, and S107 can be executed by an information output module such as a display screen 109.

[0182] The operations performed by each device in the electronic device 100 can be specifically referred to in the relevant descriptions of the method embodiments above, and will not be elaborated here.

[0183] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses a layered mobile operating system as an example to exemplify the software structure of electronic device 100.

[0184] Figure 7 A software architecture block diagram of an electronic device 100 according to an embodiment of this application is shown.

[0185] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, a mobile operating system is divided into four layers, from top to bottom: the application layer, the application framework / core service layer, the system libraries and runtime, and the kernel layer.

[0186] The application layer can include a series of application packages.

[0187] like Figure 7 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS.

[0188] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0189] like Figure 7 As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0190] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0191] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0192] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0193] A phone manager is used to provide communication functions for electronic devices. For example, it manages call status (including connection and disconnection).

[0194] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0195] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of completed downloads or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0196] Runtime can refer to all the code libraries, frameworks, etc., required for a program to run. For example, for the C language, the runtime includes a series of function libraries required for C programs to run. For the Java language, in addition to the core libraries, the runtime also includes the virtual machine required for Java programs to run. The aforementioned core libraries can include the functionalities that the Java language needs to call.

[0197] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0198] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0199] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0200] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0201] A 2D graphics engine is a graphics engine for 2D drawing.

[0202] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0203] It should be understood that each step in the above method embodiments can be completed by integrated logic circuits in the processor hardware or by instructions in software form. The method steps disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor.

[0204] This application also provides an electronic device that may include a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method performed by the electronic device as described in any of the above embodiments.

[0205] This application also provides a chip system including at least one processor for implementing the methods executed by the electronic device in any of the above embodiments. In one possible design, the chip system further includes a memory for storing program instructions and data, the memory being located within or outside the processor.

[0206] A chip system can consist of chips or include chips and other discrete components.

[0207] Optionally, there may be one or more processors in the chip system. The processor can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor, implemented by reading software code stored in memory.

[0208] Optionally, the chip system may contain one or more memories. These memories may be integrated with the processor or disposed separately; this application does not limit this. For example, the memory may be a non-transient processor, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or disposed on different chips. This application does not specifically limit the type of memory or the arrangement of the memory and processor.

[0209] For example, the chip system may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a micro controller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0210] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method executed by the electronic device in any of the above embodiments.

[0211] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the method executed by the electronic device as described in any of the above embodiments.

[0212] The various embodiments of this application can be combined arbitrarily to achieve different technical effects.

[0213] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0214] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0215] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0216] The terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0217] In summary, the above description is merely an embodiment of the technical solution of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made based on the disclosure of this application should be included within the scope of protection of this application.

Claims

1. A method for generating images from new perspectives, characterized in that, The method is applied to an electronic device, and the method includes: Obtain the foreground and background images of the first image, wherein the viewpoint of the first image is the first viewpoint; Repair the background image; The MPI information of each of the N sub-layers of the repaired background image is determined. Each sub-layer corresponds to a depth. The MPI information of a sub-layer includes: the depth of the sub-layer under the first viewpoint, the color and volume density of each pixel in the sub-layer, and N is a preset value greater than 1. Determine the MPI information of a sub-layer of the foreground image, wherein the MPI information of a sub-layer of the foreground image includes: the depth of the sub-layer under the first viewpoint, the color of each pixel in the sub-layer, and the volume density; Determining the MPI information of a sub-layer of the foreground image specifically includes: Determine the MPI information of each of the Y sub-layers of the first image. Each sub-layer corresponds to a depth. The MPI information of a sub-layer includes: the depth of the sub-layer, the color and volume density of each pixel in the sub-layer, and Y is a preset value greater than 1. Based on the depth and volume density in the MPI information of each of the Y sub-layers of the first image, the weight maps of each of the Y sub-layers of the first image are determined according to the following formula: ， in, This represents the weight of a pixel in the i-th sublayer. This represents the volume density of the pixels in the i-th sublayer. This represents the distance between the i-th sublayer and the (i+1)-th sublayer. The weight map of a sublayer includes the contribution of the color of each pixel in the corresponding sublayer to the color of the first image. The weight maps of each of the Y sub-layers are matched with the foreground segmentation map of the first image to determine the first depth, which is the depth of the sub-layer whose weight map is most similar to the foreground segmentation map of the first image. The first depth is determined as the depth of a sub-layer of the foreground image; Based on the principle of volume rendering, a second image is generated according to the MPI information of each of the N sub-layers of the repaired background image, the MPI information of one sub-layer of the foreground image, and the second perspective. The second perspective is different from the first perspective.

2. The method according to claim 1, characterized in that, Determine the MPI information of each of the N sub-layers of the repaired background image, specifically including: Determine the disparity values ​​of each of the N sub-layers of the repaired background image, where the disparity value of a sub-layer is equal to the reciprocal of the depth of the sub-layer; Based on the disparity values ​​of the N sub-layers and the repaired background image, determine the color and volume density of the pixels in each of the N sub-layers.

3. The method according to claim 2, characterized in that, Determine the disparity values ​​of each of the N sub-layers of the repaired background image, specifically including: The repaired background image is input into the sub-layer disparity prediction network to obtain the disparity values ​​of each of the N sub-layers of the repaired background image.

4. The method according to claim 1, characterized in that, Determining the MPI information of a sublayer of the foreground image specifically includes: The color of the foreground pixel in a sub-layer of the foreground image is determined as the color of the foreground pixel in the first image.

5. The method according to any one of claims 1-4, characterized in that, Determining the MPI information of a sublayer of the foreground image specifically includes: If the depth of a sublayer of the foreground image is the minimum depth among N+1 sublayers, then the volume density of pixels in a sublayer of the foreground image is determined to be 10. 3 The N+1 sub-layers include: one sub-layer of the foreground image and N sub-layers of the background image; If the depth of a sublayer of the foreground image is not the minimum depth among the N+1 sublayers, then the volume density of the pixels in the sublayer of the foreground image is determined to be a first value, the product of the first value and the second value is a preset value, and the second value is the distance between the sublayer of the foreground image and the nearest sublayer among the N sublayers in the background image.

6. The method according to any one of claims 1-4, characterized in that, Before acquiring the foreground and background images of the first image, the method further includes: Display the first image; A user operation was received to generate a new perspective image from the first image.

7. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored on the memory, wherein the processor executes the computer program to implement the method as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-6.

9. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1-6.

Images