Three-dimensional reconstruction method and electronic device

Abstract

This application provides a 3D reconstruction method and an electronic device. In the process of 3D reconstruction based on 3D Gaussian sputtering technology, after determining the pixel features of each target pixel in L images, when obtaining the 3D point features corresponding to each 3D point based on the pixel features of each target pixel, for multiple target pixels corresponding to the same 3D point, when fusing the pixel features of these multiple target pixels to obtain the 3D point features corresponding to the 3D point, the correlation between the shooting viewpoint direction and the target viewpoint direction of each target pixel is referenced. For target pixels whose shooting viewpoint direction is closer to the target viewpoint direction, the pixel features of the target pixel account for a larger proportion in the 3D point features of the 3D point. This makes the fused 3D point features more suitable for generating the 3D Gaussian sphere properties of each 3D point under the target viewpoint direction, thereby improving the rendering quality when rendering based on the 3D Gaussian sphere properties.

Description

Technical Field

[0001] This application relates to the field of image information processing technology, and more particularly to three-dimensional reconstruction methods and electronic devices. Background Technology

[0002] 3D reconstruction involves generating a digital model of a 3D scene (such as objects, people, etc.) from image data taken from different perspectives. Based on this digital model, the 3D scene can be reconstructed, or images describing the scene from new perspectives can be generated.

[0003] 3D reconstruction can be applied in many fields, such as computer vision and image rendering. It provides researchers with a way to digitize real 3D scenes, making it easier to reproduce them. Summary of the Invention

[0004] This application provides a 3D reconstruction method and electronic device for more reasonably generating images of 3D scenes from the target shooting perspective.

[0005] In a first aspect, this application provides a three-dimensional reconstruction method applied to an electronic device, comprising: acquiring L images, wherein the L images are images obtained from shooting a first object from different angles; L is an integer greater than or equal to 2; determining the first pixel feature vector of each target pixel in the L images, wherein a target pixel corresponds to a three-dimensional point in the point cloud of the first object, and the three-dimensional point corresponding to the target pixel coincides with the target pixel when projected onto the image to which the target pixel belongs; the three-dimensional points corresponding to each target pixel include a first type of three-dimensional point and a second type of three-dimensional point, wherein the first type of three-dimensional point corresponds to target pixels in multiple images; the second type of three-dimensional point corresponds to a target pixel in only one image; fusing the first pixel feature vectors of all target pixels corresponding to each first type of three-dimensional point based on a first viewing direction to obtain the first three-dimensional point feature vector corresponding to each first type of three-dimensional point; wherein all target pixels corresponding to a first type of three-dimensional point include a first pixel and a second pixel, compared to the second pixel The shooting angle direction of the first pixel is closer to the first viewpoint direction, and the proportion of the first pixel feature vector of the first pixel in the first three-dimensional point feature vector corresponding to the first three-dimensional point of the first type is greater than that of the first pixel feature vector of the second pixel; the shooting angle direction of a target pixel is the direction in which the optical center of the camera points to the three-dimensional point corresponding to the target pixel when the image of the target pixel is captured; the 3D Gaussian sphere attribute of each three-dimensional point is determined based on the first three-dimensional point feature vector corresponding to each three-dimensional point; the 3D Gaussian sphere attribute of a three-dimensional point is used to describe at least the shape, position and color distribution of the spherical region where the three-dimensional point is located; the first three-dimensional point feature vector corresponding to a second type of three-dimensional point is the first pixel feature vector of the target pixel corresponding to the second type of three-dimensional point; the 3D Gaussian sphere attribute of each three-dimensional point is used to render according to the first viewpoint direction to obtain a target image, which describes the content of the first object captured in the first viewpoint direction.

[0006] In the above embodiments, after determining the pixel features of each target pixel in L images, when obtaining the 3D point features corresponding to each 3D point based on the pixel features of each target pixel, for multiple target pixels corresponding to the same 3D point (first type of 3D point), when fusing the pixel features of these multiple target pixels to obtain the 3D point features corresponding to the 3D point, the electronic device considers the correlation between the shooting view direction and the target view direction of each target pixel. For target pixels whose shooting view direction is closer to the target view direction, the pixel features of that target pixel account for a larger proportion in the 3D point features corresponding to the 3D point. Here, the target view direction is the view direction specified during rendering. Considering the contribution of the target view direction to the pixel features of each target pixel to the 3D point features corresponding to the 3D point during fusion allows the fused 3D point features to better obtain the 3D Gaussian sphere properties applicable to the 3D point of the target view direction, thereby improving the quality of the rendering result. For example, it prevents ghosting in the image.

[0007] In conjunction with the first aspect, in some embodiments, the first pixel feature vector of a target pixel includes at least the pixel alignment feature of the target pixel; wherein, the process of determining the pixel alignment feature of the target pixel includes: extracting image features of a first image, the first image including the target pixel; the image features recording the pixel features of each pixel in the first image; projecting the three-dimensional point corresponding to the target pixel in the point cloud onto the first image, and when the projection position coincides with the position of the target pixel, using the pixel feature of the target pixel in the image features as the pixel alignment feature of the target pixel.

[0008] In the above embodiments, reprojecting the 3D points in the point cloud onto the image to determine the target pixel can remove erroneous 3D points from the point cloud. If a pixel corresponds to a 3D point in the point cloud of the photographed object, but the 3D point cannot coincide with the pixel when projected onto the image to which the pixel belongs, then the pixel cannot be called the target pixel.

[0009] In conjunction with the first aspect, in some embodiments, the first pixel feature vector of a target pixel includes the pixel alignment feature of the target pixel plus the color value of the target pixel.

[0010] In conjunction with the first aspect, in some embodiments, the first pixel feature vectors of all target pixels corresponding to each first type of 3D point are fused based on the first viewing direction to obtain the first 3D point feature vector corresponding to each first type of 3D point. Specifically, this includes: inputting the first pixel feature vector of each target pixel corresponding to the i-th 3D point in each first type of 3D point as a first value vector, the shooting viewing direction of each target pixel as a key vector, and the first viewing direction as a query vector into a first attention mechanism model to obtain the first 3D point feature vector corresponding to the i-th 3D point; the value of i ranges from 1 to the number of each first type of 3D point; wherein, the first attention mechanism model is used to calculate the attention weight between the query vector and each key vector; the more similar a key vector is to the query vector, the greater the attention weight between the key vector and the query vector; the first attention mechanism model is also used to perform a weighted summation of each attention weight on each first value vector to obtain the first 3D point feature vector corresponding to the i-th 3D point.

[0011] In conjunction with the first aspect, in some embodiments, the 3D Gaussian sphere attribute of each three-dimensional point is determined based on the first three-dimensional point feature vector corresponding to each three-dimensional point. Specifically, this includes: inputting the first three-dimensional point feature vector corresponding to each three-dimensional point into the first multilayer perceptron (MLP) network model in sequence, and converting the first three-dimensional point feature vector corresponding to each three-dimensional point into a 3D Gaussian sphere attribute through the first MLP network to obtain the 3D Gaussian sphere attribute of the three-dimensional point.

[0012] In the above embodiments, the first MLP network model can be the MLP network model a involved in the following embodiments.

[0013] In conjunction with the first aspect, in some embodiments, the first pixel feature vector of a target pixel is the pixel alignment feature of that target pixel. After determining the first pixel feature vector of each target pixel in L images, the method further includes: determining the second pixel feature vector of each target pixel in L images, wherein the second pixel feature vector of a target pixel is the pixel alignment feature of that target pixel plus the color value of that target pixel; fusing the second pixel feature vectors of all target pixels corresponding to each first type of 3D point based on the first viewing direction to obtain the second 3D point feature vector corresponding to each first type of 3D point; determining the 3D Gaussian sphere attribute of each 3D point based on the first 3D point feature vector corresponding to each 3D point, specifically including: determining the 3D Gaussian sphere attribute of each 3D point based on the first 3D point feature vector and the second 3D point feature vector corresponding to each 3D point; wherein the second 3D point feature vector corresponding to a second type of 3D point in each 3D point is the second pixel feature vector of the target pixel corresponding to that second type of 3D point.

[0014] In the above embodiments, the pixel alignment feature of the target pixel is obtained from the image features. This image feature is obtained by extracting features from the image containing the target pixel. A neural network with image feature extraction capabilities can be used to extract features from the image containing the target pixel to obtain image features. Image features can include the pixel features of each pixel in the image, and the pixel features of a pixel include the color features of that pixel and the geometric features of the region where the pixel is located. However, the color features obtained after feature extraction have insufficient ability to represent color. If only the pixel alignment feature is used to determine the 3D Gaussian sphere properties of a 3D point, there is a possibility of color distortion. Therefore, it is necessary to use a color-pixel alignment feature, and then use both the color-pixel alignment feature and the pixel alignment feature together to determine the 3D Gaussian sphere properties of the 3D point, resulting in better color representation. Here, the pixel alignment feature is still retained in the color-pixel alignment feature because the color features and geometric features in the pixel alignment feature can be used to positively influence the newly added color value in generating the 3D Gaussian sphere properties.

[0015] In conjunction with the first aspect, in some embodiments, the 3D Gaussian sphere properties of one of the three-dimensional points include: the rotation component and position of the spherical region where the three-dimensional point is located in the world coordinate system, the scale component, opacity, and spherical harmonic factor of the spherical region where the three-dimensional point is located, wherein the spherical harmonic factor is used to characterize the color distribution of the spherical region.

[0016] In conjunction with the first aspect, in some embodiments, the 3D Gaussian sphere attribute of each 3D point is determined based on the first 3D point feature vector and the second 3D point feature vector corresponding to each 3D point. Specifically, this includes: determining a first attribute in the 3D Gaussian sphere attribute of the k-th 3D point based on the first 3D point feature vector of the k-th 3D point; and determining a second attribute in the 3D Gaussian sphere attribute of the k-th 3D point based on the second 3D point feature vector of the k-th 3D point. Here, k ranges from 1 to the number of 3D points, and the second attribute is the color distribution of the spherical region where the k-th 3D point is located. The first attribute and the second attribute are combined as the 3D Gaussian sphere attribute of the k-th 3D point.

[0017] In conjunction with the first aspect, in some embodiments, determining a first attribute in the 3D Gaussian sphere properties of the k-th 3D point based on the first 3D point feature vector of the k-th 3D point, and determining a second attribute in the 3D Gaussian sphere properties of the k-th 3D point based on the second 3D point feature vector of the k-th 3D point, specifically includes: inputting the first and second 3D point feature vectors of the k-th 3D point into a second MLP network model to obtain the first attribute and the second attribute; the second MLP network model includes a first MLP network layer, a second MLP network layer, and a third MLP network layer; the first MLP network layer is used to process the first 3D point feature vector of the k-th 3D point to obtain a processed first 3D point feature vector; the second MLP network layer is used to convert the processed first 3D point feature vector into the first attribute; the third MLP network layer is used to combine the processed first 3D point feature vector and the second 3D point feature vector of the k-th 3D point, and then convert the combined 3D point feature vector into the second attribute.

[0018] In the above embodiments, the processed three-dimensional point feature 1 (first three-dimensional point feature vector) is combined with the three-dimensional point feature 2 (second three-dimensional point feature vector) and then input into the MLP network layer 3 to obtain attribute 2. This allows the process of obtaining attribute 2 to be standardized by the processed three-dimensional point feature 1 that obtains attribute 1, thus establishing the connection between attribute 1 and attribute 2.

[0019] In conjunction with the first aspect, in some embodiments, the second pixel feature vectors of all target pixels corresponding to each first type of 3D point are fused based on the first viewing direction to obtain the second 3D point feature vector corresponding to each first type of 3D point. Specifically, this includes: using the second pixel feature vector of each target pixel corresponding to the i-th 3D point in each first type of 3D point as the second value vector, the shooting viewing direction of each target pixel as the key vector, and the first viewing direction as the query vector, inputting them into the second attention mechanism model to obtain the second 3D point feature vector corresponding to the i-th 3D point; the value of i ranges from 1 to the number of each first type of 3D point; wherein, the second attention mechanism model is used to calculate the attention weight between the query vector and each key vector; the more similar a key vector is to the query vector, the greater the attention weight between the key vector and the query vector; the first attention mechanism model is also used to perform a weighted summation of each attention weight on each second value vector to obtain the second 3D point feature vector corresponding to the i-th 3D point.

[0020] In a second aspect, embodiments of this application provide an electronic device comprising: one or more processors and a memory; the memory is coupled to the one or more processors and is used to store computer program code, the computer program code including computer instructions, wherein the one or more processors invoke the computer instructions to cause the electronic device to perform the method implemented in the first aspect.

[0021] Thirdly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on an electronic device, cause the electronic device to perform the method as implemented in the first aspect.

[0022] Fourthly, embodiments of this application provide a chip system applied to an electronic device. The chip system includes one or more processors, which are used to invoke computer instructions to cause the electronic device to perform the method as implemented in the first aspect.

[0023] Fifthly, embodiments of this application provide a computer program product containing instructions that, when run on an electronic device, cause the electronic device to perform the method as implemented in the first aspect.

[0024] It is understood that the electronic device provided in the second aspect, the computer storage medium provided in the third aspect, the chip system provided in the fourth aspect, and the computer program product provided in the fifth aspect are all used to execute the methods provided in the embodiments of this application. Therefore, other beneficial effects that can be achieved can be referred to the beneficial effects in the corresponding methods, and will not be repeated here. Attached Figure Description

[0025] Figure 1 This is an exemplary shooting scenario shown in an embodiment of this application;

[0026] Figure 2 A schematic diagram is shown illustrating the process of 3D reconstruction based on 3D Gaussian sputtering technology;

[0027] Figure 3 An exemplary flowchart is shown for fusing pixel alignment features based on attention mechanism model 1;

[0028] Figure 4 An exemplary flowchart is shown when fusing color-pixel alignment features based on attention mechanism Model 2;

[0029] Figure 5 The diagram shows a schematic of obtaining the 3D Gaussian sphere properties of a 3D point based on the pixel features of all target pixels corresponding to the 3D point;

[0030] Figure 6This diagram illustrates the rendering of an image based on the properties of a 3D Gaussian sphere at each 3D point.

[0031] Figure 7 An exemplary flowchart is shown for 3D reconstruction based on 3D Gaussian sputtering technology;

[0032] Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0033] In one possible implementation, 3D reconstruction is performed based on NeRF (neural radiance fields) technology. In this method, a digital model describing the 3D scene can be determined using multiple images obtained during the capture of the scene. Then, the 3D scene can be reconstructed based on this digital model, or an image of the 3D scene from a new perspective can be generated. It should be noted that the 3D scene involved in this embodiment can be a photographed object, such as a person or scenery; this embodiment does not limit the scope of the application.

[0034] In this approach, the digital model describing the 3D scene is: the dense volume information of multiple 3D points in the 3D scene. The dense volume information of each 3D point can be obtained by learning each image based on the camera parameters of each image from multiple images through a neural network. The camera parameters of an image include the direction of the light rays emitted by the camera through each pixel position in the image when the image is captured. The direction of the emitted light rays includes the camera, the pixel, and the 3D point of the pixel in the 3D scene, which can be used to determine the imaging transformation from 3D point to pixel.

[0035] The density volume information of a 3D point includes its density and color distribution. The density of a 3D point can be represented by the opacity, light reflection and refraction properties, or other attributes of the region containing that point.

[0036] Subsequently, ray tracing rendering algorithms can be used to reconstruct the 3D scene by utilizing the dense volume information of each 3D point. For example, given a viewpoint direction (target viewpoint direction), the ray tracing rendering algorithm can generate a large number of rays emitted from the given target viewpoint direction to simulate the process of a camera capturing a 3D scene from that direction. Some or all of these rays can intersect with 3D points. The density and color of the 3D points determine the degree of light transmission and scattering during the rendering of the 3D points by the intersecting rays, thus obtaining the final color representation of the 3D points and rendering them into pixels. By providing several target viewpoint directions for rendering, images of the 3D scene from various viewpoint directions can be reproduced, achieving the reconstruction of the 3D scene. Here, the viewpoint direction can be the direction from the camera to the 3D scene in the world coordinate system, at a specified position and orientation, and can be represented by a vector. This viewpoint direction describes the range of the 3D scene that the camera can capture at a specified position and orientation; that is, the viewpoint direction can be used to describe the field of view of the camera when capturing a 3D scene at a specified position and orientation.

[0037] In the aforementioned NeRF-based 3D reconstruction technology, an implicit rendering method is used. This implicitness manifests in the fact that the dense volumetric information of 3D points does not directly include the final color representation of the 3D point image. Instead, the density and color of the 3D points together determine the degree of light transmission and scattering when intersecting with them, thus obtaining the final color representation of the 3D point image. This implicit rendering method requires inferring the degree of light transmission and scattering caused by the density and color of the 3D points. This process has high computational complexity, which is detrimental to rendering speed and makes it difficult to implement on devices with limited computing power, such as mobile phones.

[0038] To improve rendering speed and achieve faster 3D reconstruction, a method based on 3D Gaussian splatting is proposed. In this method, an electronic device can determine a digital model describing a 3D scene using L images obtained from capturing the scene. This digital model can then be used for various aspects of 3D reconstruction, such as reconstructing the 3D scene, editing target attributes, generating images from new perspectives describing the scene, and applying it to 3D camera movements or 3D video generation. Here, L is an integer greater than or equal to 2, and the L images are those obtained from capturing the 3D scene from different angles.

[0039] It should be noted that a 3D scene includes objects captured by the camera within the shooting scene, such as people and scenery. For example... Figure 1 In the shooting scene shown, the subject can be a person. The camera of the electronic device can capture at least two images from different angles to describe the scene. Figure 1The figures in the images. It should be noted that when capturing Image 1 and Image 2 of the L images, the camera's field of view should partially overlap, so that some target pixels in Image 1 and Image 2 correspond to the same 3D point. Here, Image 1 is one of the L images, and Image 2 is at least one of the L images other than Image 1. The content of Images 1 and 2 is illustrative; it can actually be other content and should not be construed as limiting the embodiments of this application.

[0040] A 3D reconstruction scene can be used as a reference. Figure 1 As shown in (2), the L images may include image 1 and image 2. A three-dimensional reconstruction method is implemented based on the L images. By specifying the target viewing direction, an image under the target viewing direction can be obtained. This target viewing direction may be different from the viewing direction when the L images were taken; it is a new viewing direction.

[0041] It's important to note that the larger the shooting scene, the more images are needed for 3D reconstruction, and the larger L becomes. Conversely, if the shooting scene is small, fewer images can be used to achieve good 3D reconstruction results. For example, refer to... Figure 1 As shown, when the camera is taking portrait photos at a relatively close distance, L can be 2.

[0042] In this approach, the digital model describing the 3D scene is: the 3D Gaussian sphere properties of multiple 3D points in the 3D scene. The 3D Gaussian sphere properties of a 3D point describe the shape, position, and color distribution of the spherical region containing that 3D point. In some possible cases, the 3D Gaussian sphere properties of a 3D point include: the rotation component R and position μ of the spherical region containing the 3D point in the world coordinate system, the scale component S, the opacity α, and the spherical harmonic factor c of the spherical region containing the 3D point. The rotation component can be used to characterize the orientation of the spherical region in the world coordinate system, the scale component can be used to characterize the size of the spherical region, and the spherical harmonic factor can be used to characterize the color distribution of the spherical region. In other possible cases, the 3D Gaussian sphere properties of a 3D point include: the position of the spherical region containing the 3D point in the world coordinate system, the covariance matrix of the spherical region containing the 3D point, the opacity, and the color distribution of the spherical region containing the 3D point. The color distribution can be represented by the spherical harmonic factor.

[0043] Subsequently, electronic devices can use 3D Gaussian sputtering rendering (also known as Gaussian sputtering differentiable rendering) to render the 3D Gaussian sphere properties of each 3D point and perform 3D reconstruction. This process includes: given a viewpoint direction (target viewpoint direction), determining the visible 3D points under the target viewpoint direction, determining the 3D Gaussian sphere of the visible 3D points based on the 3D Gaussian sphere properties of the visible 3D points, and projecting the 3D Gaussian sphere of the visible 3D points onto the image plane to generate an image of the 3D scene under the target viewpoint direction.

[0044] It should be noted that by providing several target viewpoints for rendering, images of the 3D scene from various viewpoints can be reproduced, thus achieving the reconstruction of the 3D scene.

[0045] Compared to the aforementioned approach (NeRF-based 3D reconstruction), the 3D reconstruction method based on 3D Gaussian sputtering technology uses explicit rendering, which is faster than the implicit rendering discussed earlier and can be implemented on devices with lower computing power, such as mobile phones. This is because the rendering process involves projecting a 3D Gaussian sphere of 3D points onto an image plane to generate an image. This process is much faster than the aforementioned approach, which involves emitting a large number of rays from a given target viewpoint to intersect with the 3D points, simulating a camera capturing an image of a 3D scene from the target viewpoint.

[0046] In the 3D reconstruction method based on 3D Gaussian sputtering technology, the process of determining the 3D Gaussian sphere attributes corresponding to multiple 3D points describing a 3D scene based on L images includes: First, feature extraction is performed on the L images to determine the pixel features (which can be represented as vectors) of each pixel in the L images. Based on the L images and the intrinsic and extrinsic parameter data of the camera that captured the L images, the corresponding 3D points in the 3D scene are predicted (the 3D points can be represented by their positions in the world coordinate system), and the set of 3D points is used as the point cloud of the 3D scene. Then, the pixel features of each target pixel in the L images are determined. A target pixel corresponds to a 3D point in the point cloud of the 3D scene, and when this 3D point is projected onto the image to which the target pixel belongs, it coincides with the target pixel. Then, based on the pixel features of the target pixels corresponding to the 3D points, the 3D point features (which can be represented as vectors) corresponding to each 3D point are obtained. Finally, the 3D point features corresponding to each 3D point are transformed into the 3D Gaussian sphere attributes of each 3D point through a neural network model. The camera's intrinsic and extrinsic parameters can include parameters such as camera pose and focal length. These parameters can be used to characterize the camera's position, attitude, or shooting direction.

[0047] It should be noted that when a 3D point corresponds to a target pixel in only one image, the electronic device can use the pixel features of that target pixel as the 3D point feature corresponding to that 3D point. When a 3D point corresponds to a target pixel in multiple images, the electronic device can fuse the pixel features of the target pixels in the multiple images and use the fused pixel features as the 3D point feature corresponding to that 3D point.

[0048] In this context, a single 3D point corresponds to target pixels in multiple images. This can be understood as the 3D point being represented by multiple target pixels, with each pixel describing the 3D point from a different shooting angle. For example, ... Figure 1 The three-dimensional point P shown in (1) can be represented by pixel P1 or pixel P2, and pixel P1 and pixel P2 come from different pixels in different L images. Therefore, after fusing the pixel features of the multiple target pixels, the features of the three-dimensional point under multiple shooting angles can be obtained, which can represent the three-dimensional point. The fused pixel features can be regarded as the three-dimensional point features of the three-dimensional point. The fusion mentioned here can be fused by averaging the pixel features of multiple target pixels. However, such a fusion method does not take into account the correlation between the shooting angle direction and the target angle direction of multiple target pixels. The fusion effect is not good. The three-dimensional point features corresponding to the fused three-dimensional point cannot represent the three-dimensional point well from the target angle direction, so that the content obtained by rendering the 3D Gaussian sphere property of the three-dimensional point under the target angle direction is inaccurate, which affects the rendering result.

[0049] To more accurately fuse pixel features from multiple target pixels (greater than or equal to two target pixels) and thus better obtain the 3D point features corresponding to the 3D point jointly described by multiple target pixels, an alternative 3D reconstruction method based on 3D Gaussian sputtering technology is proposed. In this method, after determining the pixel features of each target pixel in L images, when obtaining the 3D point features corresponding to each 3D point based on the pixel features of each target pixel, for multiple target pixels corresponding to the same 3D point, when fusing the pixel features of these multiple target pixels to obtain the 3D point features corresponding to the 3D point, the electronic device considers the correlation between the shooting view direction and the target view direction of each target pixel. For target pixels whose shooting view direction is closer to the target view direction, the pixel features of that target pixel account for a larger proportion in the 3D point features corresponding to the 3D point. Here, the target view direction is the view direction specified during rendering. The shooting view direction of the target pixel is the direction in which the optical center of the camera points to the 3D point corresponding to the target pixel when the image containing the target pixel is captured.

[0050] Subsequently, the electronic device performs 3D Gaussian sputtering rendering based on the 3D Gaussian sphere properties of each 3D point to achieve 3D reconstruction, including reproducing the 3D scene or generating an image of the 3D scene in the target viewpoint direction.

[0051] By considering the contribution of the target viewpoint direction to the pixel features of each target pixel and the 3D point features corresponding to the 3D point during fusion, the 3D Gaussian sphere properties of the 3D point determined by the fused 3D point features can be more applicable to the target viewpoint direction, thereby improving the rendering quality when rendering based on 3D Gaussian sphere properties.

[0052] In some possible cases, the pixel features of the target pixel may include at least one of pixel alignment features and color-pixel alignment features. When the pixel features of the target pixel include both pixel alignment features and color-pixel alignment features, the electronic device can determine 3D point feature 1 for each 3D point based on the pixel alignment features of each target pixel in L images, and determine 3D point feature 2 for each 3D point based on the color-pixel alignment features of each target pixel in L images. Then, the 3D Gaussian sphere properties of each 3D point are determined based on both the cue feature 1 and the 3D point feature 2.

[0053] Here, the color-pixel alignment feature of the target pixel is the pixel alignment feature of the target pixel plus the color value (e.g., RGB value) of the target pixel. The pixel alignment feature is obtained from the image features. This image feature is obtained by feature extraction from the image containing the target pixel. An encoder with image feature extraction capabilities (e.g., a convolutional neural network) can be used to extract features from the image containing the target pixel. Image features can include the pixel features of each pixel in the image, and the pixel features of a pixel include the color feature of that pixel and the geometric features of the region where the pixel is located. However, the color features obtained after feature extraction have insufficient ability to represent color. If only the pixel alignment feature is used to determine the 3D Gaussian sphere properties of a 3D point, there is a possibility of color distortion. Therefore, it is necessary to combine the pixel alignment feature and the color value to obtain the color-pixel alignment feature, and then use the color-pixel alignment feature and the pixel alignment feature together to determine the 3D Gaussian sphere properties corresponding to the 3D point, resulting in better color representation in the final rendering result. The pixel alignment feature is still retained in the color-pixel alignment feature because the color and geometric features in the pixel alignment feature can be used to positively influence the newly added color value in generating the 3D Gaussian sphere properties.

[0054] Figure 2An exemplary process is illustrated in which pixel alignment features of the target pixel and color-pixel alignment features are combined with 3D Gaussian sputtering techniques to achieve 3D reconstruction. A description of this process can be found in the following... Figure 2 Middle (1)- Figure 2 Description of the steps shown in (8).

[0055] See Figure 2 In section (1), the electronic device first acquires images of a three-dimensional scene (which can be referred to as the object being photographed, for example) taken from different angles. Figure 1 The L images obtained when the person shown in the image are displayed. L is an integer greater than or equal to 2. Figure 2 The example given is L = 2. Let's denote the two images captured by the electronic device as Image 1 and Image 2. Other values ​​of L can be found in the description of L = 2, and will not be repeated here.

[0056] In some possible scenarios, to ensure the speed and quality of 3D reconstruction using 3D reconstruction methods by electronic devices, the value range of L can be set, for example, L is an integer greater than or equal to 2 and less than or equal to 10. When the number of images (M) acquired by the electronic device that can be used for 3D reconstruction is greater than the maximum value of L (e.g., 10), the electronic device can filter the M images and determine the 10 images that meet the conditions for 3D reconstruction. The filtering process includes: determining the degree of matching (similarity) between the shooting direction of the M images and the target viewing direction; and determining the 10 images with the highest matching degree as the images for 3D reconstruction.

[0057] In some other possible scenarios, the minimum value of L is 2, to ensure the quality of 3D reconstruction performed by the electronic device. When the number (M) of images (M) available for 3D reconstruction acquired by the electronic device is greater than 2, the electronic device can filter the M images to determine which images meet the criteria for 3D reconstruction. The filtering process includes: determining the degree of matching (similarity) between the shooting direction and the target viewing direction of the M images, and using images with a matching degree greater than a preset value as images for 3D reconstruction. If the number of images with a matching degree greater than the preset value is less than 2, then the top 2 images with the highest matching degree among the M images are selected as images for 3D reconstruction.

[0058] In this context, 2 and 10 are illustrative examples. In practice, the value of L can depend on the processing speed of the electronic device; the faster the processing speed, the larger L can be. This should not be construed as limiting the scope of the embodiments described in the application.

[0059] Then, the electronic device begins to determine the pixel alignment features of each target pixel in Image 1 and Image 2. The process of determining the pixel alignment features of each target pixel can be found in [reference needed]. Figure 2 (2) Figure 2 Middle (3) and Figure 2 The content shown in (4) is shown in the middle.

[0060] refer to Figure 2 At point (2), the electronic device extracts point clouds based on images 1 and 2 to obtain the point cloud of the photographed object. The point cloud here includes a set of multiple three-dimensional points in the photographed object. Each three-dimensional point can be represented in the form of a vector, which includes three parameters, namely the distance of the three-dimensional point from the three axes of the world coordinate system.

[0061] The point cloud extraction process includes: First, using the intrinsic and extrinsic parameter data of the camera that captured L images (Image 1 and Image 2) and the depth information of these L images, the position of each pixel in the L images is converted into coordinate points in the world coordinate system. Each pixel's coordinate point serves as its corresponding 3D point in the world coordinate system. Then, duplicate 3D points are deduplicated, ensuring each 3D point is retained only once. Because overlapping areas may exist in different images—that is, different pixels describe the same position of the photographed object—duplicate 3D points need to be deduplicated. Deduplication methods can include: if the distance between two 3D points (3D point a and 3D point b) is close to or equal to 0, and the colors of the pixels corresponding to the two 3D points are the same, then the two 3D points are considered to overlap, and one of the 3D points, for example, 3D point a, is deleted. A correspondence is then established between the pixels corresponding to the deleted 3D point a and 3D point b.

[0062] See again Figure 2 At point (3), the electronic device extracts image features of image 1, denoted as image feature 1, and extracts image features of image 2, denoted as image feature 2. Image feature 1 and image feature 2 respectively include the pixel features of each pixel in image 1 and image 2. The pixel features of a pixel include the color features of that pixel and the geometric features of the region where the pixel is located.

[0063] Then, refer to Figure 2 At point (4), the electronic device performs feature alignment based on point cloud and image features to determine the pixel alignment features of each target pixel in Image 1 and Image 2. This process includes: when projecting the 3D point corresponding to the pixel in the point cloud onto the image to which the pixel belongs, if the projection position coincides with the pixel, then the pixel is the target pixel. Then, the electronic device uses the pixel features of the target pixel in the image features (the image features of the image to which the target pixel is located) as the pixel alignment features of the target pixel.

[0064] It should be noted that if a pixel corresponds to a 3D point in the point cloud of the photographed object, but the 3D point cannot coincide with the pixel when projected onto the image to which the pixel belongs, then the pixel cannot be called the target pixel.

[0065] It should also be noted that the 3D points corresponding to each target pixel in Images 1 and 2 include both first-type and second-type 3D points. The first-type 3D points correspond to target pixels in multiple images out of L images; this can be understood as first-type 3D points having target pixels in both Images 1 and 2. The second-type 3D points have target pixels in only one image.

[0066] After determining the pixel alignment features of each target pixel in Images 1 and 2, refer to Figure 2 As shown in (5), the color value can be added to the pixel alignment feature of each target pixel to obtain the color-pixel alignment feature of each target pixel in image 1 and the color-pixel alignment feature of each target pixel in image 2.

[0067] Then, the electronic device uses the pixel alignment features of the target pixels corresponding to the second type of 3D points as 3D point feature 1 corresponding to the second type of 3D points. The pixel alignment features of multiple target pixels corresponding to the first type of 3D points are fused to obtain 3D point feature 1 corresponding to the first type of 3D points. This fusion process can be referenced... Figure 2 The content shown at (6a) is as follows. Furthermore, the electronic device uses the color-pixel alignment features of the target pixels corresponding to the second type of 3D points as the 3D point feature 2 corresponding to the second type of 3D points. The 3D point feature 2 corresponding to the first type of 3D points is obtained by fusing the color-pixel alignment features of multiple target pixels corresponding to the first type of 3D points; this fusion process can be referred to... Figure 2 The content shown in (6b) is shown in the middle.

[0068] like Figure 2 As shown in section (6a), the electronic device can fuse the pixel alignment features of all target pixels corresponding to the first type of three-dimensional point to obtain the fused pixel alignment features, which are used as the three-dimensional point features 1 corresponding to the first type of three-dimensional point. This process includes: fusing the pixel alignment features of all target pixels corresponding to the i-th three-dimensional point (three-dimensional point i) in each type of three-dimensional point. Figure 2 The pixel alignment features of two target pixels (i.e., 2 pixels in the image) are used as values ​​(also called value vectors). The shooting viewpoint direction of each target pixel corresponding to 3D point i is used as a key (also called key vector). The target viewpoint direction is used as a query (also called query vector). These are input into an attention mechanism model (attention mechanism model 1) that focuses on geometric perception for fusion, resulting in the fused pixel alignment features, which are used as the 3D point feature 1 of the i-th 3D point. The value of i ranges from 1 to the total number of all first-class 3D points.

[0069] The process of obtaining the fused pixel alignment features of 3D point i using the attention mechanism model 1 can be referenced from [reference needed]. Figure 3 Description of steps S11-S14.

[0070] S11. Take the pixel alignment features of each target pixel corresponding to the three-dimensional point i as the value, the shooting view direction of each target pixel as the key, and the target view direction as the query input into the attention mechanism model 1.

[0071] S12. The attention weights between the target viewing direction and the shooting viewing direction of each target pixel are calculated using the attention mechanism model 1.

[0072] Electronic devices use an attention mechanism model to calculate the attention weight between each key (the shooting view direction of each target pixel) and the query (the target view direction). The more similar a key is to the query, the greater the attention weight of that key.

[0073] In some cases, the attention weight A can be obtained using the following formula.

[0074]

[0075] In the above formula, Q is the query vector, K is the key vector, and d k This refers to the dimension of the key vector, also known as the scaling factor. It should be noted that the aforementioned formula is an exemplary formula for calculating attention weights in an attention mechanism model; other calculation methods are possible, and this application does not limit this approach. For example, the attention weight A can be calculated based on the angle between each key vector and the query vector; the larger the angle, the smaller the attention weight.

[0076] S13. The pixel alignment features of each target pixel are adjusted by using attention mechanism model 1 based on the attention weights corresponding to the shooting view direction of each target pixel.

[0077] The adjustments involved here include multiplying the attention weight corresponding to the shooting viewpoint direction of the target pixel by the pixel alignment feature of the target pixel. This way, the pixel alignment feature of the target pixel corresponding to a shooting viewpoint direction with a larger attention weight can be further enhanced.

[0078] S14. The pixel alignment features of each target pixel after adjustment are fused using attention mechanism model 1 to obtain the fused pixel alignment features.

[0079] The fusion involved here includes: the electronic device adds the adjusted pixel alignment features of each target pixel through an attention mechanism model 1 to obtain the fused pixel alignment features. The dimension of the fused pixel alignment features is the same as the dimension of the pixel alignment features of the target pixels.

[0080] See again Figure 2 As shown in section (6b), the electronic device can fuse the color-pixel alignment features of all target pixels corresponding to the first type of 3D points to obtain the fused color-pixel alignment features, which serve as the 3D point feature 2 corresponding to the first type of 3D point. This process includes: fusing the color-pixel alignment features of all target pixels corresponding to the i-th 3D point (3D point i) in each type of 3D point. Figure 2 The color-pixel alignment features of two target pixels (i, y) are used as values ​​(value vectors), the shooting viewpoint direction of each target pixel corresponding to 3D point i is used as a key (key vector), and the target viewpoint direction is used as a query (query vector). These are input into a color-focused attention mechanism model (attention mechanism model 1) for fusion to obtain the fused color-pixel alignment features, which are used as the 3D point feature 2 of the i-th 3D point. The value of i ranges from 1 to the total number of all first-class 3D points.

[0081] Here, the process of obtaining the 3D point feature 2 of the i-th 3D point (3D point i) among the first type of 3D points can be referred to the section on... Figure 4 Description of steps S21-S24.

[0082] S21. Take the color-pixel alignment feature of each target pixel corresponding to the three-dimensional point i as the value, take the shooting view direction of each target pixel as the key, and take the target view direction as the query input into the attention mechanism model 2.

[0083] S22. The attention weights between the target viewing direction and the shooting viewing direction of each target pixel are calculated using the attention mechanism model 2.

[0084] S23. The color-pixel alignment features of each target pixel are adjusted by using attention mechanism model 2 based on the attention weights corresponding to the shooting view direction of each target pixel.

[0085] S24. The adjusted color-pixel alignment features of each target pixel are fused using attention mechanism model 2 to obtain the fused pixel alignment features.

[0086] Here, the content involved in steps S21-S24 is similar to that involved in the aforementioned steps S11-S14. The pixel alignment features involved in steps S11-S14 are modified to color-pixel alignment features, and attention mechanism model 1 is modified to attention mechanism model 2. This embodiment of the application will not elaborate further on this.

[0087] Since the 3D point feature 2 corresponding to the 3D point includes more detailed color information about the 3D point, the electronic device can determine the spherical harmonic factor (which can be used to characterize color distribution) of the spherical region where the 3D point is located in the 3D Gaussian sphere attribute based on the 3D point feature 2 corresponding to the 3D point. Furthermore, it can use the 3D point feature 1 corresponding to the 3D point to determine other attributes in the 3D Gaussian sphere attribute besides the spherical harmonic factor of the spherical region where the 3D point is located.

[0088] like Figure 2 As shown in section (7), the electronic device can input the 3D point feature 1 and the 3D point feature 2 corresponding to the 3D point into a multilayer perceptron (MLP) network model (denoted as MLP network model 1) to obtain the 3D Gaussian sphere property corresponding to the 3D point. MLP network model 1 includes MLP network layer 1, MLP network layer 2 and MLP network layer 3. Among them, MLP network layer 1 is used to process the 3D point feature 1 corresponding to the 3D point to obtain the processed 3D point feature 1 (e.g. Figure 2 The first layer of the MLP network outputs the processed pixel alignment features corresponding to the 3D points. The second layer of the MLP network transforms the processed 3D point feature 1 into attribute 1 of the 3D Gaussian sphere attributes of the 3D point. Here, attribute 1 includes all attributes of the 3D Gaussian sphere attributes except for the spherical harmonic factor. The third layer of the MLP network combines the processed 3D point feature 1 and the 3D point feature 2, and transforms the combined 3D point feature into attribute 2 of the 3D Gaussian sphere attributes of the 3D point. Here, attribute 2 is the spherical harmonic factor of the spherical region where the 3D point is located. Then, attribute 1 and attribute 2 are combined to obtain the 3D Gaussian sphere attributes of the 3D point.

[0089] In some possible cases, the processing performed by MLP network layer 1 on the 3D point feature 1 corresponding to the 3D point may include at least one of dimensionality increase and / or dimensionality reduction. For example, when the 3D point feature 1 corresponding to the 3D point is greater than the dimension of attribute 1, dimensionality reduction can be performed, or dimensionality increase can be performed followed by dimensionality reduction. When the 3D point feature 1 corresponding to the 3D point is less than the dimension of attribute 1, dimensionality increase can be performed, or dimensionality reduction can be performed followed by dimensionality increase. The specific processing method depends on the actual application, and this application embodiment does not limit this.

[0090] Here, the electronic device can sequentially input the three-dimensional point feature 1 and three-dimensional point feature 2 corresponding to each three-dimensional point into the MLP network model 1 to obtain the 3D Gaussian sphere properties corresponding to each three-dimensional point.

[0091] It should be noted that the structure of MLP network model 1 is not unique. Figure 2The examples shown are merely illustrative and should not be construed as limiting the embodiments of this application. For instance, the input to the MLP network layer 3 may not include the processed 3D point feature 1, but only use the 3D point feature 2 to transform and obtain attribute 2. The embodiments of this application do not limit this.

[0092] It should also be noted that in the structure of this MLP network model 1, combining the processed 3D point feature 1 with the 3D point feature 2 before inputting it into the MLP network layer 3 to obtain attribute 2 allows the calculation of attribute 2 to be standardized by the 3D point feature 1 (processed) that yields attribute 1, establishing the connection between attribute 1 and attribute 2. When the 3D point is of the first type, the 3D point features input into the MLP network model 1 are the fused pixel alignment features (3D point feature 1) and the fused color-pixel alignment features (3D point feature 2) corresponding to the first type of 3D point. When the 3D point is of the second type, the 3D point features input into the MLP network model 1 are the unfused pixel alignment features (3D point feature 1) and the unfused color-pixel alignment features (3D point feature 2) corresponding to the second type of 3D point.

[0093] In some possible scenarios, to improve the processing efficiency of electronic devices, the device can generate the color-pixel alignment features of all target pixels corresponding to a 3D point after determining the pixel alignment features of all target pixels corresponding to that 3D point. If the 3D point is a second-class 3D point, the MLP network model 1 is used to determine the 3D Gaussian sphere properties of the 3D point. If the 3D point is a first-class 3D point, the pixel features are fused using an attention mechanism model, and then the MLP network model 1 is used to determine the 3D Gaussian sphere properties of the 3D point.

[0094] Figure 5 The diagram shows a schematic of obtaining the 3D Gaussian sphere properties of a 3D point based on the pixel features of all target pixels corresponding to the 3D point.

[0095] like Figure 5As shown, taking a 3D point P as an example, this 3D point P is a first-class 3D point, and its corresponding target pixels include pixel P1 in image 1 and pixel P2 in image 2. Here, the electronic device uses the shooting viewpoint direction of pixel P1 and the shooting viewpoint direction of pixel P2 as keys, the pixel alignment features of pixel P1 and the pixel alignment features of pixel P2 as values, and the target viewpoint direction as a query input to the attention mechanism model 1 to obtain the fused pixel alignment features, which serve as 3D point feature 1 of 3D point P. Similarly, the electronic device uses the shooting viewpoint direction of pixel P1 and the shooting viewpoint direction of pixel P2 as keys, the color-pixel alignment features of pixel P1 (pixel alignment features plus color value) and the color-pixel alignment features of pixel P2 as values, and the target viewpoint direction as a query input to the attention mechanism model 2 to obtain the fused color-pixel alignment features, which serve as 3D point feature 2 of 3D point P. Then, the electronic device inputs 3D point feature 1 and 3D point feature 2 of 3D point P into MLP network model 1 to obtain the 3D Gaussian sphere attribute corresponding to the 3D point P. Subsequently, if the 3D Gaussian sphere of the 3D point P is visible in the target view direction, the 3D Gaussian sphere attribute corresponding to the 3D point P can be rendered into the target image, which is an image describing the photographed object in the target view direction.

[0096] Figure 5 The description uses the first type of 3D point as an example. The process of obtaining the 3D Gaussian sphere property corresponding to a second type of 3D point based on the pixel features of all target pixels can be found in [reference needed]. Figure 5 Simply remove the content related to pixel feature fusion; this application's embodiments will not elaborate further on this.

[0097] refer to Figure 2 As shown in Figure (8), after determining the 3D Gaussian sphere properties of each 3D point, the electronic device can render the image based on the target shooting viewpoint direction to obtain an image describing the photographed object from the target viewing viewpoint direction. The rendering process can be found in the following... Figure 6 As shown.

[0098] like Figure 6 As shown, during rendering, the electronic device can determine the view frustum of the 3D Gaussian sphere corresponding to each 3D point based on the target viewpoint direction. The view frustum simulates the scene of observing the 3D Gaussian sphere corresponding to each 3D point from the target viewpoint direction. The 3D Gaussian sphere corresponding to each 3D point is characterized by its properties. The electronic device selects the 3D Gaussian sphere within the view frustum and projects it onto the image plane; 3D Gaussian spheres outside the view frustum are not projected. Then, the electronic device displays the content on the projected image plane, obtaining the image (target image) of the object being photographed from the target viewpoint direction.

[0099] This can be understood as follows: when viewed from the target's perspective, the 3D points corresponding to the 3D Gaussian sphere properties representing the 3D Gaussian sphere within the view frustum are visible. The 3D points corresponding to the 3D Gaussian sphere properties representing the 3D Gaussian sphere outside the view frustum are invisible.

[0100] It should be noted here that... Figure 2 The operation shown in (2) is as follows Figure 2 The execution order of the operations shown in (3) is not sequential. When calculating the three-dimensional point features of a three-dimensional point, Figure 2 The operation shown in (6a) and Figure 2 The order of operations shown in (6b) is not important. To ensure efficiency in 3D reconstruction, Figure 2 The operation shown in (6a) and Figure 2 The operations shown in (6b) can be performed simultaneously.

[0101] In some cases, to further improve the quality of the rendering results, before rendering, after obtaining the 3D Gaussian sphere properties corresponding to each 3D point, these properties can be optimized. This optimization may include trimming and / or densification. Trimming involves removing noise from each 3D Gaussian sphere property. Densification involves increasing the number of 3D Gaussian spheres to obtain more 3D Gaussian sphere properties corresponding to more 3D points. Then, the trimmed and / or densified 3D Gaussian sphere properties are used for rendering.

[0102] It should be noted that here Figure 2 In addition to the rendering technology used, the rendering technology used in (8) can be the 3D Gaussian sputtering rendering technology mentioned above.

[0103] It should be noted here that... Figure 2 This example uses the pixel features of the target pixel, including pixel alignment features and color-pixel alignment features, as an example. In some possible embodiments, the pixel features of the target pixel may only include pixel alignment features or color-pixel alignment features, and this application does not limit this.

[0104] Figure 7 An exemplary flowchart is shown, illustrating the process of an electronic device performing 3D reconstruction based on 3D Gaussian sputtering rendering technology when the pixel features of the target pixel are pixel alignment features or color-pixel alignment features.

[0105] S101. The electronic device acquires L images, which are images obtained when the subject is photographed from different angles.

[0106] For details regarding the L images, please refer to the aforementioned content; they will not be repeated here. It should be noted that these L images still contain depth information, allowing the electronic device to determine the point cloud of the subject.

[0107] S102. Determine the pixel feature a of each target pixel in L images. The target pixel corresponds to a three-dimensional point in the point cloud of the photographed object.

[0108] For information on point clouds of the subject being photographed and related content on determining point clouds, please refer to the aforementioned section. Figure 2 The description of the content shown in (2) will not be repeated here.

[0109] Here, pixel feature 'a' can include pixel alignment features or color-pixel alignment features. For details on obtaining pixel alignment features, please refer to the aforementioned section. Figure 2 The description of the content shown in section (4) is provided above. For details regarding obtaining color-pixel alignment features, please refer to the aforementioned section. Figure 2 The description of the content shown in (5) is omitted here.

[0110] S103. Based on the target viewpoint direction, fuse the pixel features a of all target pixels corresponding to each first type of three-dimensional point to obtain the three-dimensional point features a corresponding to each first type of three-dimensional point.

[0111] When pixel feature a is a pixel alignment feature, the electronic device can use the aforementioned attention mechanism model 1 to fuse the pixel features a of all target pixels corresponding to the first type of 3D points to obtain the 3D point features a corresponding to each of the first type of 3D points. At this time, the 3D point features a are the aforementioned fused pixel alignment features.

[0112] When pixel feature a is a color-pixel alignment feature, the electronic device can use the aforementioned attention mechanism model 2 to fuse the pixel features a of all target pixels corresponding to the first type of 3D points to obtain the 3D point features a corresponding to each of the first type of 3D points. At this time, the 3D point features a are the aforementioned fused color-pixel alignment features.

[0113] S104. Determine the 3D Gaussian sphere attribute of each 3D point based on the 3D point feature a corresponding to each 3D point, where the 3D point feature a corresponding to the second type of 3D point is the pixel feature a of the target pixel corresponding to the second type of 3D point.

[0114] The electronic device sequentially inputs the 3D point features 'a' corresponding to each 3D point into the MLP network model 'a'. The MLP network model 'a' then transforms the 3D point features 'a' corresponding to each 3D point into the 3D Gaussian sphere properties corresponding to that 3D point. The MLP network layers in the MLP network model 'a' can be used to process the 3D point features 'a' corresponding to each 3D point (including dimensionality increase and / or dimensionality reduction), and then transform the processed 3D point features 'a' into the 3D Gaussian sphere properties corresponding to each 3D point.

[0115] S105. Render the image of the photographed object according to the target viewpoint direction using the 3D Gaussian sphere properties of each 3D point.

[0116] For a description of the rendering process, please refer to the foregoing. Figure 2 The content shown in (8) will not be repeated here.

[0117] It should be noted here that when pixel feature a is a pixel alignment feature, in order to better characterize the color distribution of the spherical region where the 3D point is located, in addition to using pixel feature a, pixel feature b can also be introduced. This pixel feature b is the aforementioned color-pixel alignment feature. For the process of determining the 3D Gaussian sphere properties of a 3D point using pixel features a and pixel feature b, please refer to the aforementioned... Figure 2 Middle (1) - Figure 2 The content shown in (7) will not be repeated here.

[0118] It should also be noted that the aforementioned MLP network models (including MLP network model a and MLP network model 1) are based on training. In practice, it has been found that when training MLP network models, they are prone to getting trapped in local optima when predicting the position and opacity of the spherical region containing a 3D point in a 3D Gaussian attribute. That is, the obtained position and opacity attributes are not globally optimal, easily leading to errors. Therefore, during model training, the predicted position and opacity can be converted into depth probability values ​​or bias values. Then, these depth probability values ​​or bias values ​​can be used to obtain the final position and opacity attributes. This conversion reduces the impact of local optima on the final prediction results, making the model more likely to obtain a globally optimal solution, thus improving the accuracy and robustness of the prediction.

[0119] It should also be noted that, in this application embodiment, besides using L images to achieve 3D reconstruction, 3D reconstruction can also be achieved based on a single-view video sequence; this application embodiment does not limit this. Furthermore, the subject being photographed should remain stationary or be a stationary object. If the subject moves during the capture of L images or the single-view video sequence, the L images and the single-view video sequence used for 3D reconstruction should be acquired again. The camera used for shooting can be the camera of an electronic device, or it can be another electronic device that captures L images and transmits the camera's intrinsic and extrinsic parameters to the electronic device performing 3D reconstruction.

[0120] The exemplary electronic device provided in the embodiments of this application will be introduced first below.

[0121] Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application.

[0122] The following uses an electronic device as an example to illustrate the embodiments in detail. It should be understood that the electronic device may have more than Figure 8 The more or fewer components shown can be combined into two or more components, or they can have different component configurations. Figure 8 The various components shown can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0123] The electronic device may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0124] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0125] Electronic devices implement display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0126] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD). The display panel can also be manufactured using organic light-emitting diodes (OLEDs), active-matrix organic light-emitting diodes (AMOLEDs), flexible light-emitting diodes (FLEDs), miniled, microled, micro-OLEDs, quantum dot light-emitting diodes (QLEDs), etc. In some embodiments, the electronic device may include one or N displays 194, where N is a positive integer greater than 1.

[0127] Electronic devices can achieve shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0128] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, converting it into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and color. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0129] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device may include one or N cameras 193, where N is a positive integer greater than 1.

[0130] Digital signal processors (DSPs) are used to process digital signals, including digital image signals and other digital signals. For example, when an electronic device is selecting a frequency, a DSP can perform Fourier transforms on the frequency energy.

[0131] Video codecs are used to compress or decompress digital video. Electronic devices can support one or more video codecs. This allows the electronic device to play or record video in various encoded formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0132] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0133] In this embodiment of the application, the processor 110 can call computer instructions stored in the internal memory 121 to cause the electronic device to execute the method in this embodiment of the application.

[0134] This application also provides a chip system including at least one processor for implementing the functions involved in the methods performed by the electronic device in any of the above embodiments.

[0135] In one possible design, the chip system also includes a memory for storing program instructions and data, which may be located within or outside the processor.

[0136] The chip system can consist of chips or include chips and other discrete components.

[0137] Optionally, the chip system may contain one or more processors. These processors can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor, implemented by reading software code stored in memory.

[0138] Optionally, the chip system may contain one or more memories. These memories may be integrated with the processor or separated from it; this application does not limit the specific implementation.

[0139] For example, the memory can be a non-transient processor, such as a read-only memory (ROM), which can be integrated on the same chip as the processor or set on different chips. This application does not specifically limit the type of memory or the way the memory and processor are set.

[0140] This application also provides a computer program product comprising: a computer program (also referred to as code or instructions) that, when run, causes a computer to perform the method executed by the electronic device in any of the above embodiments.

[0141] This application also provides a computer-readable storage medium storing a computer program (also referred to as code or instructions). When the computer program is run, it causes the computer to perform the method executed by the electronic device in any of the above embodiments.

[0142] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0143] As used in the above embodiments, depending on the context, the term "when..." can be interpreted as meaning "if...", "after...", "in response to determining...", or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if (the stated condition or event) is interpreted as meaning "if determining...", "in response to determining...", "when (the stated condition or event) is detected", or "in response to detecting (the stated condition or event)".

[0144] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to and includes any or all possible combinations of one or more of the listed items.

[0145] The terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0146] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0147] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A three-dimensional reconstruction method, characterized in that, Applied to electronic devices, including: Obtain L images, where the L images are images obtained from taking pictures of the first object from different angles; where L is an integer greater than or equal to 2. The first pixel feature vector of each target pixel in L images is determined. A target pixel corresponds to a 3D point in the point cloud of the first object, and when the 3D point corresponding to the target pixel is projected onto the image to which the target pixel belongs, it coincides with the target pixel. The 3D points corresponding to each target pixel include a first type of 3D point and a second type of 3D point. The first type of 3D point corresponds to target pixels in multiple images. The second type of 3D point corresponds to a target pixel in only one image. Based on the first viewpoint direction, the first pixel feature vectors of all target pixels corresponding to each first type of 3D point are fused to obtain the first 3D point feature vector corresponding to each first type of 3D point. All target pixels corresponding to a first type of 3D point include a first pixel and a second pixel. Compared with the shooting viewpoint direction of the second pixel, the shooting viewpoint direction of the first pixel is closer to the first viewpoint direction. The proportion of the first pixel feature vector of the first pixel in the first 3D point feature vector corresponding to a first type of 3D point is greater than that of the second pixel feature vector. The shooting viewpoint direction of a target pixel is the direction in which the optical center of the camera points to the 3D point corresponding to the target pixel when the image of the target pixel is captured. The 3D Gaussian sphere attribute of each 3D point is determined based on the first 3D point feature vector corresponding to each 3D point; the 3D Gaussian sphere attribute of a 3D point is used to describe at least the shape, position and color distribution of the spherical region where the 3D point is located; the first 3D point feature vector corresponding to a second type of 3D point in each 3D point is the first pixel feature vector of the target pixel corresponding to the second type of 3D point. The target image is obtained by rendering the 3D Gaussian sphere properties of each 3D point according to the first viewpoint direction. The target image describes the content of the first object captured in the first viewpoint direction.

2. The method according to claim 1, characterized in that, The first pixel feature vector of a target pixel includes at least the pixel alignment feature of the target pixel; wherein, the process of determining the pixel alignment feature of the target pixel includes: Extract image features from a first image, which includes the target pixel; the image features record the pixel features of each pixel in the first image; The three-dimensional point corresponding to the target pixel in the point cloud is projected onto the first image. If the projection position coincides with the position of the target pixel, the pixel feature of the target pixel in the image features is used as the pixel alignment feature of the target pixel.

3. The method according to claim 2, characterized in that, The first pixel feature vector of a target pixel includes the pixel alignment feature of the target pixel plus the color value of the target pixel.

4. The method according to claim 1, characterized in that, Based on the first-view direction, the first pixel feature vectors of all target pixels corresponding to each first-type 3D point are fused to obtain the first 3D point feature vector corresponding to each first-type 3D point, specifically including: The first pixel feature vector of each target pixel corresponding to the i-th 3D point in each first type of 3D point is used as the first value vector, the shooting view direction of each target pixel is used as the key vector, and the first view direction is used as the query vector. These are input into the first attention mechanism model to obtain the first 3D point feature vector corresponding to the i-th 3D point. The value of i ranges from 1 to the number of each first type of 3D point. The first attention mechanism model is used to calculate the attention weight between the query vector and each key vector. The more similar a key vector is to the query vector, the greater the attention weight between the key vector and the query vector. The first attention mechanism model is also used to perform a weighted summation of the first value vector based on the attention weight to obtain the first 3D point feature vector corresponding to the i-th 3D point.

5. The method according to any one of claims 1-4, characterized in that, The 3D Gaussian sphere properties of each 3D point are determined based on the feature vector of the first 3D point corresponding to each 3D point, specifically including: The first three-dimensional point feature vectors corresponding to each three-dimensional point are sequentially input into the first multilayer perceptron (MLP) network model. The first MLP network model converts the first three-dimensional point feature vectors corresponding to each three-dimensional point into 3D Gaussian sphere attributes, thereby obtaining the 3D Gaussian sphere attributes of the three-dimensional points.

6. The method according to claim 2, characterized in that, When the first pixel feature vector of a target pixel is the pixel alignment feature of that target pixel, after determining the first pixel feature vectors of each target pixel in L images, the method further includes: The second pixel feature vector of each target pixel in L images is determined. The second pixel feature vector of a target pixel is the pixel alignment feature of the target pixel plus the color value of the target pixel. Based on the first viewpoint direction, the second pixel feature vectors of all target pixels corresponding to each first type of 3D point are fused to obtain the second 3D point feature vectors corresponding to each first type of 3D point. The 3D Gaussian sphere properties of each 3D point are determined based on the feature vector of the first 3D point corresponding to each 3D point, specifically including: The 3D Gaussian sphere properties of each 3D point are determined based on the first 3D point feature vector and the second 3D point feature vector corresponding to each 3D point; the second 3D point feature vector corresponding to a second type of 3D point in each 3D point is the second pixel feature vector of the target pixel corresponding to the second type of 3D point.

7. The method according to claim 6, characterized in that, The 3D Gaussian sphere properties of one of the three-dimensional points include: the rotation component and position of the spherical region where the three-dimensional point is located in the world coordinate system, the scale component, opacity, and spherical harmonic factor of the spherical region where the three-dimensional point is located, and the spherical harmonic factor is used to characterize the color distribution of the spherical region.

8. The method according to claim 6, characterized in that, The 3D Gaussian sphere properties of each 3D point are determined based on the first and second 3D point feature vectors corresponding to each 3D point, respectively, including: Based on the first three-dimensional point feature vector of the kth three-dimensional point among all the three-dimensional points, a first attribute in the 3D Gaussian sphere properties of the kth three-dimensional point is determined, and based on the second three-dimensional point feature vector of the kth three-dimensional point among all the three-dimensional points, a second attribute in the 3D Gaussian sphere properties of the kth three-dimensional point is determined; wherein, k ranges from 1 to the number of three-dimensional points, and the second attribute is the spherical harmonic factor of the spherical region where the kth three-dimensional point is located; The first and second attributes are combined as the 3D Gaussian sphere attributes of the k-th 3D point.

9. The method according to claim 8, characterized in that, The first attribute of the 3D Gaussian sphere property of the k-th 3D point is determined based on the first 3D point feature vector of the k-th 3D point, and the second attribute of the 3D Gaussian sphere property of the k-th 3D point is determined based on the second 3D point feature vector of the k-th 3D point, specifically including: The first and second 3D feature vectors of the k-th 3D point are input into the second MLP network model to obtain the first attribute and the second attribute. The second MLP network model includes a first MLP network layer, a second MLP network layer, and a third MLP network layer. The first MLP network layer processes the first 3D feature vector of the k-th 3D point to obtain a processed first 3D feature vector. The second MLP network layer converts the processed first 3D feature vector into the first attribute. The third MLP network layer combines the processed first 3D feature vector and the second 3D feature vector of the k-th 3D point, and then converts the combined 3D feature vector into the second attribute.

10. The method according to any one of claims 6-9, characterized in that, Based on the first-view direction, the second pixel feature vectors of all target pixels corresponding to each first-type 3D point are fused to obtain the second 3D point feature vector corresponding to each first-type 3D point, specifically including: The second pixel feature vector of each target pixel corresponding to the i-th 3D point in each of the first type of 3D points is used as the second value vector, the shooting view direction of each target pixel is used as the key vector, and the first view direction is used as the query vector. These are input into the second attention mechanism model to obtain the second 3D point feature vector corresponding to the i-th 3D point. The value of i ranges from 1 to the number of each of the first type of 3D points. The second attention mechanism model is used to calculate the attention weight between the query vector and each key vector. The more similar a key vector is to the query vector, the greater the attention weight between the key vector and the query vector. The first attention mechanism model is also used to perform a weighted summation of the second value vector based on the attention weight to obtain the second 3D point feature vector corresponding to the i-th 3D point.

11. An electronic device, characterized in that, include: One or more processors and a memory; the memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1-10.

12. A computer-readable storage medium comprising computer instructions, characterized in that, When the computer instructions are executed on an electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1-10.

13. A chip system applied to an electronic device, characterized in that, The chip system includes one or more processors, which are configured to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1-10.

14. A computer program product containing instructions, characterized in that, When the computer program product is run on an electronic device, it causes the electronic device to perform the method as described in any one of claims 1-10.

Images