A three-dimensional reconstruction method of an object and an electronic device

Abstract

The application discloses a kind of object three-dimensional reconstruction method and electronic equipment, applied to computer vision technical field, comprising: in the surrounding space of object enclosing ball, establish N camera sampling view angle;Determine each missing view angle in N camera sampling view angle not covered to original input image based on original input image;Establish the missing view angle set formed by at least 1 missing view angle;For the missing view angle in the missing view angle set, obtain the image similar in view angle from original input image as reference image, and generate the target image of missing view angle using new view angle image generation model, obtain the missing view angle image set containing each target image;Based on original input image and missing view angle image set, three-dimensional reconstruction is carried out to object, and the three-dimensional reconstruction result of object is obtained.The scheme of the application can effectively realize the three-dimensional reconstruction of object, and the reconstruction quality can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to an object three-dimensional reconstruction method and an electronic device. BACKGROUND

[0002] Current 3D reconstruction technology requires dense multi-view input, for example, each sample in the commonly used Nerf-Synthetic dataset contains hundreds of images, and each sample in the MVImgNet dataset contains about 30 images. Through dense multi-view input, high-quality object 3D reconstruction can be achieved.

[0003] However, in actual applications, the input images often have insufficient view coverage, especially at non-typical views such as the top and bottom of the object. Lack of image input can cause geometric distortion, holes, or texture loss in the 3D reconstruction result, affecting the reconstruction quality.

[0004] To sum up, how to effectively realize object three-dimensional reconstruction and effectively improve the reconstruction quality is a technical problem that needs to be solved by the technical personnel in the field at present. SUMMARY

[0005] The present application provides an object three-dimensional reconstruction method and an electronic device to effectively realize object three-dimensional reconstruction and effectively improve the reconstruction quality.

[0006] To solve the above technical problems, the present application provides the following technical solutions:

[0007] The present application provides an object three-dimensional reconstruction method, comprising:

[0008] In the surrounding space of the object bounding sphere, N camera sampling view angles are established; N is a positive integer;

[0009] Based on the original input images, each missing view angle in the N camera sampling view angles that is not covered by the original input images is determined;

[0010] A missing view angle set composed of at least one missing view angle is established;

[0011] For the missing view angle in the missing view angle set, an image with a similar view angle is obtained from the original input images as a reference image, and a new view image generation model is used to generate a target image of the missing view angle, to obtain a missing view image set containing each target image;

[0012] Based on the original input images and the missing view image set, the object is three-dimensionally reconstructed to obtain a three-dimensional reconstruction result of the object.

[0013] The present application also provides an electronic device, comprising:

[0014] a memory for storing a computer program;

[0015] a processor for implementing the steps of the object three-dimensional reconstruction method as described above when executing the computer program.

[0016] The present application scheme considers that incomplete view coverage will affect the quality of object three-dimensional reconstruction, and therefore optimizes the view coverage. Specifically, first, N camera sampling views are established in the surrounding space of the object bounding sphere, and then based on the original input images, each missing view in the N camera sampling views that is not covered by the original input images is determined. It can be understood that if the N camera sampling views are all covered by the original input images, it means that the original input images achieve complete view coverage, and high-quality object three-dimensional reconstruction can be directly achieved. If there are missing views, the present application scheme will establish a missing view set composed of at least one missing view based on each missing view. Then, for the missing views in the missing view set, the image of the view is supplemented, specifically, an image with a similar view is obtained from the original input images as a reference image, and a new view image generation model is used to generate the target image of the missing view, so that the missing view image set containing each target image can be obtained. The missing view image set is a view supplement to the original input images, and therefore based on the original input images and the missing view image set, the view coverage degree can be effectively improved, and the object can be three-dimensionally reconstructed to obtain a high-quality three-dimensional reconstruction result of the object.

[0017] In summary, the present application scheme can effectively achieve object three-dimensional reconstruction and improve the reconstruction quality. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0019] Figure 1 The implementation flowchart of the object three-dimensional reconstruction method provided by the specific embodiment of the present application;

[0020] Figure 2 The principle diagram of the pre-optimization and post-optimization in the specific embodiment of the present application;

[0021] Figure 3 The structural diagram of the electronic device provided by the specific embodiment of the present application;

[0022] Figure 4A structural schematic diagram of a computer readable storage medium of the present application. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, any person skilled in the art can obtain all other embodiments without creative work, which are within the protection scope of the present application.

[0024] It should be noted that, in the description of the present application, the terms "comprise", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment. The terms "first", "second" and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0025] In order for those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments. Please refer to Figure 1 Figure 1 An implementation flowchart of the object three-dimensional reconstruction method provided by the present application can include the following steps:

[0026] Step S101: In the surrounding space of the object enclosing sphere, N camera sampling view angles are established.

[0027] The present application can perform three-dimensional reconstruction of an object, which can be any object that can be photographed, and does not affect the implementation of the present application. The object enclosing sphere refers to a sphere that encloses the object. The diameter of the object enclosing sphere can be exactly equal to the longest axis of the object, so that the object enclosing sphere is tangent to the object, or it can be slightly larger than the longest axis of the object.

[0028] Sampling in the surrounding space of the object enclosing sphere refers to selecting sampling points in the surrounding area of the object enclosing sphere, so as to obtain a set of surrounding space view angles, that is, N camera sampling view angles. It can be understood that, in order to ensure the comprehensiveness of the view angles covered by the present application, N camera sampling view angles can be uniformly selected in the surrounding space of the object enclosing sphere. And N is a positive integer, the larger the value of N, the higher the density of the camera sampling view angles, the higher the accuracy of the coverage of the present application, but the calculation amount will also increase accordingly. In practical applications, N can be valued between 100 and 300. ​

[0029] In one specific embodiment of the present application, step S101 can specifically include:

[0030] N grid points are uniformly arranged on the object enclosing sphere, and the direction formed by the N grid points and the object center is respectively taken as the orientation parameter of each camera sampling view angle;

[0031] For any camera sampling view angle, the position parameter of the camera sampling view angle is obtained based on the observation radius of the camera, the coordinates of the object center, and the orientation parameter of the camera sampling view angle.

[0032] This embodiment considers that the camera sampling view angle can be represented as the orientation parameter and the position parameter, so that the orientation parameter of each of the N camera sampling view angles can be determined respectively, and then the corresponding position parameter is obtained in combination with the observation radius of the camera.

[0033] Specifically, N grid points can be uniformly arranged on the object enclosing sphere, for example, N Fibonacci grid points are sampled on the object enclosing sphere as the N grid points uniformly arranged on the object enclosing sphere. In order to facilitate calculation, the object enclosing sphere is regarded as a unit sphere, that is, the radius R obj of the object enclosing sphere can be regarded as 1. In actual application, the N grid points are arranged by using the Fibonacci grid sampling algorithm, which can ensure that the N grid points are uniformly distributed on the sphere surface and ensure the accuracy when calculating the coverage rate.

[0034] After the N grid points are uniformly arranged on the object enclosing sphere, the direction formed by each of the N grid points and the object center C is respectively the orientation parameter of each of the N camera sampling view angles. Taking the radius R obj of the object enclosing sphere as 1 and the coordinates of the object center C as (0, 0, 0) for example, if the north pole of the object enclosing sphere is one of the grid points, the coordinates of the grid point are (0, 0, 1), and the direction formed by the grid point and the object center C is the orientation parameter of the corresponding camera sampling view angle.

[0035] After obtaining the orientation parameter of each of the N camera sampling view angles, the position parameter of each of the N camera sampling view angles can be obtained in combination with the observation radius of the camera and the coordinates of the object center. For example, in one specific embodiment, for any camera sampling view angle, the position parameter of the camera sampling view angle can be obtained according to the calculation manner of O i =C+Rc×P i . Wherein, O i represents the position parameter of the i-th camera sampling view angle, C represents the coordinates of the object center, Rc represents the observation radius of the camera, and P iP i represents the orientation parameter of the i-th camera sampling view. It can be understood that i can take a value from 1 to N.

[0036] Still taking the grid point (0, 0, 1) as an example, since the coordinates of the object center C are (0, 0, 0), the orientation parameter P i of the corresponding camera sampling view in this example is (0, 0, 1), for example, Rc is 2, and the position parameter of the camera sampling view corresponding to the grid point in this example is (0, 0, 2), that is, at the position (0, 0, 2), one of the N camera positions is represented, and the direction from the camera position to the object center is the orientation of the camera sampling view. i Still taking the grid point (0, 0, 1) as an example, since the coordinates of the object center C are (0, 0, 0), the orientation parameter P i of the corresponding camera sampling view in this example is (0, 0, 1), for example, Rc is 2, and the position parameter of the camera sampling view corresponding to the grid point in this example is (0, 0, 2), that is, at the position (0, 0, 2), one of the N camera positions is represented, and the direction from the camera position to the object center is the orientation of the camera sampling view.

[0037] In addition, it should be noted that the value of the observation radius Rc of the camera can be set according to actual needs, but generally needs to be greater than the radius R of the object bounding sphere. obj To ensure that the object is completely within the view cone, that is, Rc can be set as Rc = a x R obj Here, the coefficient a needs to be greater than 1, and is usually set to a value between 2 and 3.

[0038] Step S102: determining each missing view in the N camera sampling views that is not covered by the original input image based on the original input image.

[0039] The original input image can include one or more images, and generally needs to include multiple images, that is, the original input image is a set including multiple images.

[0040] After obtaining the N camera sampling views, it is necessary to determine each view in the N camera sampling views that is not covered by the original input image as a missing view based on the original input image. Specifically, for any one camera sampling view, the degree of overlap between the camera sampling view and the original input image can be measured to determine whether the camera sampling view belongs to the missing view. Of course, the specific implementation of measuring the degree of overlap can be various, for example, the number of images in the original input image that overlap with the camera sampling view can be used as a measurement standard.

[0041] For example, in a specific embodiment of the present application, determining each missing view in the N camera sampling views that is not covered by the original input image based on the original input image can specifically include:

[0042] For any one camera sampling view in the N camera sampling views, determining the number of images in the original input image that overlap with the camera sampling view as a view;

[0043] Judging whether the number is lower than a preset overlap threshold.

[0044] If yes, it is determined that the camera sampling view is a missing view which is not covered by the original input images.

[0045] If no, it is determined that the camera sampling view is not a missing view which is not covered by the original input images.

[0046] In this embodiment, for any one of the N camera sampling views, the number of images in the original input images which overlap with the camera sampling view in view is determined, which is relatively simple and convenient in implementation. After the number of images which overlap in view is obtained, if the number is lower than a preset overlap threshold, it is determined that the camera sampling view is a missing view, otherwise it is not a missing view.

[0047] Further, in a specific embodiment of the present application, for any one of the N camera sampling views, determining the number of images in the original input images which overlap with the camera sampling view in view can specifically include:

[0048] determining the camera view of each image in the original input images;

[0049] for any one of the N camera sampling views, sequentially selecting each image from the original input images, and when the view angle between the camera sampling view and the image selected from the original input images is lower than a preset value, it is determined that the camera sampling view overlaps with the view of the image selected from the original input images, otherwise it does not overlap;

[0050] after sequentially selecting each image from the original input images and completing the determination, the number of images in the original input images which overlap with the camera sampling view in view is determined.

[0051] In this embodiment, for any one of the N camera sampling views, it is compared with each image in the original input images in sequence, so as to obtain the view angle between them, if the view angle is lower than a preset value, it is determined that the views overlap, otherwise they do not overlap. Therefore, for any one of the camera sampling views, after comparing it with each image in the original input images in sequence, the number of images which overlap with the view can be determined.

[0052] For example, in a specific embodiment, the above process can be represented as . P i represents the camera sampling view of the i-th camera in the N camera sampling views, that is, the orientation parameter of the i-th camera sampling view.d j represents the camera view of the j-th image in the D original input images. is the angle between the i-th camera sampling view and the camera view of the j-th image, if it is lower than a preset value 0.5*FOV, it can be determined that the two are overlapped, otherwise it is not overlapped. Overlap(i, j) is the judgment result of the angle overlap between the i-th camera sampling view and the j-th image. In this example, the preset value is specifically set to 0.5*FOV, FOV is a preset camera field of view, and in other cases, the value of the preset value can be adjusted according to actual needs.

[0053] In the above example, there are D images in the original input image, so it can be understood that for any one of the N camera sampling views, the number of images in the original input image that are overlapped with the camera sampling view ranges between [0, D].

[0054] In addition, it needs to be explained that in the above example, the camera view of each image in the original input image needs to be determined, and in specific implementation, it can be determined based on a related pose recognition model, which is relatively mature and convenient to implement, and the present application will not be expanded.

[0055] Step S103: establishing a missing view angle set composed of at least one missing view angle.

[0056] Through the above operation, each missing view angle in the N camera sampling views that is not covered by the original input image can be determined, and after obtaining these missing view angles, a missing view angle set composed of at least one missing view angle can be established based on the missing view angles, that is, one or more missing view angles in the missing view angles can be selected as the missing view angle set, for example, a simple implementation is to directly select all the missing view angles as the constructed missing view angle set.

[0057] In a specific embodiment of the present application, step S103 can specifically include:

[0058] determining whether the number of all missing view angles is lower than a preset first number threshold;

[0059] if yes, establishing a missing view angle set composed of all missing view angles;

[0060] if no, selecting K1*F missing view angles from all missing view angles based on the farthest point sampling algorithm, and establishing a missing view angle set composed of K1*F missing view angles;

[0061] wherein F represents the number of all missing view angles, and K1 is a preset coefficient value less than or equal to 1.

[0062] The implementation considers that if the number of missing view angles is not particularly large, that is, the number of all missing view angles is lower than the first number threshold, all missing view angles can be directly taken as the constructed missing view angle set. However, if the number of missing view angles is large, part of the missing view angles can be selected to construct the missing view angle set. Such an operation considers that in actual application, in order to effectively guarantee the quality of three-dimensional reconstruction, the value of N is usually set to be large, so that the density of the sampling view angles is high, which is beneficial to improve the coverage rate. On this basis, the number of missing view angles is likely to be large, and the density of these missing view angles is also high. Therefore, if part of the missing view angles is appropriately selected to construct the missing view angle set, the calculation amount can be effectively reduced, and the coverage rate will not be greatly affected, so that the quality of the three-dimensional reconstruction result of the object can be effectively guaranteed. Therefore, in the implementation, when the number of missing view angles is large, part of the missing view angles is selected to construct the missing view angle set.

[0063] In addition, when selecting the missing view angles, the missing view angles can be uniformly selected. Therefore, in the implementation, K1xF missing view angles are uniformly selected from all missing view angles based on the farthest point sampling algorithm, so as to establish a missing view angle set composed of K1xF missing view angles, K1 is a preset coefficient value less than or equal to 1, for example, 0.2 in one occasion. Through the farthest point sampling algorithm, the selected missing view angles can be uniformly distributed. Therefore, the missing view angle set composed of the missing view angles, together with the original input image, can comprehensively cover all directions of the object, so as to guarantee the quality of subsequent three-dimensional reconstruction.

[0064] In the above embodiment, the specific value of the preset first number threshold can be set according to actual needs, for example, can be set as a fixed value. However, it can be understood that the value of the first number threshold is 0 at the minimum and the number of images D in the original input image at the maximum. Further, in one specific implementation of the application, the coverage rate evaluation result of the original input image can be obtained. If the coverage rate evaluation result of the original input image is a centralized distribution of the original input image, the first number threshold can be set to 0. If the coverage rate evaluation result of the original input image is a decentralized distribution of the original input image, the first number threshold can be set to a value greater than 0, for example, usually set to a small integer. In the implementation, by selecting an appropriate value of the first number threshold, the number of missing view angles can be effectively adjusted, which is beneficial to make the number of missing view angles a moderate and appropriate value.

[0065] Specifically, if the original input images are centrally distributed, it indicates that there are a large number of uncovered view angles, and thus the number of missing view angles is prone to be large. In this case, the first number threshold is set to the minimum value 0, which is beneficial to appropriately reduce the number of missing view angles. Conversely, if the original input images are dispersedly distributed, it indicates that there are not many uncovered view angles, and thus the number of missing view angles is prone to be insufficient. In this case, the value of the first number threshold can be appropriately increased, which is beneficial to appropriately increase the number of missing view angles.

[0066] Step S104: For a missing view angle in the set of missing view angles, an image with a similar view angle is obtained from the original input images as a reference image, and a target image of the missing view angle is generated by using a new view image generation model, to obtain a set of missing view angle images containing each target image.

[0067] For each missing view angle in the set of missing view angles, a new view image generation model needs to be used to generate an image under the view angle, which is referred to as a target image of the missing view angle. In this process, an image with a similar view angle to the missing view angle in the original input images needs to be used as a reference image. The new view image generation model used may be, for example, Zero-1-to-3 (Zero123), which is a new view image generation model based on a large-scale diffusion model.

[0068] In a specific embodiment of the present application, for a missing view angle in the set of missing view angles, an image with a similar view angle is obtained from the original input images as a reference image, and a target image of the missing view angle is generated by using a new view image generation model, to obtain a set of missing view angle images containing each target image, which can specifically include:

[0069] For each missing view angle in the set of missing view angles, an image with a similar view angle is obtained from the original input images as a reference image, and a target image of the missing view angle is generated by using a new view image generation model, to obtain a set of missing view angle images containing each target image, which can specifically include:

[0070] The relative pose between the missing view angle and the reference image is obtained.

[0071] The relative pose and the reference image are input into the new view image generation model to obtain a target image of the missing view angle generated by the new view image generation model.

[0072] After obtaining the target images of each missing view angle in the set of missing view angles, a set of missing view angle images containing each target image is obtained.

[0073] For any missing view in the missing view set, when the embodiment selects an image with a similar view angle from the original input images, the image with the smallest angle with the missing view is selected as the reference image. Then, the relative pose between the missing view and the reference image can be obtained. For the missing view, its pose includes its position parameter and orientation parameter, i.e., the position parameter O i and the orientation parameter P i in the above example. For the reference image, the reference image corresponding to the missing view is denoted as I ref , the pose of the reference image I ref is denoted as (P ref , O ref ), and the relative pose between the two can be expressed as P rel =P i P ref T , O rel =P ref T (O i -O ref ), where T denotes the transpose matrix.

[0074] After traversing each missing view in the missing view set, the target image of each missing view can be obtained using the new view image generation model, and the set of these target images is called the missing view image set, denoted as I miss .

[0075] Step S105: based on the original input images and the missing view image set, performing three-dimensional reconstruction on the object to obtain a three-dimensional reconstruction result of the object.

[0076] The original input images are denoted as I ori , and the original input images I ori and the missing view image set I miss are input into a three-dimensional reconstruction model. Then, the three-dimensional reconstruction model can be used to perform three-dimensional reconstruction on the object to obtain a three-dimensional reconstruction result of the object.

[0077] For example, a three-dimensional reconstruction model based on deep learning can be used to perform three-dimensional reconstruction. By using two-dimensional images and corresponding camera pose information collected from different view angles, the model can learn an implicit or explicit three-dimensional representation of the scene, and can render a two-dimensional image from a given camera pose to achieve high-fidelity three-dimensional modeling and new view synthesis. For example, NeRF or 3D Gaussian Splatting can be selected.

[0078] In one specific implementation, the original input images are denoted as I ori and the missing view image set is denoted as I missThe input is input to the three-dimensional reconstruction model, and the camera intrinsic parameter and camera extrinsic parameter (camera pose) corresponding to each image are input, so that the three-dimensional reconstruction result of the object can be obtained. For example, NeRF specifically outputs a three-dimensional radiation field representation of a scene, and 3DGS specifically outputs a three-dimensional Gaussian distribution set.

[0079] In one specific embodiment of the application, the following can also be included:

[0080] According to the N camera sampling view angles, the obtained three-dimensional reconstruction result of the object is rendered to obtain a rendered image under the corresponding camera sampling view angle;

[0081] The quality of each rendered image is evaluated to obtain the quality evaluation result of each rendered image, and it is determined whether the rendered images that do not meet the quality requirements exceed a preset second quantity threshold;

[0082] If yes, the process is ended;

[0083] If no, a to-be-optimized image set composed of the rendered images that do not meet the quality requirements is established based on the rendered images that do not meet the quality requirements;

[0084] The rendered images in the to-be-optimized image set are quality-optimized to obtain an optimized image set;

[0085] Based on the original input images, the missing view angle image set, and the optimized image set obtained in this round of iteration, the object is re-reconstructed in three dimensions to obtain a three-dimensional reconstruction result of the object, and the operation of rendering the obtained three-dimensional reconstruction result of the object according to the N camera sampling view angles to obtain a rendered image under the corresponding camera sampling view angle is returned.

[0086] For reference Figure 2 In the above embodiments, the pre-optimization process is realized, that is, the missing view angles are completed by detecting the missing view angles and supplementing the missing view angle image set, so as to improve the spatial coverage of the input images and further improve the completeness of the object reconstruction. In this embodiment, the post-optimization process is further provided, which is considered that although the above embodiment jointly uses the original input images I ori and the missing view angle image set I miss as input images to ensure a high coverage, due to the fact that the quality of the original input images is uneven and the complexity of different parts of the object itself is different, the reconstructed 3D content may not necessarily have extremely high quality at any view angle. Therefore, the iterative optimization strategy based on the rendering quality feedback is provided in this embodiment of the application.

[0087] In this implementation, the three-dimensional reconstruction result of the obtained object is rendered in the surround view, and the reconstruction quality of the 3D content is evaluated. When rendering in the surround view, the obtained 3D content is rendered according to the N camera sampling view angles set in advance, so as to obtain 2D rendering images under the corresponding camera sampling view angles. It can be understood that there are N 2D rendering images in total.

[0088] For each rendering image, quality evaluation is required to obtain the respective quality evaluation result of each rendering image. The specific implementation of quality evaluation can be various, for example, the quality evaluation can be performed according to the “technical defects” of the image, such as noise, compression artifacts, blur, exposure problems, etc., to obtain the quality evaluation result. For another example, the quality evaluation can be performed based on the aesthetic score of the image.

[0089] In a specific implementation of the present application, the quality evaluation of each rendering image to obtain the respective quality evaluation result of each rendering image can specifically include:

[0090] For any rendering image, the image technical score and the image aesthetic score of the rendering image are determined and weighted and superimposed, and the value obtained after the weighted superimposition is taken as the quality evaluation result of the rendering image.

[0091] This implementation considers that if only the technical score is used to evaluate the image quality, the image that is “technically clean but may have poor composition” may be selected, and if only the aesthetic score is used, the image that is “good-looking but may have defects / transformations” may be selected. Therefore, in this implementation, the two are weighted so that both images with low quality and geometric / noise problems and images that are unacceptable in vision, such as color anomalies and texture splitting, can be captured. That is, in this implementation, the image quality can be more accurately measured by the image technical score and the image aesthetic score.

[0092] Specifically, when performing the image technical score, the DOVER evaluation model can be used, which can effectively evaluate the “technical defects” of the image, such as noise, compression artifacts, blur, exposure problems, etc., to obtain the image technical score of the rendering image. When performing the image aesthetic score, for example, the Aesthetic Score Predictor model can be specifically used, which is based on CLIP embedding and MLP trained on AVA data to obtain the average aesthetic score of the picture, which meets the human preference index. Of course, in other specific implementations, other types of models can be used to implement the image technical score and the image aesthetic score of the rendering image.

[0093] When the weighted superposition of the image technology score and the image aesthetic score is performed, the weight value can be set as required, for example, the weight value of the image technology score is 0.7 by default in actual application, and the weight value of the image aesthetic score is 0.3 by default, so that more emphasis is placed on the technical quality.

[0094] After the quality assessment of each rendered image is performed and the respective quality assessment results of each rendered image are obtained, the number of rendered images that do not meet the quality requirement can be obtained. For example, in one case, the value obtained after the weighted superposition of the image technology score and the image aesthetic score of each rendered image is the quality assessment result of the rendered image, and if the value obtained after the weighted superposition is lower than the preset score threshold, it can be determined that the rendered image is a rendered image that does not meet the quality requirement, otherwise, it is a rendered image that meets the quality requirement.

[0095] If the number of rendered images that do not meet the quality requirement does not exceed the preset second number threshold, it indicates that the rendering quality of the 3D content is very high, and therefore the process can be directly ended, otherwise, iteration optimization needs to be continued. The value of the second number threshold can be set as required, and the lower the setting is, the higher the requirement is. For example, in one case, the second number threshold is set to the minimum value 0, which means that only when each rendered image meets the quality requirement, the iteration is ended, otherwise, the iteration needs to be continued.

[0096] If the number of rendered images that do not meet the quality requirement exceeds the preset second number threshold, at this time, a to-be-optimized image set composed of rendered images that do not meet the quality requirement needs to be established based on the rendered images that do not meet the quality requirement. For example, in one simple way, all rendered images that do not meet the quality requirement are directly used as the to-be-optimized image set established.

[0097] In one specific embodiment of the present application, the to-be-optimized image set composed of rendered images that do not meet the quality requirement can be specifically established based on the rendered images that do not meet the quality requirement, which can include:

[0098] determining whether the number of rendered images that do not meet the quality requirement is lower than a preset third number threshold;

[0099] if yes, establishing a to-be-optimized image set composed of all rendered images that do not meet the quality requirement;

[0100] if no, selecting M rendered images from all rendered images that do not meet the quality requirement based on the farthest point sampling algorithm, and establishing a to-be-optimized image set composed of the M rendered images.

[0101] The implementation considers that if the number of rendered images that do not meet the quality requirement is not particularly large, that is, the number of rendered images that do not meet the quality requirement is lower than the third quantity threshold, all rendered images that do not meet the quality requirement can be directly taken as the to-be-optimized image set. However, if there are many rendered images that do not meet the quality requirement, only part of them can be selected to construct the to-be-optimized image set. This operation can reduce the amount of calculation, and subsequent three-dimensional reconstruction of the object needs to be performed based on the original input image, the missing view image set, and the optimized image set obtained in this round of iteration. If the number of images in the optimized image set is too large, the distortion degree of the reconstructed 3D content may be increased. Therefore, only part of them is selected to construct the to-be-optimized image set at this time.

[0102] Of course, when selecting, the rendered images that do not meet the quality requirement can be selected relatively uniformly. Therefore, M rendered images can be selected relatively uniformly from all rendered images that do not meet the quality requirement by using the farthest point sampling algorithm, and a to-be-optimized image set composed of the M rendered images is established. M is a preset quantity value, which can be set to a relatively appropriate value according to actual needs.

[0103] After obtaining the to-be-optimized image set, quality optimization needs to be performed on each rendered image in the to-be-optimized image set to obtain an optimized image set. The quality optimization can usually be implemented by using a multi-modal generation model. For example, in one specific implementation, the quality optimization of the rendered images in the to-be-optimized image set to obtain the optimized image set can specifically include the following steps.

[0104] For any rendered image in the to-be-optimized image set, the rendered image and the guide text are input into a preset multi-modal generation model to obtain an optimized image for the rendered image output by the multi-modal generation model.

[0105] The obtained each optimized image is taken as the obtained optimized image set.

[0106] In this implementation, the rendered image and the guide text are input into a preset multi-modal generation model, for example, a multi-modal large model such as GPT-4o / GPT-5, Gemini 2.5 Flash Image, and the like. Through the guidance of the guide text, the multi-modal generation model can optimize the input rendered image, thereby outputting a corresponding optimized image. Each optimized image constitutes the optimized image set.

[0107] In an embodiment, the guidance text can be specifically text for prompting the multi-modal generation model to perform image optimization, and the guidance text includes one or more of the following expressions: enhancing clarity, enhancing details, eliminating noise, eliminating distortion, adjusting light and contrast to present a natural balance effect, and optimizing color saturation, so as to guarantee a better guidance effect.

[0108] For example, in one specific embodiment, a reference guidance text is specifically set as: "Please optimize this low-quality image, maintain the integrity of the original scene, objects, and characters unchanged, upgrade to 4K resolution, enhance clarity and details, eliminate noise and compression distortion, adjust light and contrast to present a natural balance effect, and optimize color saturation to obtain a vivid but realistic visual effect, without changing the perspective or adding new elements." Such a guidance text effectively realizes the optimization of the image without changing the perspective or adding new elements.

[0109] After the optimization in this round, the optimized image set is obtained, together with the original input image I ori and the missing perspective image set I miss , the object can be re-reconstructed in three dimensions, so as to re-obtain the three-dimensional reconstruction result of the object. At this time, the operation of rendering the obtained three-dimensional reconstruction result of the object according to the N camera sampling perspectives to obtain the rendered image under the corresponding camera sampling perspective can be performed again, so as to perform the next iteration, until whether the rendered image that does not meet the quality requirement in a certain iteration round does not exceed the preset second quantity threshold, so that the iteration can be ended, and at this time, a very high-quality three-dimensional reconstruction result can be obtained.

[0110] In addition, it should be further pointed out that in actual application, it can be necessary to go through one or more iterations. After each iteration, the original input image I ori , the missing perspective image set I miss , and the optimized image set in this round are selected to perform three-dimensional reconstruction of the object, that is, the optimized image set obtained in the previous round does not need to be used in the current round. Such a design is considering that with the increase of the iteration round, if the optimized image set obtained in each iteration round is used to participate in the three-dimensional reconstruction of the object in the current round, the proportion of the original input image I ori and the missing perspective image set I miss will be reduced, which can increase the distortion degree of the three-dimensional reconstruction result, and is not conducive to guaranteeing the quality. Therefore, the original input image I ori , the missing perspective image set I miss , and the optimized image set obtained in the current round are used for three-dimensional reconstruction of the object each time.

[0111] The application scheme considers that incomplete view coverage will affect the quality of three-dimensional reconstruction of the object. To this end, the view coverage degree is optimized. Specifically, first, N camera sampling views are established in the surrounding space of the object bounding sphere, and then based on the original input image, each missing view in the N camera sampling views that is not covered by the original input image is determined. It can be understood that if the N camera sampling views are all covered by the original input image, it means that the original input image realizes complete view coverage, and high-quality three-dimensional reconstruction of the object can be directly realized. If there is a missing view, the application scheme will establish a missing view set composed of at least one missing view based on each missing view. Then, for the missing view in the missing view set, the image of the view is supplemented, specifically, an image with a similar view is obtained from the original input image as a reference image, and a new view image generation model is used to generate a target image of the missing view, so that a missing view image set containing each target image is obtained. The missing view image set is a view supplement to the original input image, so based on the original input image and the missing view image set, the view coverage degree can be effectively improved, and accordingly the three-dimensional reconstruction of the object can be performed to obtain a high-quality three-dimensional reconstruction result of the object.

[0112] In summary, the application scheme can effectively realize three-dimensional reconstruction of the object and improve the reconstruction quality.

[0113] Corresponding to the above method and system embodiments, the application embodiments also provide an electronic device, a computer readable storage medium and a computer program product, which can be mutually corresponding with the above.

[0114] Referring to Figure 3 As shown in the figure, the electronic device can include:

[0115] The memory 301 is configured to store a computer program.

[0116] The processor 302 is configured to execute the computer program to implement the steps of the three-dimensional reconstruction method of the object in any of the above embodiments.

[0117] The computer program product includes computer programs / instructions that are executed by the processor to implement the steps of the three-dimensional reconstruction method of the object in any of the above embodiments.

[0118] Referring to Figure 4The computer readable storage medium 40 stores a computer program 41, which, when executed by the processor, implements the steps of the object three-dimensional reconstruction method in any of the above embodiments. The computer readable storage medium 40 mentioned herein includes a RAM (Random Access Memory), an internal memory, a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a register, a hard disk, a removable disk, or any other form of storage medium known in the art.

[0119] Those skilled in the art will further appreciate that the individual steps of the example methods described in connection with the embodiments disclosed herein can be embodied in electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various examples have been described herein in terms of their functionality, which has been described generally and symbolically in flow charts. There has been described, without loss of generality, various examples in terms of their functionality, which has been described generally and symbolically in flow charts. It is to be understood that each of the functional steps illustrated in the flow charts can be embodied in computer software, hardware, or a combination of both. The software can be stored on a computer readable storage medium, which can be any available storage media that can be accessed by a computer. By way of example, and not limitation, such computer readable storage media can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other storage medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, then the coaxial cable, fiber optic cable, twisted pair, DSL, or any other medium used to carry or store the desired program code is included in the definition of medium. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), and Blu-ray® disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer readable media.

[0120] In the specification, specific examples have been described in detail in order to explain the principles and implementation of the application. The above examples are only used to help understand the method and core idea of the application. It should be pointed out that, for those skilled in the art, without departing from the principles of the application, some improvements and modifications can be made to the application, and these improvements and modifications also fall within the protection scope of the application.

Claims

1. A method for three-dimensional reconstruction of an object, characterized in that, include: In the surrounding space of the sphere, N camera sampling viewpoints are established; N is a positive integer; Based on the original input image, identify the missing viewpoints among the N camera sampling viewpoints that are not covered by the original input image; Establish a set of missing perspectives consisting of at least one of the aforementioned missing perspectives; For the missing viewpoints in the missing viewpoint set, images with similar viewpoints are obtained from the original input image as reference images, and a new viewpoint image generation model is used to generate target images of the missing viewpoints, resulting in a missing viewpoint image set containing each target image. Based on the original input image and the set of missing viewpoint images, the object is reconstructed in three dimensions to obtain the three-dimensional reconstruction result of the object; The three-dimensional reconstruction results of the object are rendered according to N camera sampling viewpoints to obtain the rendered image under the corresponding camera sampling viewpoints; Each rendered image is evaluated for quality to obtain its own quality evaluation result, and it is determined whether the number of rendered images that do not meet the quality requirements does not exceed a preset second quantity threshold. If so, the process ends; If not, then based on the rendered images that do not meet the quality requirements, establish a set of images to be optimized, consisting of the rendered images that do not meet the quality requirements; The rendered images in the image set to be optimized are subjected to quality optimization to obtain an optimized image set; Based on the original input image, the set of missing viewpoint images, and the optimized image set obtained in this iteration, the object is reconstructed in three dimensions to obtain the three-dimensional reconstruction result of the object. Then, the operation of rendering the obtained three-dimensional reconstruction result of the object according to N camera sampling viewpoints is returned to obtain the rendered image under the corresponding camera sampling viewpoint is performed. Within the space surrounding the sphere, N camera sampling viewpoints are established, including: N grid points are uniformly set on the sphere surrounding the object, and the directions formed by the N grid points and the center of the object are used as the orientation parameters of the N camera sampling viewpoints respectively. For any camera sampling viewpoint, the position parameters of the camera sampling viewpoint are obtained based on the camera's observation radius, the coordinates of the object's center, and the orientation parameters of the camera sampling viewpoint.

2. The method for three-dimensional reconstruction of an object according to claim 1, characterized in that, For each camera sampling viewpoint, based on the camera's observation radius, the coordinates of the object's center, and the orientation parameter of the camera sampling viewpoint, the position parameters of the camera sampling viewpoint are obtained, including: For any camera sampling viewpoint, according to O i =C+Rc×P i The position parameters of the camera sampling angle are obtained through the calculation method. Among them, O i This represents the position parameters of the sampling viewpoint of the i-th camera, C represents the coordinates of the center of the object, Rc represents the observation radius of the camera, and P... i This represents the orientation parameter of the sampling viewpoint of the i-th camera.

3. The method for three-dimensional reconstruction of an object according to claim 1, characterized in that, Based on the original input image, identify the missing viewpoints among the N camera sampling viewpoints that are not covered by the original input image, including: For any one of the N camera sampling views, determine the number of images in the original input image that overlap with the camera sampling view. Determine whether the quantity is lower than a preset overlap threshold; If so, the camera sampling viewpoint is determined to be a missing viewpoint not covered by the original input image; If not, then it is determined that the camera sampling view is not a missing view not covered by the original input image.

4. The method for three-dimensional reconstruction of an object according to claim 1, characterized in that, Establish a set of missing perspectives consisting of at least one of the aforementioned missing perspectives, including: Determine whether the total number of missing viewpoints is lower than a preset first threshold. If so, then establish a set of missing perspectives consisting of all missing perspectives; If not, then based on all the missing viewpoints, select K1×F missing viewpoints from them using the farthest point sampling algorithm, and establish a set of missing viewpoints consisting of K1×F missing viewpoints. Where F represents the total number of missing viewpoints, and K1 is a preset coefficient value less than or equal to 1.

5. The method for three-dimensional reconstruction of an object according to claim 1, characterized in that, For each missing viewpoint in the missing viewpoint set, an image with a similar viewpoint is obtained from the original input image as a reference image, and a new viewpoint image generation model is used to generate the target image for the missing viewpoint, resulting in a missing viewpoint image set containing each target image, including: For a missing viewpoint in the set of missing viewpoints, the image with the smallest angle between the missing viewpoint and the original input image is obtained as the reference image of the missing viewpoint. Obtain the relative pose between the missing viewpoint and the reference image; The relative pose and the reference image are input into the new perspective image generation model to obtain the target image of the missing perspective generated by the new perspective image generation model; After obtaining the target images of each missing viewpoint in the missing viewpoint set, a missing viewpoint image set containing each target image is obtained.

6. The method for three-dimensional reconstruction of an object according to claim 1, characterized in that, The quality of each rendered image is evaluated to obtain the quality evaluation result for each rendered image, including: For any given rendered image, determine the image technical score and image aesthetic score of the rendered image and perform a weighted superposition, and use the value obtained after weighted superposition as the quality evaluation result of the rendered image.

7. The method for three-dimensional reconstruction of an object according to claim 1, characterized in that, The rendered images in the set of images to be optimized are subjected to quality optimization to obtain an optimized image set, including: For any rendered image in the set of images to be optimized, the rendered image and the guiding text are input into a preset multimodal generation model to obtain an optimized image for the rendered image output by the multimodal generation model. The obtained optimized images are used as the set of optimized images; The guiding text is text used to prompt the multimodal generation model to perform image optimization, and the guiding text includes one or more of the following: enhancing sharpness, enhancing details, eliminating noise, eliminating distortion, adjusting light and contrast to present a natural balance effect, and optimizing color saturation.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the object three-dimensional reconstruction method as described in any one of claims 1 to 7 when executing the computer program.

Images