Object three-dimensional reconstruction method and electronic equipment

Abstract

The invention discloses an object three-dimensional reconstruction method and electronic equipment, and is applied to the technical field of computer vision, and the method comprises the steps: building N camera sampling visual angles in a surrounding space of an object surrounding ball; based on the original input image, determining each missing view angle which is not covered by the original input image in the N camera sampling view angles; establishing a missing view angle set composed of at least one missing view angle; for a missing visual angle in the missing visual angle set, obtaining an image with a similar visual angle from the original input image as a reference image, generating a target image of the missing visual angle by using a new visual angle image generation model, and obtaining a missing visual angle image set containing each target image; and performing three-dimensional reconstruction on the object based on the original input image and the missing view angle image set to obtain a three-dimensional reconstruction result of the object. By applying the scheme of the invention, the three-dimensional reconstruction of the object can be effectively realized, and the reconstruction quality can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a three-dimensional reconstruction method of an object and an electronic device. BACKGROUND

[0002] The current 3D reconstruction technology needs dense multi-view input. For example, each sample in a common Nerf-Synthetic dataset contains hundreds of images, and the number of images contained in each sample in the MVImgNet dataset is about 30. Through the dense multi-view input, a high-quality 3D reconstruction of an object can be achieved.

[0003] However, in actual applications, the input images often have the problem of insufficient view coverage, especially in atypical views such as the top and bottom of the object. The lack of image input will cause the 3D reconstruction result to be geometrically distorted, have holes or lack of texture, and affect the reconstruction quality.

[0004] To sum up, how to effectively realize the three-dimensional reconstruction of an object and effectively improve the reconstruction quality is a technical problem that needs to be solved by the technical personnel in the field at present. SUMMARY

[0005] The present application provides a three-dimensional reconstruction method of an object and an electronic device to effectively realize the three-dimensional reconstruction of an object and effectively improve the reconstruction quality.

[0006] To solve the above technical problems, the present application provides the following technical solutions: The present application provides a three-dimensional reconstruction method of an object, comprising: establishing N camera sampling view angles in the surrounding space of the object bounding sphere; N is a positive integer; determining each missing view angle in the N camera sampling view angles that is not covered by the original input image based on the original input image; establishing a missing view angle set composed of at least one missing view angle; for the missing view angle in the missing view angle set, acquiring an image with a similar view angle from the original input image as a reference image, and generating a target image of the missing view angle by using a new view image generation model, to obtain a missing view image set containing each target image; based on the original input image and the missing view image set, performing three-dimensional reconstruction on the object to obtain a three-dimensional reconstruction result of the object.

[0007] The present application also provides an electronic device, comprising: a memory for storing a computer program; a processor for executing the computer program to realize the steps of the three-dimensional reconstruction method of the object as described above.

[0008] The application scheme considers that incomplete view coverage will affect the quality of three-dimensional reconstruction of the object, and therefore optimizes the view coverage. Specifically, first, N camera sampling views are established in the surrounding space of the object bounding sphere, and then based on the original input image, each missing view in the N camera sampling views that is not covered by the original input image is determined. It can be understood that if the N camera sampling views are all covered by the original input image, it means that the original input image achieves complete view coverage, and high-quality three-dimensional reconstruction of the object can be directly achieved. If there are missing views, the application scheme will establish a missing view set composed of at least one missing view based on each missing view. Then, for the missing views in the missing view set, the image of the view is supplemented, specifically, an image with a similar view is obtained from the original input image as a reference image, and a new view image generation model is used to generate the target image of the missing view, so that the missing view image set containing each target image is obtained. The missing view image set is a view supplement to the original input image, and therefore based on the original input image and the missing view image set, the view coverage can be effectively improved, and the three-dimensional reconstruction of the object can be performed accordingly to obtain a high-quality three-dimensional reconstruction result of the object.

[0009] In summary, the application scheme can effectively achieve three-dimensional reconstruction of the object and improve the reconstruction quality. BRIEF DESCRIPTION OF DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the application, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0011] Figure 1 The implementation flowchart of the object three-dimensional reconstruction method provided by the specific embodiment of the application; Figure 2 The principle diagram of the pre-optimization and post-optimization in the specific embodiment of the application; Figure 3 The structural diagram of the electronic device provided by the specific embodiment of the application; Figure 4 The structural diagram of the computer readable storage medium of the application. DETAILED DESCRIPTION

[0012] With reference to the drawings and specific embodiments, the technical solutions in the embodiments of the present application will be described in detail below. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, any person skilled in the art can obtain all other embodiments without creative work, which shall fall within the protection scope of the present application.

[0013] It should be noted that, in the description of the present application, the terms "comprise", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment. The terms "first", "second" and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0014] In order for those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments. Please refer to Figure 1 , Figure 1 The embodiment flowchart of the object three-dimensional reconstruction method provided by the present application can include the following steps: Step S101: In the surrounding space of the object enclosing sphere, N camera sampling view angles are established.

[0015] The present application can perform three-dimensional reconstruction of the object, which can be any object that can be photographed, and does not affect the implementation of the present application. The object enclosing sphere refers to a sphere that encloses the object. The diameter of the object enclosing sphere can be exactly equal to the longest axis of the object, so that the object enclosing sphere is tangent to the object, or it can be slightly larger than the longest axis of the object.

[0016] Sampling in the surrounding space of the object enclosing sphere refers to selecting sampling points in the surrounding area of the object enclosing sphere, so as to obtain a set of surrounding space view angles, that is, N camera sampling view angles. It can be understood that, in order to ensure the comprehensiveness of the view angles covered by the present application, N camera sampling view angles can be uniformly selected in the surrounding space of the object enclosing sphere. And N is a positive integer, the larger the value of N, the higher the density of the camera sampling view angles, the higher the accuracy of the coverage of the present application, but the calculation amount will also increase accordingly. In practical applications, N can be valued between 100 and 300.

[0017] In one specific embodiment of the present application, step S101 can specifically include: N grid points are evenly arranged on the object surrounding sphere, and the direction formed by the N grid points and the object center is respectively taken as the orientation parameter of each camera sampling view angle; For any camera sampling view angle, the position parameter of the camera sampling view angle is obtained based on the observation radius of the camera, the coordinates of the object center, and the orientation parameter of the camera sampling view angle.

[0018] This embodiment considers that the camera sampling view angle can be represented as the orientation parameter and the position parameter, so that the orientation parameter of each of the N camera sampling view angles can be determined respectively, and then the corresponding position parameter is obtained in combination with the observation radius of the camera.

[0019] Specifically, N grid points can be evenly arranged on the object surrounding sphere, for example, N Fibonacci grid points are sampled on the object surrounding sphere as the N grid points evenly arranged on the object surrounding sphere. In order to facilitate calculation, the object surrounding sphere is regarded as a unit sphere, that is, the radius R obj of the object surrounding sphere can be regarded as 1. In actual application, the N grid points are arranged by using the Fibonacci grid sampling algorithm, which can ensure that the N grid points are evenly distributed on the sphere surface and ensure the accuracy of the calculation coverage rate.

[0020] After the N grid points are evenly arranged on the object surrounding sphere, the direction formed by each of the N grid points and the object center C is respectively the orientation parameter of each of the N camera sampling view angles. Taking the radius R obj of the object surrounding sphere as 1 and the coordinates of the object center C as (0, 0, 0) as an example, for example, the north pole of the object surrounding sphere is one of the grid points, and the coordinates of the grid point are (0, 0, 1). The direction formed by the grid point and the object center C is the orientation parameter of the corresponding camera sampling view angle.

[0021] After obtaining the orientation parameter of each of the N camera sampling view angles, the position parameter of each of the N camera sampling view angles can be obtained in combination with the observation radius of the camera and the coordinates of the object center. For example, in one specific embodiment, for any camera sampling view angle, the position parameter of the camera sampling view angle can be obtained according to the calculation manner of O i =C+Rc×P i . Wherein, O i represents the position parameter of the i-th camera sampling view angle, C represents the coordinates of the object center, Rc represents the observation radius of the camera, and P i represents the orientation parameter of the i-th camera sampling view angle. It can be understood that the value of i can be from 1 to N.

[0022] Still taking the grid point (0, 0, 1) as an example, since the coordinates of the object center C are (0, 0, 0), the orientation parameter P of the corresponding camera sampling view in this example is (0, 0, 1), for example, Rc is 2, and the position parameter of the camera sampling view corresponding to the grid point in this example is (0, 0, 2), that is, at the position (0, 0, 2), one of the N camera positions is represented, and the direction from the camera position to the object center is the orientation of the camera sampling view. i Still taking the grid point (0, 0, 1) as an example, since the coordinates of the object center C are (0, 0, 0), the orientation parameter P of the corresponding camera sampling view in this example is (0, 0, 1), for example, Rc is 2, and the position parameter of the camera sampling view corresponding to the grid point in this example is (0, 0, 2), that is, at the position (0, 0, 2), one of the N camera positions is represented, and the direction from the camera position to the object center is the orientation of the camera sampling view.

[0023] In addition, it should be noted that the value of the observation radius Rc of the camera can be set according to actual needs, but generally needs to be greater than the radius R of the object bounding sphere obj , so as to ensure that the object is completely in the view cone, that is, Rc can be set as Rc = a x R obj , where the coefficient a needs to be greater than 1, and is usually set to a value between 2 and 3.

[0024] Step S102: determining each missing view in the N camera sampling views that is not covered by the original input image based on the original input image.

[0025] The original input image can include one or more images, and generally needs to include multiple images, that is, the original input image is a set including multiple images.

[0026] After obtaining the N camera sampling views, it is necessary to determine each view in the N camera sampling views that is not covered by the original input image as a missing view based on the original input image. Specifically, for any one camera sampling view, the degree of overlap between the camera sampling view and the original input image can be measured to determine whether the camera sampling view belongs to the missing view. Of course, the specific implementation of measuring the degree of overlap between the views can be various, for example, the number of images in the original input image that overlap with the camera sampling view can be used as a measurement standard.

[0027] For example, in a specific embodiment of the present application, determining each missing view in the N camera sampling views that is not covered by the original input image based on the original input image can specifically include: For any one camera sampling view in the N camera sampling views, determining the number of images in the original input image that overlap with the camera sampling view as a view; determining whether the number is lower than a preset overlap threshold; If yes, it is determined that the camera sampling view is a missing view that is not covered by the original input image; If no, it is determined that the camera sampling view is not a missing view that is not covered by the original input image.

[0028] In this implementation, for any one of the N camera sampling perspectives, the number of images in the original input image that overlap with that camera sampling perspective is determined, which is relatively simple and convenient to implement. After obtaining the number of overlapping images, if the number is lower than a preset overlap threshold, the camera sampling perspective can be determined to be a missing perspective; otherwise, it is not a missing perspective.

[0029] Furthermore, in one specific embodiment of the present invention, for any one of the N camera sampling viewpoints, determining the number of images in the original input image that overlap with the camera sampling viewpoint can specifically include: Determine the camera viewpoint for each image in the original input image; For any one of the N camera sampling perspectives, select each image sequentially from the original input image. If the angle between the camera sampling perspective and the selected image from the original input image is lower than a preset value, it is determined that the camera sampling perspective overlaps with the perspective of the selected image from the original input image; otherwise, they do not overlap. After selecting each image sequentially from the original input image and completing the judgment, determine the number of images in the original input image that overlap with the camera's sampling viewpoint.

[0030] In this implementation, for any one of the N camera sampling viewpoints, it is compared sequentially with each image in the original input image to obtain the angle between the two viewpoints. If the angle is lower than a preset value, the viewpoints overlap; otherwise, they do not overlap. Therefore, by comparing any one camera sampling viewpoint sequentially with each image in the original input image, the number of images overlapping with that viewpoint can be determined.

[0031] For example, in one specific embodiment, the above process can be represented as follows: P i This represents the camera sampling view of the i-th camera out of N camera sampling views, that is, the orientation parameter of the i-th camera sampling view. j This represents the camera viewpoint of the j-th image out of the D original input images. is the angle between the i-th camera sampling view and the camera view of the j-th image, if it is lower than a preset value 0.5*FOV, it can be determined that the two are overlapped, otherwise it is not overlapped. Overlap(i, j) is the judgment result of the angle overlap between the i-th camera sampling view and the j-th image. In this example, the preset value is specifically set to 0.5*FOV, FOV is a preset camera field of view, and in other cases, the value of the preset value can be adjusted according to actual needs.

[0032] In the above example, there are D images in the original input image, so it can be understood that for any one of the N camera sampling views, the number of images in the original input image that are overlapped with the camera sampling view ranges between [0, D].

[0033] In addition, it should be noted that in the above example, the camera view of each image in the original input image needs to be determined, and in actual implementation, it can usually be determined based on a related pose recognition model, which is relatively mature and convenient to implement, and the present application will not be expanded.

[0034] Step S103: establishing a missing view angle set composed of at least one missing view angle.

[0035] Through the above operation, the missing view angles in the N camera sampling views that are not covered by the original input image can be determined, and after obtaining these missing view angles, a missing view angle set composed of at least one missing view angle can be established based on the missing view angles, that is, one or more missing view angles in the missing view angles can be selected as the missing view angle set, for example, a simple implementation is to directly select all the missing view angles as the constructed missing view angle set.

[0036] In a specific embodiment of the present application, step S103 can specifically include: judging whether the number of all missing view angles is lower than a preset first number threshold; if yes, establishing a missing view angle set composed of all missing view angles; if no, selecting K1*F missing view angles from all missing view angles based on a farthest point sampling algorithm, and establishing a missing view angle set composed of K1*F missing view angles; wherein F represents the number of all missing view angles, and K1 is a preset coefficient value less than or equal to 1.

[0037] The implementation considers that if the number of missing view angles is not particularly large, that is, the number of all missing view angles is lower than the first number threshold, all missing view angles can be directly taken as the constructed missing view angle set. However, if the number of missing view angles is large, part of the missing view angles can be selected to construct the missing view angle set. Such an operation considers that in actual application, in order to effectively guarantee the quality of three-dimensional reconstruction, the value of N is usually set to be large, so that the density of the sampling view angles is high, which is beneficial to improve the coverage rate. On this basis, the number of missing view angles is likely to be large, and the density of these missing view angles is also high. Therefore, if part of the missing view angles is appropriately selected to construct the missing view angle set, the calculation amount can be effectively reduced, and the coverage rate will not be greatly affected, so that the quality of the three-dimensional reconstruction result of the object can be effectively guaranteed. Therefore, in the implementation, when the number of missing view angles is large, part of the missing view angles is selected to construct the missing view angle set.

[0038] In addition, when selecting the missing view angles, the missing view angles can be uniformly selected. Therefore, in the implementation, K1xF missing view angles are uniformly selected from all missing view angles based on the farthest point sampling algorithm, so as to establish a missing view angle set composed of K1xF missing view angles, K1 being a preset coefficient value less than or equal to 1, for example, 0.2 in one occasion. Through the farthest point sampling algorithm, the selected missing view angles can be uniformly distributed. Therefore, the missing view angle set composed of the missing view angles, together with the original input image, can comprehensively cover all directions of the object, so as to guarantee the quality of subsequent three-dimensional reconstruction.

[0039] In the above embodiment, the specific value of the preset first number threshold can be set according to actual needs, for example, can be set as a fixed value. However, it can be understood that the value of the first number threshold is 0 at the minimum and the number of images D in the original input image at the maximum. Further, in one specific implementation of the application, the coverage rate evaluation result of the original input image can be obtained. If the coverage rate evaluation result of the original input image is a centralized distribution of the original input image, the first number threshold can be set to 0. If the coverage rate evaluation result of the original input image is a decentralized distribution of the original input image, the first number threshold can be set to a value greater than 0, for example, usually set to a small integer. In the implementation, by selecting an appropriate value of the first number threshold, the number of missing view angles can be effectively adjusted, which is beneficial to make the number of missing view angles a moderate and appropriate value.

[0040] Specifically, if the original input images are centrally distributed, it indicates that there are a large number of uncovered view angles, and thus the number of missing view angles is prone to be large. In this case, the first number threshold is set to the minimum value 0, which is beneficial to appropriately reduce the number of missing view angles. Conversely, if the original input images are dispersedly distributed, it indicates that there are not many uncovered view angles, and thus the number of missing view angles is prone to be insufficient. In this case, the value of the first number threshold can be appropriately increased, which is beneficial to appropriately increase the number of missing view angles.

[0041] Step S104: For a missing view angle in the set of missing view angles, an image with a similar view angle is obtained from the original input images as a reference image, and a target image of the missing view angle is generated by using a new view image generation model, to obtain a set of missing view angle images containing each target image.

[0042] For each missing view angle in the set of missing view angles, a new view image generation model needs to be used to generate an image under the view angle, which is referred to as a target image of the missing view angle. In this process, an image with a similar view angle in the original input images needs to be used as a reference image. The new view image generation model used may be, for example, Zero-1-to-3 (Zero123), which is a new view image generation model based on a large-scale diffusion model.

[0043] In a specific embodiment of the present application, for a missing view angle in the set of missing view angles, an image with a similar view angle is obtained from the original input images as a reference image, and a target image of the missing view angle is generated by using a new view image generation model, to obtain a set of missing view angle images containing each target image, which can specifically include: For each missing view angle in the set of missing view angles, an image with a similar view angle is obtained from the original input images as a reference image, and a target image of the missing view angle is generated by using a new view image generation model, to obtain a set of missing view angle images containing each target image, which can specifically include: The relative pose between the missing view angle and the reference image is obtained; The relative pose and the reference image are input into the new view image generation model to obtain a target image of the missing view angle generated by the new view image generation model; After obtaining the target image of each missing view angle in the set of missing view angles, a set of missing view angle images containing each target image is obtained.

[0044] For any one of the missing view angles in the set of missing view angles, this embodiment specifically selects an image with a minimum angle with the view angle as a reference image when selecting an image with a similar view angle from the original input images. Then, the relative pose between the missing view angle and the reference image can be obtained. For the missing view angle, the pose includes a position parameter and an orientation parameter, i.e., the position parameter O i and the orientation parameter P iFor the reference image, the reference image corresponding to the missing view angle is denoted as I ref , the pose of the reference image I ref is denoted as (P ref , O ref ), and the relative pose of the two can be expressed as P rel =P i P ref T , O rel =P ref T (O i -O ref ), where T represents a transposed matrix.

[0045] After traversing each missing view angle in the set of missing view angles, the new view image generation model can be used to obtain the respective target images, and the set of target images is referred to as a set of missing view images, denoted as I miss .

[0046] Step S105: based on the original input image and the set of missing view images, three-dimensional reconstruction is performed on the object to obtain a three-dimensional reconstruction result of the object.

[0047] The original input image is denoted as I ori , and the original input image I ori and the set of missing view images I miss are input into a three-dimensional reconstruction model, so that the three-dimensional reconstruction model can be used to perform three-dimensional reconstruction on the object to obtain a three-dimensional reconstruction result of the object.

[0048] For example, a three-dimensional reconstruction model based on deep learning can be used to perform three-dimensional reconstruction, learn an implicit or explicit three-dimensional representation of a scene through two-dimensional images collected from different view angles and corresponding camera pose information, and render a two-dimensional image under a given camera pose to achieve high-fidelity three-dimensional modeling and new view synthesis. For example, NeRF or 3D Gaussian Splatting can be selected.

[0049] In one specific embodiment, the original input image is denoted as I ori , and the set of missing view images I miss is input into a three-dimensional reconstruction model, and the camera intrinsic parameters and camera extrinsic parameters (camera pose) corresponding to each image are input, so that a three-dimensional reconstruction result of the object can be obtained. For example, NeRF specifically outputs a three-dimensional radiation field representation of a scene, and 3DGS specifically outputs a set of three-dimensional Gaussian distributions.

[0050] In one specific embodiment of the present application, the following can also be included: render the obtained three-dimensional reconstruction result of the object according to the N camera sampling view angles to obtain a rendered image under a corresponding camera sampling view angle; evaluate the quality of each rendered image to obtain a quality evaluation result of each rendered image, and determine whether the rendered images that do not meet the quality requirement exceed a preset second quantity threshold; If yes, the process ends. If no, based on the rendered images that do not meet the quality requirement, a to-be-optimized image set composed of the rendered images that do not meet the quality requirement is established. perform quality optimization on the rendered images in the to-be-optimized image set to obtain an optimized image set. based on the original input image, the missing view angle image set, and the optimized image set obtained in the current iteration, perform three-dimensional reconstruction on the object again to obtain a three-dimensional reconstruction result of the object, and return to perform the operation of rendering the obtained three-dimensional reconstruction result of the object according to the N camera sampling view angles to obtain a rendered image under a corresponding camera sampling view angle.

[0051] For reference Figure 2 In the above embodiments, the pre-optimization process is implemented, that is, the missing view angles are complemented by detecting the missing view angles and supplementing the missing view angle image set, so as to improve the spatial coverage of the input image and the completeness of the object reconstruction. In this embodiment, the post-optimization process is further provided, which is considered that although the above embodiment jointly uses the original input image I ori and the missing view angle image set I miss as the input image to ensure a high coverage, due to the fact that the quality of the original input image is uneven and the complexity of different parts of the object is different, the reconstructed 3D content may not have extremely high quality in any view angle. Therefore, the iterative optimization strategy based on the rendering quality feedback is provided in this embodiment.

[0052] In this embodiment, the obtained three-dimensional reconstruction result of the object is rendered in the surround view, and then the reconstruction quality of the 3D content is evaluated. When rendering in the surround view, the obtained 3D content is rendered according to the N camera sampling view angles set in advance, so as to obtain a 2D rendered image under a corresponding camera sampling view angle. It can be understood that there are N 2D rendered images in total.

[0053] For each rendered image, quality assessment needs to be performed, so as to obtain the respective quality assessment result of each rendered image. The specific implementation manner of quality assessment can be various, for example, the quality assessment can be performed according to the “technical defects” of the image, such as noise, compression artifacts, blur, exposure problems, etc., to obtain the quality assessment result. For another example, the quality assessment can be performed based on the aesthetic score of the image.

[0054] In a specific embodiment of the present application, the quality assessment of each rendered image to obtain the respective quality assessment result of each rendered image can specifically include: For any rendered image, the image technical score and the image aesthetic score of the rendered image are determined and weighted and superimposed, and the value obtained after the weighted and superimposed is taken as the quality assessment result of the rendered image.

[0055] This embodiment takes into account that if only the technical score is used to evaluate the image quality, the image that is “technically clean but may have poor composition” may be selected, and if only the aesthetic score is used, the image that is “good-looking but may have defects / transformations” may be selected. Therefore, in this embodiment, the two are weighted so that both images with low quality and geometric / noise problems and images that are visually unacceptable, such as color anomalies and texture splitting, can be captured. That is, in this embodiment, the image quality can be more accurately measured through the image technical score and the image aesthetic score.

[0056] Specifically, when performing the image technical score, the DOVER evaluation model can be used, which can effectively evaluate the “technical defects” of the image, such as noise, compression artifacts, blur, exposure problems, etc., to obtain the image technical score of the rendered image. When performing the image aesthetic score, for example, the Aesthetic Score Predictor model can be specifically used, which is based on CLIP embedding and MLP trained on AVA data to obtain the average aesthetic score of the picture, which meets the human preference index. Of course, in other specific embodiments, other types of models can be used to implement the image technical score and the image aesthetic score of the rendered image.

[0057] When performing the weighted and superimposed of the image technical score and the image aesthetic score, the weight value can be set as needed, for example, in actual application, the weight value of the image technical score is 0.7 by default, and the weight value of the image aesthetic score is 0.3 by default, so that more emphasis is placed on technical quality.

[0058] After the quality evaluation of each rendered image is performed and the quality evaluation result of each rendered image is obtained, the number of rendered images that do not meet the quality requirement can be obtained. For example, in one case, the value obtained after the image technology score and the image aesthetic score of each rendered image are weighted and superimposed is the quality evaluation result of the rendered image. If the value obtained after the weighted superimposition is lower than the preset score threshold, it can be determined that the rendered image is a rendered image that does not meet the quality requirement, otherwise it is a rendered image that meets the quality requirement.

[0059] If the number of rendered images that do not meet the quality requirement does not exceed the preset second number threshold, it indicates that the rendering quality of the 3D content is very high, and therefore the process can be directly ended. Otherwise, the iterative optimization needs to be continued. The value of the second number threshold can be set according to actual needs. The lower the setting is, the higher the requirement is. For example, in one case, the second number threshold is set to the minimum value 0, which means that only when each rendered image meets the quality requirement, the iteration is ended, otherwise the iteration needs to be continued.

[0060] If the number of rendered images that do not meet the quality requirement exceeds the preset second number threshold, at this time, a to-be-optimized image set composed of rendered images that do not meet the quality requirement needs to be established based on the rendered images that do not meet the quality requirement. For example, in one simple way, all rendered images that do not meet the quality requirement are directly used as the to-be-optimized image set established.

[0061] In one specific embodiment of the present application, the to-be-optimized image set composed of rendered images that do not meet the quality requirement can be specifically established based on the rendered images that do not meet the quality requirement, which can include: determining whether the number of rendered images that do not meet the quality requirement is lower than a preset third number threshold; if yes, establishing a to-be-optimized image set composed of all rendered images that do not meet the quality requirement; if no, selecting M rendered images from all rendered images that do not meet the quality requirement by using a farthest point sampling algorithm, and establishing a to-be-optimized image set composed of the M rendered images.

[0062] The implementation considers that if the number of rendered images that do not meet the quality requirement is not particularly large, that is, the number of rendered images that do not meet the quality requirement is lower than the third quantity threshold, all rendered images that do not meet the quality requirement can be directly taken as the to-be-optimized image set. However, if there are many rendered images that do not meet the quality requirement, part of them can be selected to construct the to-be-optimized image set. Such an operation can reduce the amount of calculation, and subsequent three-dimensional reconstruction of the object needs to be performed based on the original input image, the missing view image set, and the optimized image set obtained in this round of iteration. If the number of images in the optimized image set is too large, the distortion degree of the reconstructed 3D content may be increased. Therefore, only part of them is selected to construct the to-be-optimized image set at this time.

[0063] Of course, when selecting, the rendered images that do not meet the quality requirement can be selected relatively uniformly. Therefore, M rendered images can be selected relatively uniformly from all rendered images that do not meet the quality requirement by using the farthest point sampling algorithm, and a to-be-optimized image set composed of the M rendered images is established. M is a preset quantity value, which can be set to a relatively appropriate value according to actual needs.

[0064] After obtaining the to-be-optimized image set, quality optimization needs to be performed on each rendered image in the to-be-optimized image set to obtain an optimized image set. The quality optimization can usually be implemented by using a multi-modal generation model. For example, in one specific implementation, the quality optimization of the rendered images in the to-be-optimized image set to obtain the optimized image set can specifically include the following steps. For any rendered image in the to-be-optimized image set, the rendered image and the guide text are input into a preset multi-modal generation model to obtain an optimized image for the rendered image output by the multi-modal generation model. The obtained each optimized image is taken as the obtained optimized image set.

[0065] In the implementation, the rendered image and the guide text are input into a preset multi-modal generation model, for example, a multi-modal large model such as GPT-4o / GPT-5, Gemini 2.5 Flash Image, and the like. Through the guidance of the guide text, the multi-modal generation model can optimize the input rendered image, thereby outputting a corresponding optimized image. Each optimized image constitutes the optimized image set.

[0066] In an embodiment, the guidance text can be specifically text for prompting the multi-modal generation model to perform image optimization, and the guidance text includes one or more of the following expressions: enhancing clarity, enhancing details, eliminating noise, eliminating distortion, adjusting light and contrast to present a natural balance effect, and optimizing color saturation, so as to guarantee a better guidance effect.

[0067] For example, in one specific embodiment, a reference guidance text is specifically set as: "Please optimize this low-quality image, maintain the integrity of the original scene, objects, and characters unchanged, upgrade to 4K resolution, enhance clarity and details, eliminate noise and compression distortion, adjust light and contrast to present a natural balance effect, and optimize color saturation to obtain a vivid but realistic visual effect, without changing the perspective or adding new elements." Such a guidance text effectively realizes the optimization of the image without changing the perspective or adding new elements.

[0068] After the optimization in this round, the optimized image set is obtained, together with the original input image I ori and the missing perspective image set I miss , the object can be reconstructed in three dimensions again, so as to obtain the three-dimensional reconstruction result of the object again. At this time, the operation of rendering the obtained three-dimensional reconstruction result of the object according to the N camera sampling perspectives to obtain the rendered image under the corresponding camera sampling perspective can be performed again, so as to perform the next iteration, until whether the rendered image that does not meet the quality requirement in a certain iteration round does not exceed the preset second quantity threshold, so that the iteration can be ended, and at this time, a very high-quality three-dimensional reconstruction result can be obtained.

[0069] In addition, it should be further pointed out that in actual application, it can be necessary to go through one or more iterations. After each iteration, the original input image I ori , the missing perspective image set I miss , and the optimized image set in this round are selected to perform three-dimensional reconstruction of the object, that is, the optimized image set obtained in the previous round does not need to be used in the current round. Such a design is considering that with the increase of the iteration round, if the optimized image set obtained in each iteration round is used to participate in the three-dimensional reconstruction of the object in the current round, the proportion of the original input image I ori and the missing perspective image set I miss will be reduced, which can increase the distortion degree of the three-dimensional reconstruction result, and is not conducive to guaranteeing the quality. Therefore, the original input image I ori , the missing perspective image set I miss , and the optimized image set obtained in the current round are used for three-dimensional reconstruction of the object each time.

[0070] The application scheme considers that incomplete view coverage will affect the quality of three-dimensional reconstruction of the object. To this end, the view coverage degree is optimized. Specifically, first, N camera sampling views are established in the surrounding space of the object bounding sphere, and then based on the original input image, each missing view in the N camera sampling views that is not covered by the original input image is determined. It can be understood that if the N camera sampling views are all covered by the original input image, it means that the original input image realizes complete view coverage, and high-quality three-dimensional reconstruction of the object can be directly realized. If there is a missing view, the application scheme will establish a missing view set composed of at least one missing view based on each missing view. Then, for the missing view in the missing view set, the image of the view is supplemented, specifically, an image with a similar view is obtained from the original input image as a reference image, and a new view image generation model is used to generate a target image of the missing view, so that a missing view image set containing each target image is obtained. The missing view image set is a view supplement to the original input image, so based on the original input image and the missing view image set, the view coverage degree can be effectively improved, and accordingly the three-dimensional reconstruction of the object can be performed to obtain a high-quality three-dimensional reconstruction result of the object.

[0071] In summary, the application scheme can effectively realize three-dimensional reconstruction of the object and improve the reconstruction quality.

[0072] Corresponding to the above method and system embodiments, the application embodiments also provide an electronic device, a computer readable storage medium and a computer program product, which can be mutually corresponding and referred to above.

[0073] Referring to Figure 3 As shown in the figure, the electronic device can include: The memory 301 is configured to store a computer program. The processor 302 is configured to execute the computer program to implement the steps of the three-dimensional reconstruction method of the object in any of the above embodiments.

[0074] The computer program product includes computer programs / instructions that are executed by the processor to implement the steps of the three-dimensional reconstruction method of the object in any of the above embodiments.

[0075] Referring to Figure 4The computer readable storage medium 40 stores a computer program 41, which, when executed by the processor, implements the steps of the object three-dimensional reconstruction method in any of the above embodiments. The computer readable storage medium 40 mentioned herein includes a RAM (Random Access Memory), an internal memory, a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a register, a hard disk, a removable disk, or any other form of storage medium known in the art.

[0076] Those skilled in the art will further appreciate that the individual steps of the examples described in connection with the embodiments disclosed herein can be realized by electronic hardware, computer software, or any combination thereof. To clearly illustrate the interchangeability of hardware and software, and to avoid obscuring the disclosure, the various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Skilled persons can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0077] The principles and implementation manners of the present application are described herein by using specific examples, and the above description of the examples is only applicable to helping understand the method and core idea of the present application. It should be noted that, for those skilled in the art, without departing from the principles of the present application, some improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the present application.

Claims

1. A method of three-dimensional reconstruction of an object, characterized by, The method comprises the following steps: establishing N camera sampling view angles in the surrounding space of the object enclosing sphere; N is a positive integer; determining each missing view angle in the N camera sampling view angles which is not covered by the original input image based on the original input image; establishing a missing view angle set composed of at least one missing view angle; for each missing view angle in the missing view angle set, obtaining an image with a similar view angle as a reference image from the original input image, and generating a target image of the missing view angle by using a new view angle image generation model to obtain a missing view angle image set containing each target image; based on the original input image and the missing view angle image set, performing three-dimensional reconstruction on the object to obtain a three-dimensional reconstruction result of the object.

2. The object three-dimensional reconstruction method of claim 1, wherein, The method comprises the following steps: establishing N camera sampling view angles in the surrounding space of the object enclosing sphere, comprising: uniformly arranging N grid points on the object enclosing sphere, and taking the direction formed by the N grid points and the object center as the orientation parameter of each of the N camera sampling view angles; 3. The object three-dimensional reconstruction method of claim 2, wherein, for any one camera sampling view angle, obtaining the position parameter of the camera sampling view angle based on the observation radius of the camera, the coordinates of the object center, and the orientation parameter of the camera sampling view angle. For any camera sampling view, the position parameter of the camera sampling view is obtained according to the calculation manner of O i =C+Rc×P i . Wherein, O i represents the position parameter of the i-th camera sampling view angle, C represents the coordinates of the object center, Rc represents the observation radius of the camera, P i represents the orientation parameter of the i-th camera sampling view angle.

4. The method of claim 1, wherein For each camera sampling view angle, obtaining the position parameter of the camera sampling view angle based on the observation radius of the camera, the coordinates of the object center, and the orientation parameter of the camera sampling view angle, comprising: determining each missing view angle in the N camera sampling view angles which is not covered by the original input image based on the original input image, comprising: for any one camera sampling view angle in the N camera sampling view angles, determining the number of images in the original input image which overlap with the camera sampling view angle in view angle; judging whether the number is lower than a preset overlap threshold value; if yes, determining that the camera sampling view angle is a missing view angle which is not covered by the original input image; 5. The method of claim 1, wherein, if no, determining that the camera sampling view angle is not a missing view angle which is not covered by the original input image. establishing a missing view angle set composed of at least one missing view angle, comprising: judging whether the number of all missing view angles is lower than a preset first number threshold value; if yes, establishing a missing view angle set composed of all missing view angles; if no, based on all missing view angles, selecting K1×F missing view angles therefrom by using a farthest point sampling algorithm, and establishing a missing view angle set composed of K1×F missing view angles; 6. The method of claim 1, wherein, wherein F represents the number of all missing view angles, and K1 is a coefficient value less than or equal to 1. for each missing view angle in the missing view angle set, obtaining an image with a similar view angle as a reference image from the original input image, and generating a target image of the missing view angle by using a new view angle image generation model to obtain a missing view angle image set containing each target image, comprising: for each missing view angle in the missing view angle set, obtaining an image with a smallest view angle included angle with the missing view angle as the reference image of the missing view angle from the original input image; obtaining the relative pose between the missing view angle and the reference image; inputting the relative pose and the reference image into a new-view image generation model to obtain a target image of the missing view generated by the new-view image generation model; after obtaining the target image of each missing view in the set of missing views, obtaining a set of missing-view images containing each target image.

7. The method of claim 1 to 6, wherein, Further comprising: rendering the obtained three-dimensional reconstruction result of the object according to the N camera sampling views to obtain rendered images under the corresponding camera sampling views; performing quality assessment on each of the rendered images to obtain a quality assessment result of each of the rendered images, and determining whether the rendered images that do not meet the quality requirements exceed a preset second quantity threshold; if yes, ending the process; if no, based on the rendered images that do not meet the quality requirements, establishing a to-be-optimized image set composed of the rendered images that do not meet the quality requirements; performing quality optimization on the rendered images in the to-be-optimized image set to obtain an optimized image set; based on the original input images, the set of missing-view images, and the optimized image set obtained in this round of iteration, re-performing three-dimensional reconstruction on the object to obtain a three-dimensional reconstruction result of the object, and returning to perform the operation of rendering the obtained three-dimensional reconstruction result of the object according to the N camera sampling views to obtain rendered images under the corresponding camera sampling views.

8. The method of claim 7, wherein, performing quality assessment on each of the rendered images to obtain a quality assessment result of each of the rendered images, including: for any one of the rendered images, determining an image technical score and an image aesthetic score of the rendered image and performing weighted superposition, and taking the value obtained after the weighted superposition as the quality assessment result of the rendered image.

9. The method of claim 7, wherein, performing quality optimization on the rendered images in the to-be-optimized image set to obtain an optimized image set, including: for any one of the rendered images in the to-be-optimized image set, inputting the rendered image and a guide text into a preset multi-modal generation model to obtain an optimized image for the rendered image output by the multi-modal generation model; taking each of the obtained optimized images as the obtained optimized image set; wherein the guide text is a text for prompting the multi-modal generation model to perform image optimization, and the guide text includes one or more expressions of enhancing clarity, enhancing details, eliminating noise, eliminating distortion, adjusting light and contrast to present a natural balance effect, and optimizing color saturation.

10. An electronic device, comprising: including: a memory for storing a computer program; a processor for executing the computer program to implement the steps of the object three-dimensional reconstruction method according to any one of claims 1 to 9.

Images