High-fidelity 3D image rendering method, device, system and storage medium

Abstract

The application discloses a highly realistic 3D image presentation method, device, system and storage medium, and belongs to the fields of image processing and three-dimensional display technology. The method comprises the following steps: initializing an iteration variable and setting an upper limit; obtaining a ratio of two camera transfer functions; taking a target two-dimensional image from multiple perspectives by using a first camera and reconstructing a three-dimensional image; after display, taking a display image from the same perspective by using a second camera; performing similarity evaluation on the second camera image and the first camera image processed by the ratio; if all perspective similarities meet the requirements or the iteration exceeds the upper limit, the current image is the final result; otherwise, calculating an improved two-dimensional image, reconstructing a new three-dimensional image, and displaying again after incrementing the iteration. Through iteration feedback and de-filtering correction, shape distortion and color deviation are suppressed, a 3D mirror image effect of real-time, high-fidelity and naked-eye observable is realized, and the application has the same realism as a traditional mirror.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a method, apparatus, system, and storage medium for presenting highly realistic 3D images. Background Technology

[0002] In recent years, continuous breakthroughs in 3D image processing technology have powerfully driven the rapid development of 3D image acquisition and presentation technologies. The former is mainly reflected in 3D cameras and 3D structure reconstruction technology, while the latter is represented by 3D reproduction technologies such as holographic projection and 3D display technology. As related processing procedures and equipment structures become increasingly complex, the 3D images acquired through 3D image processing inevitably exhibit shape distortion and color deviation compared to the original photographed object. Especially in fields with extremely high requirements for image fidelity, such as medical diagnosis, industrial manufacturing, and cosmetic surgery, the market has created an urgent demand for 3D reproduction technologies capable of highly restoring the true appearance of objects.

[0003] Although high-performance 2D imaging cameras capable of capturing high-fidelity images and videos are already available on the market, and display devices with high spatial resolution and high-fidelity display characteristics are also commercially available, and even 3D display technologies that can display objects at life-size on and off screen and support naked-eye viewing have emerged, even simple integration of these cutting-edge technologies still cannot achieve the real-time reproduction of true colors and shapes like a traditional mirror. Summary of the Invention

[0004] To address the aforementioned problems in the prior art, the present invention provides a method, apparatus, system, and storage medium for presenting highly realistic 3D images.

[0005] The technical problem to be solved by this invention is achieved through the following technical solution: In a first aspect, the present invention provides a method for rendering highly realistic 3D images, comprising: Step A: Initialize the iteration variable i and set the iteration limit i_limit; Step B: Obtain the ratio of the transfer functions of the first camera and the second camera. ; Step C: Use the first camera to capture two-dimensional images of the original target from multiple known perspectives. ; Step D: Two-dimensional images from multiple known viewpoints captured by the first camera. Reconstruct the 3D image I_3D; Step E: Display the 3D image I_3D using a 3D display device; Step F: Use a second camera to capture two-dimensional images of the three-dimensional image I_3D displayed by the three-dimensional display device from the plurality of known viewpoints. ; Step G: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] Processed as The similarity evaluation index is used to evaluate the results under each known perspective. and The similarity between the two views is determined. If the similarity meets the requirements under all known viewpoints, then the 3D image I_3D currently displayed in step E is the final presentation result. If the similarity does not meet the requirements, then the size relationship between i and i_limit is further determined. If i ≥ i_limit, then the 3D image I_3D displayed in step E is the final presentation result. If i < i_limit, step H is executed. Step H: Calculate improved 2D images from multiple known viewpoints. Improved 2D images based on multiple known viewpoints Reconstruct the 3D image I_3D, increment i by 1, and return to step E.

[0006] Optionally, the ratio of the transfer function is pre-calibrated in the following manner: The same standard board was photographed using a first camera and a second camera under the same shooting conditions, resulting in an image of the standard board captured by the first camera. Standard board image captured by the second camera ; calculate or calculation ;in, This indicates pixel division or deconvolution.

[0007] Optionally, the method further includes: generating supplementary images from multiple unknown perspectives based on two-dimensional images captured by the first camera from multiple known viewpoints using a generative AI model; Step D specifically involves two-dimensional images taken by the first camera from multiple known viewpoints. The three-dimensional image I_3D is jointly reconstructed using supplementary images from multiple unknown perspectives generated by the generative AI model.

[0008] Optionally, in step A, i_limit is set to 100.

[0009] Secondly, the present invention provides a highly realistic 3D image presentation device, comprising: an iteration control module, a device control module, and an image processing module; the iteration control module includes: an initialization unit, a judgment and comparison unit, and an iteration count accumulation unit; the device control module includes: a first camera control unit, a second camera control unit, and a display device control unit; the image processing module includes: a parameter acquisition unit, a 3D reconstruction unit, a first image processing unit, an image evaluation unit, and a second image processing unit; wherein: The initialization unit is used to initialize the iteration variable i and set the iteration limit i_limit; The parameter acquisition unit is used to acquire the ratio of the transfer functions of the first camera and the second camera. ; The first camera control unit is used to control the first camera to capture two-dimensional images of the original target from multiple known viewpoints. ; The 3D reconstruction unit is used to reconstruct two-dimensional images from multiple known viewpoints captured by the first camera. Reconstruct the 3D image I_3D; The display device control unit is used to control the three-dimensional display device to display the three-dimensional image I_3D; The second camera control unit is used to control the second camera to capture two-dimensional images of the three-dimensional image I_3D displayed by the three-dimensional display device from the plurality of known viewpoints. ; The first image processing unit is used to process images from each known viewpoint. Processed as ; The image evaluation unit is used to evaluate each known viewpoint using a similarity evaluation index. and The similarity between the images is determined; if the similarity meets the requirements under all known viewpoints, then the three-dimensional image I_3D currently displayed by the three-dimensional display device is the final presentation result; if the similarity does not meet the requirements, then the judgment and comparison unit is triggered to work. The judgment and comparison unit is used to determine the size relationship between i and i_limit; if i ≥ i_limit, then the three-dimensional image I_3D currently displayed by the three-dimensional display device is the final presentation result; if i < i_limit, then the second image processing unit is triggered to work. The second image processing unit is configured to: calculate improved two-dimensional images from the plurality of known viewpoints. The 3D reconstruction unit is triggered based on the improved 2D images from the multiple known viewpoints. The three-dimensional image I_3D is reconstructed; the iteration count accumulation unit is triggered to increment i by 1, and then the display device control unit, the second camera control unit, the first image processing unit, and the image evaluation unit are triggered to work in sequence.

[0010] Optionally, the first camera control unit is further configured to control the first camera to capture an image of a standard board before capturing the original target, thereby obtaining an image of the standard board captured by the first camera. ; The second camera control unit is also used to control the second camera to photograph the standard board under the same shooting environment before photographing the original target, so as to obtain an image of the standard board captured by the second camera. ; The device further includes: a ratio calibration module; The ratio calibration module is used to calculate the ratio before the parameter acquisition unit operates. or calculation ;in, This indicates pixel division or deconvolution.

[0011] Optionally, the highly realistic 3D image rendering device further includes: a generative AI model; The generative AI model is used to generate supplementary images from multiple unknown perspectives based on two-dimensional images captured by the first camera from multiple known perspectives. The 3D reconstruction unit is specifically used for: reconstructing two-dimensional images based on multiple known viewpoints captured by the first camera. The three-dimensional image I_3D is jointly reconstructed using supplementary images from multiple unknown perspectives generated by the generative AI model.

[0012] Optionally, the initialization unit is specifically set to i_limit=100.

[0013] Thirdly, the present invention provides a highly realistic 3D image presentation system, including: a first camera, a second camera, a three-dimensional display device, and a controller; The controller is connected to the first camera, the second camera, and the three-dimensional display device respectively, and is used to control the first camera, the second camera, and the three-dimensional display device to work together to implement the steps of the highly realistic 3D image presentation method according to any one of claims 1-4.

[0014] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described highly realistic 3D image rendering method.

[0015] The highly realistic 3D image presentation method provided by this invention achieves real-time, high-fidelity naked-eye 3D image presentation through iterative feedback and inverse filtering correction. Specifically, this invention utilizes the ratio of the transfer functions of two cameras. The system calibrates for distortion and compares the similarity between the displayed image and the corrected image at each known viewpoint. If the similarity is not met, inverse filtering compensation is performed (calculating the improved 2D image and reconstructing the 3D image) until a preset iteration upper limit or similarity threshold is reached. This simultaneously suppresses shape distortion and color deviation introduced by the camera and display device, thus presenting a 3D mirror effect highly consistent with the original target, meeting the needs of fields with extremely high fidelity requirements, such as medical and industrial manufacturing. Furthermore, by setting iteration termination conditions, infinite loops are avoided while ensuring image quality, achieving substantial real-time operation from acquisition to presentation, allowing the target's motion to be reflected almost synchronously in the corrected 3D image.

[0016] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a highly realistic 3D image rendering method provided in an embodiment of the present invention; Figure 2 The diagram shows a structural schematic of a highly realistic 3D image rendering device provided in an embodiment of the present invention; Figure 3 It shows Figure 2 A schematic diagram of the device in use with a camera and a 3D display device; Figure 4 This is a schematic diagram of the structure of a highly realistic 3D image presentation system provided in an embodiment of the present invention. Detailed Implementation

[0018] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0019] To achieve highly realistic 3D image presentation, realizing the effect of real-time reproduction of true colors and shapes like a traditional mirror, this invention provides a highly realistic 3D image presentation method. (See attached image) Figure 1 As shown, the method includes the following steps: Step A: Initialize the iteration variable i and set the iteration limit i_limit.

[0020] For example, the iteration variable i is initialized to 0, and the iteration limit i_limit is set to 100 (times). Of course, it is not limited to this. In practice, the iteration limit i_limit can be flexibly adjusted according to the actual effect.

[0021] Step B: Obtain the ratio of the transfer functions of the first camera and the second camera. .

[0022] Here, the first camera and the second camera are digital cameras capable of providing 2D image data to the computer. The camera's transfer function refers to the mapping relationship between the original scene (or original image) and the digital image output by the camera; it describes the overall modulation effect of the camera on the image's brightness, color, and spatial frequency characteristics.

[0023] Alternatively, in one implementation, step B can involve first obtaining the transfer functions of the first camera and the second camera respectively, and then calculating their ratio. In practice, there are several ways to obtain the camera's transfer function. For example, in one implementation, a standard calibration board with known optical characteristics (such as a checkerboard, dot array, or test card with precise grayscale gradients) can be placed in front of the camera. After taking a picture, the camera's transfer function is solved by least-squares fitting or a system identification algorithm based on the difference between the known pattern on the calibration board and the imaging result. In another implementation, sub-pixel-sized point light sources (such as pinhole lights) or single-slit lights can be used for direct imaging in a dark room. The point light source image output by the camera is the system's point spread function (PSF). Performing a two-dimensional Fourier transform on the PSF yields the optical transfer function (OTF), whose modulus is the modulation transfer function (MTF) and whose phase component is the phase transfer function (PTF). This method can directly measure the camera's complex transfer characteristics in the spatial frequency domain, i.e., the transfer function.

[0024] In another implementation, step B can also directly obtain the ratio of the transfer functions of the first camera and the second camera in the manner shown below. : (1) Using the first camera and the second camera to photograph the same standard board under the same shooting conditions, the image of the standard board photographed by the first camera is obtained. Standard board image captured by the second camera ; (2) Calculation or calculation ;in, This indicates pixel division or deconvolution.

[0025] In this implementation, by having the first and second cameras photograph the same standard board under the same conditions, and then performing point-by-point division or deconvolution on the two images, the inherent patterns or textures of the standard board itself can be directly eliminated, thereby obtaining a ratio that is only related to the transfer functions of the two cameras. The ratio This objectively reflects the system differences between the two cameras, providing an accurate benchmark for subsequent inverse filtering correction and avoiding interference from the characteristics of the standard board itself on the correction parameters. Two calculation methods are provided (two-dimensional distribution). or scalar The appropriate method can be flexibly selected based on the actual system characteristics: when the camera transfer function has spatial variations, a two-dimensional distribution can be used to achieve fine compensation; when the transfer function is approximately spatially constant, an average scalar can be used to significantly reduce computational load and storage overhead, and improve real-time processing speed.

[0026] In practice, the first and second cameras can be of the same or different models, and the ratio of their transfer functions... It can be pre-calibrated, thus allowing the ratio to be... As a fixed parameter, it participates in real-time image correction, avoiding repeated calibration every time a high-fidelity 3D image is presented, thereby reducing the computational burden of online processing, which is conducive to achieving low-latency real-time 3D mirror feedback and ensuring that moving targets can be reproduced synchronously and realistically.

[0027] Step C: Use the first camera to capture two-dimensional images of the original target from multiple known perspectives. .

[0028] Thus, by acquiring two-dimensional images from multiple different perspectives, sufficient parallax information can be provided for subsequent three-dimensional reconstruction.

[0029] For example, with the target as the center, a shooting angle is set every 30° to 45° in the horizontal direction (e.g., 0°, ±30°, ±60°, ±90°), while a top-down angle (e.g., 15°~30°) can be added to obtain stereoscopic information in the height direction. The above angle settings can provide sufficient parallax and overlap area for subsequent 3D reconstruction, while avoiding data redundancy caused by too dense viewing angles.

[0030] Step D: Two-dimensional images from multiple known viewpoints captured by the first camera. Reconstruct the 3D image I_3D.

[0031] Here, existing 3D reconstruction techniques can be combined to reconstruct the 3D image (I_3D), such as stereo vision matching, multi-view dense reconstruction, or deep learning-based 3D estimation. Given that 3D reconstruction technology is already mature in relevant existing technologies, this invention will not elaborate further on it.

[0032] Step E: Display the 3D image I_3D using a 3D display device.

[0033] Here, the preferred 3D display device is a glasses-free 3D display device.

[0034] Step F: Use a second camera to capture two-dimensional images of the three-dimensional image I_3D displayed on the three-dimensional display device from the aforementioned multiple known perspectives. .

[0035] Here, the multiple known viewpoints when using the second camera to capture the 3D image I_3D displayed on the 3D display device can be the same as the multiple known viewpoints when using the first camera to capture the original target in step C.

[0036] Step G: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] Processed as The similarity evaluation index is used to evaluate the results under each known perspective. and The similarity between them.

[0037] In step G, Processed as At that time, among them This represents an operation: when the ratio When it is a scalar, This represents pixel-by-pixel multiplication, when the ratio When it is a two-dimensional distribution, This represents the convolution operation.

[0038] Similarity evaluation metrics are used for quantification and The difference between them. This metric can be any known similarity or distortion measure, such as mean squared error (MSE), peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), or normalized cross-correlation (NCC).

[0039] In practice, Processed as Even so, deviations (spatial distortion and chromaticity deviation) are still unavoidable between the two, which can be expressed as (1+) ) ;in The error factor, whose sources may be complex, can be equivalently considered as a slight change in the display device's transfer function. This invention, through an iterative feedback process, can gradually reduce... Until the preset fidelity requirement is met.

[0040] Specifically, such as Figure 2 As shown, if the similarity requirement is met in all known viewpoints, then the currently displayed 3D image I_3D in step E is the final presentation result.

[0041] Understandably, when the similarity calculated under each known viewpoint meets the preset threshold (e.g., structural similarity index ≥ 0.99 or normalized cross-correlation close to 1), it is determined that the currently displayed 3D image has reached the required fidelity and there is no need to continue iterative correction.

[0042] like Figure 2As shown, if the condition that the similarity under all known viewpoints meets the requirement is not met, then the relationship between i and i_limit is further determined; if i ≥ i_limit, then the 3D image I_3D displayed in step E is the final presentation result; if i < i_limit, step H is executed.

[0043] In step G, the similarity evaluation operation provides a clear termination criterion for iterative error compensation, which ensures that the final output 3D image is highly consistent with the original target and avoids unnecessary recalculation.

[0044] Step H: Calculate the improved two-dimensional images under the above-mentioned multiple known viewpoints. Improved two-dimensional images based on these known perspectives Reconstruct the 3D image I_3D, increment i by 1, and return to step E.

[0045] Understandable In the molecules This indicates a convolution operation, where the denominator contains... This indicates pixel-wise multiplication or convolution operations.

[0046] Furthermore, improved two-dimensional images based on various known viewpoints The existing method mentioned in step D can also be used to reconstruct the 3D image I_3D, which will not be repeated here.

[0047] The highly realistic 3D image presentation method provided by this invention achieves real-time, high-fidelity naked-eye 3D image presentation through iterative feedback and inverse filtering correction. Specifically, this invention utilizes the ratio of the transfer functions of two cameras. The system calibrates for distortion and compares the similarity between the displayed image and the corrected image at each known viewpoint. If the similarity is not met, inverse filtering compensation is performed (calculating the improved 2D image and reconstructing the 3D image) until a preset iteration upper limit or similarity threshold is reached. This simultaneously suppresses shape distortion and color deviation introduced by the camera and display device, thus presenting a 3D mirror effect highly consistent with the original target, meeting the needs of fields with extremely high fidelity requirements, such as medical and industrial manufacturing. Furthermore, by setting iteration termination conditions, infinite loops are avoided while ensuring image quality, achieving substantial real-time operation from acquisition to presentation, allowing the target's motion to be reflected almost synchronously in the corrected 3D image.

[0048] Compared to existing technologies, traditional mirrors can only present a two-dimensional virtual image to the naked eye and require multiple components to achieve a 360° field of view. This invention, however, can provide a three-dimensional real image observable to the naked eye. Compared to holographic technology, which does not compensate for image deviations and cannot track object movement in real time, this invention uses reverse filtering (step G...) Processed as The operation and iterative error compensation enable real-time, high-fidelity 3D presentation; in addition, conventional 3D displays are only used to play off-the-shelf content and do not guarantee image fidelity. As an integrated system of imaging, display and correction, this invention can provide a high-fidelity 3D mirror effect similar to that of traditional optical mirrors.

[0049] In one embodiment, the highly realistic 3D image presentation method provided by the present invention may further include the following steps: based on two-dimensional images captured by a first camera from multiple known viewpoints, generating supplementary images from multiple unknown viewpoints using a generative AI model. Correspondingly, in step D, a three-dimensional image I_3D can be reconstructed jointly based on the two-dimensional images captured by the first camera from multiple known viewpoints and the supplementary images from multiple unknown viewpoints generated by the generative AI model.

[0050] Here, "unknown viewpoints" refer to viewpoints not actually captured by the first camera, such as the back of the target or occluded areas. Thus, in the 3D reconstruction process of step D, images from known viewpoints actually captured by the first camera are simultaneously used along with supplementary images generated by the generative AI model to jointly reconstruct the 3D image. By introducing generative AI to supplement missing viewpoint information, a more complete 3D model can be obtained without increasing the number of physical cameras. This effectively alleviates the problem of model holes or distortion caused by insufficient viewpoints, thereby improving the realism and completeness of the final 3D image.

[0051] The aforementioned generative AI models can employ neural network architectures based on multi-view synthesis. Examples include EscherNet and Multi-View Geometric Diffusion (MVGD) proposed at CVPR (Conference on Computer Vision and Pattern Recognition) 2024, PixelNeRF proposed at CVPR 2021, and Multi-View Autoregressive (MV-AR) announced at ICCV 2025.

[0052] In this invention, the end-to-end latency of the entire process from image acquisition (the first camera capturing the original target) to the final corrected 3D image display on the display device should be lower than a predetermined threshold perceptible to the human eye. This threshold is typically 20-50ms. When the latency is less than this range, the observer cannot perceive the time difference between image changes and target movement, thus obtaining an instant feedback experience similar to observing a traditional optical glass mirror. In practical applications, this latency requirement can be met through the aforementioned hardware acceleration and iterative feedback upper limit control.

[0053] Specifically, the steps of 2D image processing, 3D reconstruction, and similarity evaluation after image acquisition can all be executed in parallel by at least one hardware accelerator, namely a graphics processing unit (GPU), a field-programmable gate array (FPGA), or a digital signal processor (DSP). Furthermore, for tasks such as using generative AI models to complete missing viewpoints, generative AI models can be used for feedforward inference. Through the synergy of the aforementioned hardware and models, the end-to-end delay from the first camera capturing the original target image to the final 3D display device outputting the corrected image can be compressed to the millisecond level, thereby supporting real-time mirroring of dynamic targets.

[0054] The method provided in this invention can be applied to electronic devices. Specifically, the electronic device can be a desktop computer, a portable computer, a smart mobile terminal, a server, etc.

[0055] Based on the same inventive concept, the present invention also provides a highly realistic 3D image presentation device. This device is a computer program software product that can be deployed in an electronic device to implement the steps of the aforementioned highly realistic 3D image presentation method. See also Figure 2 As shown, the device includes: an iterative control module 1, an equipment control module 2, and an image processing module 3.

[0056] The iterative control module 1 includes an initialization unit 11, a judgment and comparison unit 12, and an iteration count accumulation unit 13; the device control module 2 includes a first camera control unit 21, a second camera control unit 22, and a display device control unit 23; and the image processing module 3 includes a parameter acquisition unit 31, a 3D reconstruction unit 32, a first image processing unit 33, an image evaluation unit 34, and a second image processing unit 35.

[0057] Initialization unit 11 is used to initialize the iteration variable i and set the iteration limit i_limit; here, preferably, the iteration limit i_limit = 100 (times), but it is not limited to this. In practice, the iteration limit i_limit can be flexibly adjusted according to the actual effect.

[0058] Parameter acquisition unit 31 is used to acquire the ratio of the transfer functions of the first camera and the second camera. ; The first camera control unit 21 is used to control the first camera a to capture two-dimensional images of the original target from multiple known perspectives. ; 3D reconstruction unit 32 is used for two-dimensional images from multiple known viewpoints captured by the first camera a. Reconstruct the 3D image I_3D; Display device control unit 23 is used to control the three-dimensional display device to display the three-dimensional image I_3D; The second camera control unit 22 is used to control the second camera b to capture two-dimensional images of the three-dimensional image I_3D displayed by the three-dimensional display device from multiple known perspectives. ; The first image processing unit 33 is used to process images from each known viewpoint. Processed as ; Image evaluation unit 34 is used to evaluate each known viewpoint using a similarity evaluation index. and The similarity between the images is determined; if the similarity meets the requirements under all known viewpoints, the 3D image I_3D currently displayed by the 3D display device is the final presentation result; if the similarity does not meet the requirements, the judgment and comparison unit 12 is triggered to work. The comparison unit 12 is used to determine the size relationship between i and i_limit; if i ≥ i_limit, the three-dimensional image I_3D currently displayed by the three-dimensional display device is the final presentation result; if i < i_limit, the second image processing unit 35 is triggered to work. The second image processing unit 35 is used to: calculate improved two-dimensional images from multiple known viewpoints. ; Triggering 3D reconstruction unit 32 based on improved 2D images from multiple known viewpoints Reconstruct the three-dimensional image I_3D; trigger the iteration count accumulation unit 13 to increase i by 1, and then sequentially trigger the display device control unit 23, the second camera control unit 22, the first image processing unit 33, and the image evaluation unit 34 to work.

[0059] In one embodiment, the first camera control unit 21 is further configured to control the first camera a to capture an image of the standard board before capturing the original target, thereby obtaining an image of the standard board captured by the first camera a. ; The second camera control unit 22 is also used to control the second camera b to photograph the standard board under the same shooting environment before photographing the original target, so as to obtain the standard board image photographed by the second camera b. ; Accordingly, the above-mentioned device may further include: a ratio calibration module; This ratio calibration module is used to calculate the ratio before the parameter acquisition unit operates. or calculation ;in, This indicates pixel division or deconvolution.

[0060] For example, Figure 3 It shows Figure 2 The diagram shown illustrates the device working in conjunction with a camera and a 3D display device. Figure 3 As shown, the process includes the following: (1) The parameter acquisition unit 31 acquires the ratio of the transfer functions of the first camera a and the second camera b. ; (2) First camera control unit 21 Figure 3 (Not shown in the image) Control the first camera a to capture two-dimensional images of the original target from multiple known perspectives. , Figure 3 Chinese Abbreviation; (3) 3D reconstruction unit 32 based on the images captured by the first camera a Reconstructing 3D images; (4) Display device control unit 23 Figure 3 (Not shown in the image) Controls a 3D display device to display the 3D image; (5) Second camera control unit 22 ( Figure 3 (Not shown) Controls the second camera b to capture two-dimensional images of the three-dimensional images displayed on the three-dimensional display device from multiple known perspectives. , Figure 3 Chinese Abbreviation; (6) The first image processing unit 33 will Processed as ; (7) Image evaluation unit 34 uses similarity evaluation index to evaluate each known viewpoint. and The similarity between the three-dimensional images is considered. If the similarity under all known viewpoints meets the requirements, the three-dimensional image I_3D currently displayed by the three-dimensional display device is the final presentation result, that is, "process ends". If the condition that the similarity under all known viewpoints meets the requirements is not met, the iteration control module 1 is triggered to work.

[0061] Specifically, the judgment and comparison unit 12 in the iteration control module 1 will judge the relationship between i and i_limit; if i ≥ i_limit, the three-dimensional image I_3D currently displayed by the three-dimensional display device is the final presentation result, that is, "process ends"; while if i < i_limit, the second image processing unit 35 is triggered to work.

[0062] (8) The second image processing unit 35 calculates improved two-dimensional images from multiple known viewpoints. , Figure 3 Chinese Abbreviated representation; then, the 3D reconstruction unit 32 is triggered to generate improved 2D images based on multiple known viewpoints. Reconstruct the three-dimensional image; trigger the iteration count accumulation unit 13 in the iteration control module 1 to increase i by 1, trigger the display device control unit 23 to control the three-dimensional display device to display the three-dimensional image, and then repeat the above steps (5) and subsequent steps until "process ends".

[0063] In one embodiment, the highly realistic 3D image rendering device further includes: a generative AI model; Generative AI models are used to generate supplementary images from multiple unknown perspectives based on two-dimensional images captured by a first camera from multiple known perspectives. Correspondingly, the 3D reconstruction unit 32 is specifically used for: two-dimensional images based on multiple known viewpoints captured by the first camera. The three-dimensional image I_3D is jointly reconstructed using supplementary images from multiple unknown perspectives generated by the generative AI model.

[0064] In one embodiment, the initialization unit 11 specifically sets i_limit=100.

[0065] In one embodiment, the present invention also provides a highly realistic 3D image presentation device, which further includes a voice control unit for responding to user voice commands and controlling interactive functions such as perspective switching and special effects addition.

[0066] Based on the aforementioned highly realistic 3D image rendering method and apparatus, the present invention also provides a highly realistic 3D image rendering system, such as... Figure 4 As shown, it includes: a first camera, a second camera, a 3D display device, and a controller; The controller is connected to the first camera, the second camera, and the 3D display device respectively, and is used to control the first camera, the second camera, and the 3D display device to work together to realize the steps of the above-mentioned highly realistic 3D image presentation method.

[0067] Specifically, the controller is equipped with the aforementioned highly realistic 3D image rendering device, and the highly realistic 3D image rendering method is implemented by operating the device.

[0068] In another embodiment of the present invention, a computer-readable storage medium is also provided. This computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-described highly realistic 3D image rendering method.

[0069] It should be noted that the description of the device / system / computer-readable storage medium embodiments is relatively simple because they are basically similar to the method embodiments; relevant details can be found in the description of the method embodiments. Furthermore, all embodiments of the above-described highly realistic 3D image rendering method are applicable to the device / system / computer-readable storage medium embodiments and can achieve the same or similar beneficial effects.

[0070] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A method for rendering highly realistic 3D images, characterized in that, include: Step A: Initialize the iteration variable i and set the iteration limit i_limit; Step B: Obtain the ratio of the transfer functions of the first camera and the second camera. ; Step C: Use the first camera to capture two-dimensional images of the original target from multiple known perspectives. ; Step D: Based on two-dimensional images captured by the first camera from multiple known viewpoints. Reconstruct the 3D image I_3D; Step E: Display the 3D image I_3D using a 3D display device; Step F: Use a second camera to capture two-dimensional images of the three-dimensional image I_3D displayed by the three-dimensional display device from the plurality of known viewpoints. ; Step G: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] Processed as The similarity evaluation index is used to evaluate each of the known viewpoints. and The similarity between them; if the similarity under all known viewpoints meets the requirements, then the currently displayed 3D image I_3D in step E is the final presentation result; If the similarity does not meet the requirements, the relationship between i and i_limit is further determined; if i ≥ i_limit, the 3D image I_3D displayed in step E is the final result. If i < i_limit, proceed to step H; Step H: Calculate the improved two-dimensional images under the multiple known viewpoints. ; Improved 2D images based on the aforementioned multiple known viewpoints Reconstruct the 3D image I_3D, increment i by 1, and return to step E.

2. The highly realistic 3D image presentation method according to claim 1, characterized in that, The ratio of the transfer function is pre-calibrated in the following manner: The same standard board was photographed using a first camera and a second camera under the same shooting conditions, resulting in an image of the standard board captured by the first camera. Standard board image captured by the second camera ; calculate or calculation ;in, This indicates pixel division or deconvolution.

3. The highly realistic 3D image presentation method according to claim 1, characterized in that, The method further includes: generating supplementary images from multiple unknown perspectives based on two-dimensional images captured by the first camera from multiple known viewpoints, using a generative AI model; Step D specifically involves two-dimensional images taken by the first camera from multiple known viewpoints. The three-dimensional image I_3D is jointly reconstructed using supplementary images from multiple unknown perspectives generated by the generative AI model.

4. The highly realistic 3D image presentation method according to claim 1, characterized in that, In step A, i_limit is set to 100.

5. A highly realistic 3D image presentation device, characterized in that, include: Iteration control module, device control module, and image processing module; The iterative control module includes: an initialization unit, a judgment and comparison unit, and an iteration count accumulation unit; the device control module includes: a first camera control unit, a second camera control unit, and a display device control unit; the image processing module includes: a parameter acquisition unit, a 3D reconstruction unit, a first image processing unit, an image evaluation unit, and a second image processing unit; wherein: The initialization unit is used to initialize the iteration variable i and set the iteration limit i_limit; The parameter acquisition unit is used to acquire the ratio of the transfer functions of the first camera and the second camera. ; The first camera control unit is used to control the first camera to capture two-dimensional images of the original target from multiple known viewpoints. ; The 3D reconstruction unit is used to reconstruct two-dimensional images from multiple known viewpoints captured by the first camera. Reconstruct the 3D image I_3D; The display device control unit is used to control the three-dimensional display device to display the three-dimensional image I_3D; The second camera control unit is used to control the second camera to capture two-dimensional images of the three-dimensional image I_3D displayed by the three-dimensional display device from the plurality of known viewpoints. ; The first image processing unit is used to process images from each known viewpoint. Processed as ; The image evaluation unit is used to evaluate each known viewpoint using a similarity evaluation index. and The similarity between the images is determined; if the similarity meets the requirements under all known viewpoints, then the three-dimensional image I_3D currently displayed by the three-dimensional display device is the final presentation result; if the similarity does not meet the requirements, then the judgment and comparison unit is triggered to work. The judgment and comparison unit is used to determine the size relationship between i and i_limit; if i ≥ i_limit, then the three-dimensional image I_3D currently displayed by the three-dimensional display device is the final presentation result; if i < i_limit, then the second image processing unit is triggered to work. The second image processing unit is configured to: calculate improved two-dimensional images from the plurality of known viewpoints. ; Trigger 3D reconstruction unit 32 based on the improved two-dimensional images from the multiple known viewpoints The three-dimensional image I_3D is reconstructed; the iteration count accumulation unit is triggered to increment i by 1, and then the display device control unit, the second camera control unit, the first image processing unit, and the image evaluation unit are triggered to work in sequence.

6. The highly realistic 3D image presentation device according to claim 5, characterized in that, The first camera control unit is also configured to control the first camera to capture an image of the standard board before capturing the original target, thereby obtaining an image of the standard board captured by the first camera. ; The second camera control unit is also used to control the second camera to capture images of the standard board under the same shooting conditions before capturing the original target, thereby obtaining an image of the standard board captured by the second camera. ; The device further includes: a ratio calibration module; The ratio calibration module is used to calculate the ratio before the parameter acquisition unit operates. or calculation ;in, This indicates pixel division or deconvolution.

7. The highly realistic 3D image presentation device according to claim 6, characterized in that, The highly realistic 3D image rendering device also includes: a generative AI model; The generative AI model is used to generate supplementary images from multiple unknown perspectives based on two-dimensional images captured by the first camera from multiple known perspectives. The 3D reconstruction unit is specifically used for: reconstructing two-dimensional images based on multiple known viewpoints captured by the first camera. The three-dimensional image I_3D is jointly reconstructed using supplementary images from multiple unknown perspectives generated by the generative AI model.

8. The highly realistic 3D image presentation device according to claim 1, characterized in that, The initialization unit is specifically set to i_limit=100.

9. A highly realistic 3D image rendering system, characterized in that, include: First camera, second camera, 3D display device and controller; The controller is connected to the first camera, the second camera, and the three-dimensional display device respectively, and is used to control the first camera, the second camera, and the three-dimensional display device to work together to implement the steps of the highly realistic 3D image presentation method according to any one of claims 1-4.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the highly realistic 3D image rendering method according to any one of claims 1-4.

Images