Computed hologram generation method for multi-depth three-dimensional point cloud display

Abstract

The invention discloses a computer-generated hologram generation method for multi-depth three-dimensional point cloud display, and relates to the technical field of three-dimensional point cloud image processing. The method comprises the following steps: generating a target amplitude, carrying out boundary continuation and multi-plane propagation on a layered point cloud, and constructing the target amplitude with relevance on a reconstruction plane through pixel-level ROI splicing; in the iterative phase optimization stage, the multi-plane amplitude error is used as loss, and the single phase hologram is iteratively optimized through multi-plane stochastic gradient descent (SGD), so that the reconstruction amplitude of each reconstruction plane approaches the target amplitude. According to the method, the amplitude of a multi-plane target is approached by a single field and a complex field at the same time, the superposition of cross-layer complex fields is avoided, and a systematic dark band caused by coherent superposition does not appear at a junction. And the multi-plane consistency loss punishes energy leakage of a non-target area on all planes, so that crosstalk between the planes is inhibited.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional point cloud image processing, and in particular to a computer hologram generation method for multi-depth three-dimensional point cloud display. BACKGROUND

[0002] Three-dimensional point cloud display, as one of the core technologies of three-dimensional visualization, has wide application prospects in virtual reality, medical imaging, industrial detection and other fields. Computer hologram (CGH) generation is a key link to realize three-dimensional point cloud display. Among them, the method based on physical model (PBM) has become the mainstream technology path because it can accurately describe the propagation characteristics of light field.

[0003] The current mainstream CGH generation method is mostly based on the idea of layered point cloud processing. By discretely layering the three-dimensional point cloud according to depth, the holographic information corresponding to each layer is calculated and fused respectively, and finally a single hologram is generated to realize multi-depth three-dimensional display. This kind of method uses propagation operators such as angular spectrum method (ASM) to complete the light field propagation calculation, which meets the basic needs of three-dimensional display to a certain extent.

[0004] The existing layered CGH generation method based on PBM framework faces three key artifact problems in multi-depth three-dimensional point cloud display, which seriously affects the display quality: Interlayer coherent interference: When the complex fields corresponding to multiple depth layers are directly superimposed, a deterministic coherent relationship is formed, resulting in systematic dark bands at the junction of the reconstructed images, which destroys the visual continuity.

[0005] Band-limited ringing: Due to the steep change of the spatial boundary of the point cloud data, Gibbs-type ringing artifacts will be produced in the limited bandwidth propagation process, which appears as alternating stripes of light and dark on the edge of the target area.

[0006] Interlayer crosstalk: The energy of each depth layer is prone to cross-layer leakage during propagation and fusion, resulting in excessive response on non-target planes, which causes mutual interference of image information of different depths and reduces the depth resolution.

[0007] These artifacts are not independent, but have complex coupling relationships. The existing methods are difficult to suppress all three types of artifacts at the same time, resulting in insufficient clarity, fidelity and visual comfort of multi-depth three-dimensional point cloud display. SUMMARY

[0008] The technical problem to be solved by the present application is to provide a computer hologram generation method for multi-depth three-dimensional point cloud display that can avoid the superposition of cross-layer complex fields and suppress the crosstalk between planes.

[0009] To solve the above technical problems, the technical solution adopted by the present application is: a computer hologram generation method for multi-depth three-dimensional point cloud display, comprising the following steps: generating a target amplitude, boundary extension and multi-plane propagation are performed on the layered point cloud, and the target amplitude with correlation is constructed by pixel-level ROI splicing on the reconstruction plane; an iterative phase optimization stage, taking multi-plane amplitude error as loss, iteratively optimizing the single-phase hologram by multi-plane stochastic gradient descent (SGD) to make the reconstruction amplitude of each reconstruction plane approximate to the target amplitude.

[0010] Further, the generating the target amplitude comprises the following steps: layering the input point cloud to obtain point cloud subsets of each layer; extending the boundary of each layer of point cloud and generating an enhanced point set; generating the complex amplitude field of each point cloud layer on the hologram plane.

[0011] Further, the iterative phase optimization comprises the following steps: in the optimization process, the complex field on the hologram plane is obtained; the complex wave field on each target plane is obtained; the phase field is updated after defining the loss function.

[0012] The beneficial effects of the above technical solutions are that the method simultaneously approximates the multi-plane target amplitude with a single complex field, avoiding the superposition of cross-layer complex fields, so that the systematic dark band caused by coherent superposition no longer appears at the junction. The multi-plane consistency loss will punish the energy leakage of non-target regions on all planes, thereby suppressing the crosstalk between planes. BRIEF DESCRIPTION OF DRAWINGS

[0013] The present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0014] Figure 1 the main flowchart of the method according to the first embodiment of the present application; Figure 2 the detailed flowchart of the method according to the first embodiment of the present application; Figure 3 the principle block diagram of the system according to the second embodiment of the present application. DETAILED DESCRIPTION

[0015] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0016] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that the present application can be practiced without the specific details set forth in this description, that the present application can be practiced with other devices, and that the present application can be practiced using other technologies. Therefore, the specific details set forth in the following description should not be taken as limiting the present application.

[0017] Embodiment one: Generally, as Figure 1 shown, the embodiment of the present application discloses a kind of calculation hologram generation method for multi-depth three-dimensional point cloud display, the method comprises the following steps: S1: generating target amplitude, boundary extension and multi-plane propagation are carried out to layered point cloud, and target amplitude with relevance is constructed with pixel-level ROI splicing on reconstruction plane; S2: iterative phase optimization stage, with multi-plane amplitude error as loss, single-phase hologram is iteratively optimized by multi-plane stochastic gradient descent (SGD), so that the reconstruction amplitude of each reconstruction plane is approximated to target amplitude.

[0018] The above steps will be described in detail in combination with specific methods: Three main artifact mechanisms and their coupling relationship under PBM framework: interlayer crosstalk, interlayer coherent interference and band-limited effect. Based on these insights, we propose a coupled multi-plane SGD (CM-SGD) for the generation of CGH for multi-depth three-dimensional point cloud display. CM-SGD adopts a two-stage framework, and its overall process is as shown in Figure 2 .

[0019] The overall design of the algorithm aims to suppress artifacts: for interlayer coherent interference dark band, CM-SGD uses a single complex field to fit multiple plane amplitudes, avoiding the deterministic coherent relationship of direct superposition of multiple complex fields; for band-limited ringing, boundary extension in the target amplitude generation stage migrates the boundary high frequency to outside the ROI, significantly reducing the Gibbs-type ringing in the ROI; for interlayer crosstalk, the multi-plane consistency constraint punishes excessive response on non-target planes during the optimization process, thereby suppressing cross-layer leakage.

[0020] Step S1: Target Amplitude Synthesis, TAS Let the input point cloud be , where a single point includes three-dimensional space coordinates and amplitude.

[0021] Step S1-1: layer the input point cloud to obtain each layer point cloud subset: Layer the point cloud according to the discrete depth set to obtain each layer point cloud subset: ; wherein is the total number of layers.

[0022] Step S1-2, the boundary of each layer is extended and an enhanced point set is generated: To suppress the band-limited ringing caused by the boundary, the boundary of each layer is extended in the grid domain and an enhanced point set is generated.

[0023] Specifically, each layer point cloud subset is projected onto the x-y plane and divided into uniform grid cells at a fixed spatial resolution, denoted as , the grid cell set on layer is denoted as ; let be the grid index corresponding to , where is the grid cell size; For each grid , its intensity is defined as: ; With the support as the seed, based on the intensity field , the boundary is extended on the unoccupied area of using the pre-trained LaMa model to obtain the enhanced grid set , and then the inverse mapping is performed to obtain the enhanced point set .

[0024] LaMa is a deep learning-based image inpainting model with large receptive field and frequency consistency constraints, so its output is smoother and more consistent in boundary structure and amplitude continuation. Intuitively, boundary extension is equivalent to flattening the spatial domain abrupt change outward, thereby shifting the main high frequency to the outside of the ROI; under the propagation of limited bandwidth, high-frequency energy is more dissipated in the form of ringing outside the ROI, thereby reducing the overshoot and banding inside the ROI.

[0025] Step S1-3, generate the complex amplitude field of each point cloud layer on the hologram plane: Let the hologram be located on the plane, use the pre-trained HGN model to generate the complex amplitude of the enhanced point set on the hologram plane; HGN is a deep learning-based PBM hologram generation model, which is equivalent to the exponential term of the Green function kernel in the ideal case.

[0026] Given and the axial diffraction distance , the complex wavefront contribution at the hologram sampling point , the complex amplitude of the ; The angular spectrum method (ASM) is adopted to propagate the to each target reconstruction plane , we have: ; where is the transverse frequency, , ; At this time, The amplitude response at the reconstruction plane is: ; Based on The binary ROI mask is generated on the geometry support of the hologram plane , as a cropping mask shared across planes; On each reconstruction plane , the amplitudes of each layer are pixel-wise cropped and added to construct the target amplitude: .

[0027] Although the occlusion is not explicitly modeled, the strategy of ROI stitching limits the effective support area of each layer at the target layer, avoiding crosstalk within a single In addition, since originates from wave propagation, the target amplitude set has a physical correlation across planes, which will be used as a convergence constraint for cross-plane coupling in the next stage, further suppressing the cross-layer energy leakage.

[0028] Step S2: Iterative Phase Optimization (IPO) The goal of iterative phase optimization is to make the reconstructed amplitude of a single phase-only hologram on each reconstruction plane approximate the target amplitude set generated by step S1 through end-to-end multi-plane wave propagation and automatic differentiation.

[0029] S2-1, during the optimization process, the complex field on the hologram plane is: ; where is the iteration round, is the phase variable of the th round; When​​ initial phase Random or heuristic initial values are acceptable.

[0030] S2-2, get the complex wave field on each target plane: Given a set of target depths , the angular spectrum method (ASM) is used as the propagation operator for scalar waves; in each iteration, first propagate forward to each target plane to get the complex wave field on that plane: ; where is the numerically reconstructed amplitude, is the phase.

[0031] S2-3, define the loss function: The loss function in the optimization process is defined as the amplitude matching loss for multiple planes, using the mean square error (MSE) as the metric: ; where are the horizontal and vertical pixel numbers of the target planes, respectively; this loss simultaneously minimizes the two-norm error between the numerically reconstructed amplitude and the target amplitude on all reconstructed planes, thus being able to handle the energy coupling and constraint conflicts among different planes in the same optimization process.

[0032] S2-4, update the phase field: Based on automatic differentiation, get and accordingly update the phase field using the first-order optimization method: ; where is the learning rate, and after each update, the phase field is mapped back to the basic interval according to 2π to maintain the unit modulus complex exponential representation of the complex field, thus eliminating the multi-value redundancy and improving the numerical stability.

[0033] The method of multi-plane SGD in this application simultaneously approximates the target amplitudes of multiple planes with a single complex field, avoiding the superposition of cross-layer complex fields, thus no longer appearing the systematic dark bands caused by coherent superposition at the junction. The multi-plane consistency loss will punish the energy leakage in non-target regions on all planes, thus suppressing the cross-plane crosstalk.

[0034] In summary, the method described in this application has the following advantages: 1) The method described in this application eliminates the interlayer coherent interference dark bands, improving the visual continuity: The present application simultaneously approximates the target amplitude of multiple depth layers with a single phase hologram through single complex field fitting design in the iterative phase optimization stage, avoiding the coherent effect of direct superposition of multiple complex fields. The generation path of interlayer coherent interference is cut off at the root, making the transition of reconstructed images of each depth layer at the junction natural, completely eliminating systematic dark bands, and significantly improving the visual fluency and realism of multi-depth three-dimensional display.

[0035] 2) Inhibit band-limited ringing artifacts and improve image clarity: The present application introduces a boundary extension strategy in the target amplitude generation stage (TAS): first, through the pre-trained LaMa image repair model, the intensity field of the grid occupied by the point cloud layer is used as a seed to generate an enhanced point set by boundary extension in the unoccupied area. This operation flattens the spatial abrupt change outward, causing the main high-frequency components to migrate to the outside of the region of interest (ROI). In the subsequent limited bandwidth propagation, high-frequency energy is more dissipated in the form of ringing outside the ROI, significantly reducing the overshoot and stripe artifacts in the ROI, making the detail reconstruction of three-dimensional point cloud clearer and improving the display accuracy.

[0036] 3) Reduce interlayer crosstalk and enhance depth resolution: ROI splicing constraint in the target amplitude generation stage: based on the geometric support of the point cloud in the hologram plane, a binary ROI mask is generated, and the amplitudes of each layer propagated to the reconstruction plane are pixel-level cropped and then superimposed, limiting the effective support area of each layer from the target layer, avoiding the crosstalk caused by uncontrolled diffusion of energy within a single layer; Multi-plane consistency loss in the iterative phase optimization stage: the amplitude matching error (mean square error, MSE) of all reconstruction planes is used as the total loss, and the excessive response of non-target planes is simultaneously punished in the optimization process. This cross-plane constraint ensures that the energy of each depth layer is only distributed in the corresponding target plane, effectively suppressing cross-layer energy leakage and significantly enhancing the depth resolution of multi-depth three-dimensional display, making the point cloud information of different depths more distinct.

[0037] 4) Maintain the physical relevance of the target amplitude and improve the stability of the optimization and the fidelity of the reconstruction: In the target amplitude generation stage (TAS), the present application constructs relevance through physical propagation: first, the pre-trained HGN model is used to generate an enhanced point set in the complex amplitude field of the hologram plane, and then the angle spectrum method (ASM) is used to propagate it to each target reconstruction plane; the generated target amplitude set is derived from the real wave propagation process and has cross-plane physical relevance. This relevance serves as a convergence constraint for cross-plane coupling in the iterative optimization stage, improving the convergence speed and numerical stability of the phase optimization, and ensuring the matching degree of the reconstructed amplitude and the real three-dimensional point cloud, ultimately improving the fidelity of multi-depth three-dimensional display.

[0038] 5) Technical practicability and efficiency are considered, and it is suitable for multi-scene application: The core model used in the application is a pre-trained model, which does not need to be retrained for each point cloud input, thereby reducing the calculation cost. Meanwhile, the first-order stochastic gradient descent (SGD) algorithm is used in the iterative phase optimization stage, and automatic differentiation is used to realize efficient parameter updating, thereby considering the requirements of high precision and high efficiency.

[0039] Embodiment two In an exemplary embodiment, the present application also provides a computer system, which can be a server or a terminal, and its internal structure diagram can be shown as Figure 3 The computer device includes a processor, a memory, an input / output interface (I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement the computer hologram generation method for multi-depth three-dimensional point cloud display as described in embodiment one.

[0040] Those skilled in the art can understand that Figure 3 the structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. In an exemplary embodiment, a computer device is provided, which includes a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the steps in each method embodiment described above.

[0041] In an exemplary embodiment, the present application also provides a computer readable storage medium storing a computer program, which is executed by a processor to implement the steps in each method embodiment described above.

[0042] In an exemplary embodiment, the present application also provides a computer program product including a computer program, which is executed by a processor to implement the steps in each method embodiment described above.

[0043] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0044] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments of each method. In the embodiments provided in the present application, any reference to memory, database or other medium can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc.

[0045] The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a data processing logic of a programmable logic device, etc., without being limited thereto.

[0046] The technical features of the above embodiments can be combined arbitrarily. In order to make the description simple, not all possible combinations of the technical features in the above embodiments are described, but as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0047] The principles and implementation manners of the present application are described herein by using specific examples, and the above examples are only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manners and application ranges will have changes. In conclusion, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A computational hologram generation method for multi-depth 3D point cloud display, characterized in that... Includes the following steps: Generate target amplitude, extend the boundary of the layered point cloud and propagate it in multiple planes, and construct the target amplitude with correlation by stitching together pixel-level ROIs on the reconstruction plane; In the iterative phase optimization stage, the multi-plane amplitude error is used as the loss, and the single phase hologram is iteratively optimized by multi-plane stochastic gradient descent (SGD) to make the reconstructed amplitude of each reconstruction plane approach the target amplitude.

2. The computational hologram generation method for multi-depth 3D point cloud display as described in claim 1, characterized in that, The generation of the target amplitude includes the following steps: The input point cloud is layered to obtain subsets of the point cloud in each layer; The boundary of each point cloud layer is extended and an enhanced point set is generated; Generate the complex amplitude field of each point cloud layer on the hologram plane.

3. The computational hologram generation method for multi-depth 3D point cloud display as described in claim 2, characterized in that, The method for layering the input point cloud to obtain subsets of the point cloud in each layer includes the following steps: Let the input point cloud be , of which a single point Includes three-dimensional spatial coordinates and amplitude; Point cloud By discrete depth set By dividing the data into layers, we obtain subsets of the point cloud at each layer: ； in This represents the total number of floors.

4. The computational hologram generation method for multi-depth 3D point cloud display as described in claim 3, characterized in that, The method for extending the boundary of each point cloud layer and generating an enhanced point set includes the following steps: Subsets of each layer of point cloud Projected onto the xy plane and divided into uniform grid cells with a fixed spatial resolution, let the set of grid cells be denoted as . ,layer The set of grid cells on is ;make Corresponding grid index ,in This refers to the grid cell size; For each grid Its strength is defined as: ； To occupy support As a seed, based on the intensity field ,exist The boundary is extended using a pre-trained LaMa model on the unoccupied regions to obtain an enhanced mesh set. Then, an inverse mapping is performed to obtain the enhanced point set with the boundary expansion. .

5. A computational hologram generation method for multi-depth 3D point cloud display as described in claim 4, characterized in that, The method for generating the complex amplitude field of each point cloud layer on the hologram plane includes the following steps: Suppose the hologram is located at Plane, using a pre-trained HGN model to generate augmented point sets. Complex amplitude on the holographic plane; Given With axial diffraction distance Rapidly generate holographic sampling points using HGN Complex wavefront contribution Then the first The complex amplitude of the layer is: ； Using the angle spectrum method (ASM) to Propagate to each target reconstruction plane ,get: ； in, For transverse frequency, , ; at this time, The amplitude response at the reconstructed plane is: ； based on A binary ROI mask is generated using geometric support on the holographic plane. , serving as a clipping mask shared across planes; In each reconstruction plane Above, the amplitudes of each layer are cropped at the pixel level and then summed to construct the target amplitude: 。 6. The computational hologram generation method for multi-depth 3D point cloud display as described in claim 1, characterized in that, Iterative phase optimization includes the following steps: During the optimization process, the complex field on the hologram plane is obtained; The complex wave field on each target plane is obtained; The phase field is updated after the loss function is defined.

7. The computational hologram generation method for multi-depth 3D point cloud display as described in claim 6, characterized in that, The method for obtaining the complex field on the hologram plane includes the following steps: During the optimization process, the complex field on the hologram plane is: ； in, For iteration rounds, For the first The phase variable of the wheel; when At that time, the initial phase Random or heuristic initial values ​​can be used.

8. The computational hologram generation method for multi-depth 3D point cloud display as described in claim 7, characterized in that, The method for obtaining the complex wave field on each target plane includes the following steps: Given a set of target depths Angle spectral method (ASM) is used as the propagation operator for scalar waves; in each iteration, firstly... Forward propagation to each target plane The complex wave field on this plane is obtained: ； in The amplitude of the numerical reconstruction. For phase.

9. The computational hologram generation method for multi-depth 3D point cloud display as described in claim 8, characterized in that: The loss function in the optimization process is defined as the multi-plane amplitude matching loss, measured by mean squared error (MSE). ； in, These represent the number of horizontal and vertical pixels on the target plane, respectively. Based on automatic differentiation Based on this, the phase field is updated using a first-order optimization method: ； in This is the learning rate.

Images