A naked-eye 3D augmented reality interactive display system

Abstract

This invention discloses a naked-eye 3D augmented reality interactive display system, relating to the field of naked-eye 3D display and real-world interaction technology. The invention acquires and fuses user limb depth images through a multi-source depth data acquisition module, completes the mapping of depth and parallax and viewpoint correction through a parallax physical depth mapping module, divides regions to determine occlusion types through an occlusion relationship determination module, generates a dynamically smooth occlusion layer through an adaptive occlusion layer generation module, and displays the occlusion effect fusion display module by superimposing the occlusion layer with multi-viewpoint virtual images. This effectively solves the problems of inaccurate occlusion relationships between user limbs and virtual objects and visual abruptness in naked-eye 3D interaction, significantly improving the spatial consistency and visual smoothness of the interaction, and achieving a realistic and natural naked-eye 3D augmented reality interactive experience.

Description

Technical Field

[0001] This invention relates to the field of naked-eye 3D display and reality interaction technology, specifically to a naked-eye 3D augmented reality interactive display system. Background Technology

[0002] Naked-eye 3D display technology is a display technology that allows users to view stereoscopic images without wearing any auxiliary devices. It projects images from different viewpoints onto the user's eyes through optical structures such as lenticular lenses, creating a stereoscopic perception. Due to its natural experience and suitability for public settings, it has been widely used in museums, commercial displays, and other fields. Augmented reality technology, on the other hand, overlays virtual information onto the real environment through spatial positioning and real-time rendering. The combination of the two can achieve a device-free immersive experience, with broad application prospects. However, existing technologies still have many problems:

[0003] When a user’s limbs (such as hands, arms, etc.) enter the naked-eye 3D display area and overlap with virtual objects in space, the system cannot correctly determine the spatial occlusion relationship between the user’s limbs and virtual objects, resulting in visual errors such as virtual objects penetrating the display of the user’s limbs or the user’s limbs abnormally occluding virtual objects.

[0004] In existing technologies, some solutions attempt to remove the user's limb area from the display screen using simple image segmentation methods. However, this method cannot achieve accurate depth-level occlusion determination and will cause unnatural "holes" in the virtual object in the user's limb area. Other solutions attempt to determine occlusion by setting a fixed depth threshold. However, since the parallax depth of virtual objects in naked-eye 3D display is dynamically changing, the fixed threshold method cannot adapt to various interactive scenarios.

[0005] Therefore, how to determine and display the correct spatial occlusion relationship between the user's limbs and virtual objects is a technical problem that urgently needs to be solved in the field of naked-eye 3D augmented reality interactive display technology.

[0006] To address the aforementioned shortcomings, a technical solution is provided. Summary of the Invention

[0007] The purpose of this invention is to solve the problem of how to determine and display the correct spatial occlusion relationship between the user's limbs and virtual objects, and to propose a naked-eye 3D augmented reality interactive display system.

[0008] The objective of this invention can be achieved through the following technical solutions:

[0009] A naked-eye 3D augmented reality interactive display system includes:

[0010] Multi-source depth data acquisition module: Infrared structured light depth sensors are deployed on the upper and lower bezels of the naked-eye 3D display device to acquire, denoise and fuse the original depth images, correct the depth values ​​of edge occlusion areas, and generate time-aligned physical depth images of user limbs and parallax depth images of virtual objects.

[0011] Parallax physical depth mapping module: acquires optical imaging parameters, constructs a positive and inverse mapping relationship between depth and parallax, corrects the viewing angle by combining eye-tracking data, and completes coordinate transformation and image registration through a spatial calibration matrix;

[0012] Occlusion relationship determination module: Divide the depth determination sub-region, determine the occlusion type and assign occlusion mark value, smooth the boundary of adjacent regions and generate an occlusion relationship mark matrix;

[0013] Adaptive mask layer generation module: Filters the effective occlusion area and generates a dynamic mask layer based on morphological processing, transparency calculation, and time-series smoothing;

[0014] Occlusion effect fusion display module: It overlays and synthesizes the mask layer with multi-viewpoint virtual images, and sends them to the display device based on a preset frame rate to achieve real-time naked-eye 3D interactive display.

[0015] As a further improvement of the present invention, the specific operation steps of the multi-source depth data acquisition module are as follows:

[0016] First and second infrared structured light depth sensors are deployed on the upper and lower bezels of the display screen of the naked-eye 3D display device. The optical axes of the first and second sensors intersect at a preset interactive distance in front of the center of the display area, forming a stereoscopic depth acquisition field of view covering the interactive area.

[0017] Two sets of original depth images are acquired separately and bilateral filtering is performed to denoise them. Then, pixel-level fusion is performed to obtain the joint depth value.

[0018] When the absolute value of the difference between the denoised depth values ​​of the first and second infrared structured light depth sensors exceeds the preset depth difference threshold, the corresponding pixel is determined to be in the edge occlusion area, and the minimum value of the denoised depth values ​​of the first and second infrared structured light depth sensors is taken as the corrected final physical depth value.

[0019] Based on the final physical depth values ​​of all pixels, a time-aligned physical depth image of the user's limbs is generated frame by frame. The parallax depth image of the virtual object is read from the depth buffer of the naked-eye 3D rendering engine. The falling edge of the vertical synchronization signal output by the naked-eye 3D rendering engine is used as the trigger source to trigger the sensor to synchronously acquire data and generate the time-aligned physical depth image and parallax depth image.

[0020] As a further improvement of the present invention, the specific operation steps of the parallax physical depth mapping module are as follows:

[0021] Obtain the optical imaging parameters of the naked-eye 3D display device, including the lenticular grid spacing, optimal viewing distance, standard human eye spacing, and display sub-pixel spacing; construct the positive mapping relationship between parallax and equivalent out-of-screen depth, and the inverse mapping relationship between physical depth and equivalent parallax.

[0022] The real-time spatial coordinates of the user's eyes are obtained by an eye-tracking unit deployed on the display device. Combined with the reference coordinates of the center of the display area, the user's viewing offset in the horizontal and vertical directions is calculated.

[0023] Based on preset horizontal and vertical parallax correction coefficients, combined with viewing offset and optimal viewing distance, the equivalent parallax depth value is corrected from the perspective.

[0024] The spatial calibration of the sensor and display device is completed in advance using a calibration board to obtain a spatial calibration matrix containing translation and rotation parameters;

[0025] Based on the spatial calibration matrix, coordinate transformation is performed on the corrected user limb equivalent parallax depth image, so that each pixel of the user limb equivalent parallax depth image corresponds one-to-one with the physical pixel of the display screen and the pixel coordinate of the virtual object parallax depth image, and is registered to the same image coordinate system, thus completing the unified mapping of parallax and physical depth.

[0026] As a further improvement of the present invention, the specific operation steps of the occlusion relationship determination module are as follows:

[0027] Under a unified image coordinate system, the depth image covering the interactive area is divided into... The pixel has several depth determination sub-regions, and the pixel coordinates of the sub-regions correspond one-to-one with the pixel coordinates of the corrected user limb equivalent parallax depth image and the virtual object parallax depth image.

[0028] Traverse all pixels in each sub-region, extract the equivalent disparity depth value of the user's limb after correction and the disparity depth value of the virtual object for each pixel, and calculate the arithmetic mean of each value to obtain the mean disparity depth of the user's limb and the virtual object. Combined with the preset depth similarity judgment threshold, assign occlusion labels such as foreground occlusion, background penetration or coplanar transition to the depth judgment sub-region.

[0029] Based on the labels, occlusion marker values ​​are assigned to pixels. For adjacent sub-regions with different labels, the depth gradient direction is extracted to redefine the occlusion boundary. Bilinear interpolation is used to smooth the marker values ​​of two pixels on each side of the boundary. The pixel occlusion marker values ​​of all depth-determined sub-regions are counted to generate an occlusion relationship marker matrix with the same resolution as the interactive region image.

[0030] As a further improvement of the present invention, the specific operation steps of the adaptive mask layer generation module are as follows:

[0031] The effective occlusion area of ​​the user's limbs is obtained by filtering pixels with occlusion marker values ​​greater than 0 on a pixel-by-pixel basis using the occlusion relationship marker matrix.

[0032] Perform morphological opening and closing operations on the effective occlusion region; extract the outer contour boundary of the effective occlusion region to generate a boundary pixel set;

[0033] The boundary weight value is obtained based on the minimum Euclidean distance from the pixel in the effective occlusion area to all pixels in the boundary pixel set. The transparency channel value is then calculated by combining the occlusion marker value and the occlusion type difference.

[0034] Acquire real color and texture images of the user's limbs that are time-aligned with depth data, and fuse them with the transparency channel values ​​to generate an initial four-channel base mask layer;

[0035] The Lucas-Kanade optical flow method is used to estimate the two-dimensional motion vector field of the user's limbs between adjacent frames, and a prediction masking layer is generated.

[0036] The boundary position deviation is obtained based on the boundary pixel coordinates of the base mask layer and the predicted mask layer.

[0037] The masking layer is either subjected to weighted average temporal smoothing based on the boundary position deviation, or the current frame masking layer is directly output.

[0038] Temporal smoothing is performed on the alpha channel values ​​of all pixels; the temporally smoothed alpha channel values ​​are then re-fused with the RGB channels of the initial four-channel base mask layer image to generate the final four-channel dynamic mask layer image.

[0039] As a further improvement of the present invention, the specific operation steps of the occlusion effect fusion display module are as follows:

[0040] Obtain the number of viewpoints of the naked-eye 3D display device and the horizontal viewing angle offset corresponding to each viewpoint. The number of viewpoints is a positive integer not less than 2 and is consistent with the number of viewpoints divided by the cylindrical lens of the device.

[0041] Obtain a corresponding number of multi-viewpoint virtual object image sequences from the naked-eye 3D rendering engine, with each image corresponding to a specific viewing angle;

[0042] For each viewpoint, calculate the horizontal projection offset based on the corresponding horizontal viewing angle offset, and perform a horizontal translation transformation on the dynamic masking layer to generate a viewpoint-specific masking layer;

[0043] The exclusive masking layer is superimposed on the virtual object image of the corresponding viewpoint. Pixels with an opacity channel value greater than 0 are merged with the pixel values ​​of the user's real body color and texture image, while pixels with an opacity channel value equal to 0 retain the original pixel values ​​of the virtual object.

[0044] After all viewpoint images are fused, pixel-level synthesis is performed based on the sub-pixel interlacing arrangement rules of the naked-eye 3D display device to generate naked-eye 3D interlaced output images, which are then sent to the naked-eye 3D display device at a preset frame rate to achieve real-time naked-eye 3D visual presentation of the correct spatial occlusion relationship between the user's limbs and virtual objects.

[0045] Compared with the prior art, the beneficial effects of the present invention are:

[0046] This invention utilizes dual infrared structured light depth sensors to collect and fuse corrected physical depth data of the user's limbs, constructing a positive and inverse mapping relationship between parallax and physical depth. It combines eye-tracking data to correct the viewing angle and complete coordinate registration, dividing the data into depth-determining sub-regions and determining occlusion relationships. Based on morphological processing and temporal smoothing, a dynamic masking layer is generated and fused with multi-viewpoint virtual images before being displayed on a display device. This significantly improves the accuracy of determining the spatial occlusion relationship between the user's limbs and virtual objects, avoiding visual errors such as virtual object penetration and abnormal occlusion. Simultaneously, it optimizes the visual smoothness and spatial consistency of the interaction process, eliminates the abruptness of occlusion switching, and enhances the immersiveness and interactive realism of naked-eye 3D augmented reality interaction. Attached Figure Description

[0047] Figure 1 This is the overall system block diagram of the present invention. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] Example:

[0050] like Figure 1 As shown, a naked-eye 3D augmented reality interactive display system includes a multi-source depth data acquisition module, a parallax physical depth mapping module, an occlusion relationship determination module, an adaptive mask layer generation module, and an occlusion effect fusion display module.

[0051] Multi-source depth data acquisition module:

[0052] A first infrared structured light depth sensor is deployed on the upper bezel of the display screen of the naked-eye 3D display device, and a second infrared structured light depth sensor is deployed on the lower bezel of the display screen. The optical axes of the two depth sensors intersect at a preset interaction distance in front of the center of the display area, forming a stereoscopic depth acquisition field of view covering the interaction area.

[0053] The first original depth image of the first infrared structured light depth sensor and the second original depth image of the second infrared structured light depth sensor are acquired respectively, and bilateral filtering denoising processing is performed on the first original depth image and the second original depth image respectively.

[0054] Pixel-level fusion is performed on the two denoised depth images. For each pixel location, the formula is used. The joint depth value at the current pixel position is calculated, where, This is the denoised depth value collected by the first infrared structured light depth sensor. The denoised depth value is obtained from the second infrared structured light depth sensor;

[0055] when When determining that the current pixel is within the edge-occluded area of ​​the user's limb contour, the formula is used. The final physical depth value after correction for the current pixel is obtained, where, The depth difference threshold is indicated by a preset interaction distance calibration based on the interaction area. Describes the minimum value function;

[0056] After calculating the joint depth value of all pixels and correcting the depth value of edge occlusion areas, a time-aligned physical depth image of the user's limbs is generated frame by frame based on the final physical depth value of all pixels.

[0057] The parallax depth image of the virtual object is read from the depth buffer of the naked-eye 3D rendering engine. The depth value of each pixel in the parallax depth image represents the parallax offset of the virtual object's surface relative to the display plane.

[0058] Using the vertical synchronization signal output by the naked-eye 3D rendering engine as the trigger source, two infrared structured light depth sensors are triggered to synchronously acquire data at the falling edge of the vertical synchronization signal, generating a time-aligned physical depth image and a parallax depth image.

[0059] Parallax Physical Depth Mapping Module:

[0060] Obtain optical imaging parameters of the naked-eye 3D display device, including the lenticular grid spacing. Optimal viewing distance Standard human eye spacing and the subpixel pitch of the display screen ;

[0061] Establish a positive mapping relationship between parallax and equivalent off-screen depth using the formula. Parallax in naked-eye 3D rendering engines Converted to the equivalent physical out-of-screen depth relative to the display plane; where parallax is... When the value is positive, it indicates that the virtual object presents a screen effect, and the parallax is... When the value is negative, it indicates that the virtual object appears to be displayed on the screen.

[0062] Based on the forward mapping relationship, an inverse mapping relationship is constructed between physical depth and equivalent disparity, using the formula... Convert the user's final physical depth value to an equivalent parallax depth value. ;

[0063] The real-time spatial coordinates of the user's eyes are obtained by an eye-tracking unit deployed on a glasses-free 3D display device, thus determining the center position of the user's eyes. Combined with the center coordinates of the display area, the user's viewing offset relative to the center of the display area is calculated using the following formula:

[0064] , ;

[0065] in, These represent the horizontal and vertical viewing offset, respectively. This represents the real-time spatial coordinates of the center of the user's eyes. The reference coordinates represent the center of the naked-eye 3D display area;

[0066] Viewpoint correction is performed on the equivalent parallax value based on the viewing offset, using the formula... The corrected equivalent disparity depth value is obtained through calculation, where, This represents the horizontal parallax correction factor. This represents the vertical parallax correction factor. and The value is pre-calibrated based on the viewpoint distribution characteristics of the naked-eye 3D display device;

[0067] The spatial calibration of the sensor and the display device is completed in advance using a calibration board to obtain a spatial calibration matrix. The spatial calibration matrix contains translation and rotation parameters, which can convert the depth image coordinates acquired by the sensor into image coordinates of the display device.

[0068] Based on the spatial calibration matrix, coordinate transformation is performed on the corrected user limb equivalent parallax depth image, so that each pixel of the user limb equivalent parallax depth image corresponds one-to-one with the physical pixel of the display screen and the pixel coordinate of the virtual object parallax depth image; the corrected user limb equivalent parallax depth image and the virtual object parallax depth image are registered to the same image coordinate system of the naked-eye 3D display device, thus completing the unified mapping of parallax physical depth.

[0069] Occlusion relationship determination module:

[0070] Under the unified image coordinate system of the naked-eye 3D display area device completed by S2, the depth image covering the entire interactive area is based on The pixel size is divided into several depth determination sub-regions. Each depth determination sub-region serves as an independent preliminary occlusion type determination unit. The pixel coordinates of the sub-region correspond one-to-one with the pixel coordinates of the corrected user limb equivalent disparity depth image and the virtual object disparity depth image.

[0071] For each depth determination sub-region after division, traverse all pixels within the sub-region, extract the equivalent disparity depth value after user limb correction and the virtual object disparity depth value of each pixel, and perform arithmetic mean calculation respectively to obtain the mean value of the equivalent disparity depth of user limb correction and the mean value of the virtual object disparity depth of the current depth determination sub-region.

[0072] The mean value of the equivalent parallax depth region after user limb correction in each depth determination sub-region is numerically compared with the mean value of the virtual object parallax depth region. Combined with a pre-defined depth similarity determination threshold, a unique occlusion label type is assigned to each depth determination sub-region. The specific determination rules and label definitions are as follows:

[0073] When the mean value of the equivalent parallax depth region after user limb correction is less than the mean value of the parallax depth region of virtual object, it is determined that the overall spatial position of the user limb in the current depth sub-region is in front of the virtual object, thus occluding the virtual object, and the occlusion type label of the current depth determination sub-region is marked as foreground occlusion.

[0074] When the mean value of the equivalent parallax depth region after user limb correction is greater than the mean value of the parallax depth region of virtual object, it is determined that the overall spatial position of the user limb in the current depth sub-region is behind the virtual object and cannot occlude the virtual object. The occlusion type label of the current depth sub-region is marked as background penetration.

[0075] When the absolute value of the difference between the mean value of the equivalent disparity depth region after user limb correction and the mean value of the virtual object disparity depth region is less than the depth similarity judgment threshold, it is determined that the user limb and the virtual object are in a coplanar spatial position within the current depth sub-region, and there is no clear front-to-back occlusion relationship between the user limb and the virtual object. The occlusion type label of the current depth judgment sub-region is marked as coplanar transition.

[0076] Based on the occlusion type label of each depth-determining sub-region, an occlusion marker value is assigned to each pixel within each depth-determining sub-region. The specific assignment rule is as follows:

[0077] For the depth determination sub-region with the occlusion type label as foreground occlusion: traverse all pixels in the region. If the current pixel is within the user's limb outline, set the occlusion flag value of the current pixel to 1; if the current pixel is outside the user's limb outline, set the occlusion flag value of the current pixel to 0.

[0078] For depth determination sub-regions with occlusion type label "background penetration": there is no need to distinguish pixel positions; the occlusion mark value of all pixels in the region is uniformly set to 0.

[0079] For depth-determining sub-regions with occlusion type labeled as coplanar transition: using the formula Calculate the pixels Occlusion marker value ,in, Represents pixels The disparity depth value of the virtual object. Represents pixels The equivalent parallax depth value after user limb correction. This indicates the threshold for determining similar depths;

[0080] For adjacent depth determination sub-regions, when the occlusion type labels of adjacent regions are inconsistent, the depth gradient direction is extracted along the boundary line of the two regions, the occlusion boundary position is redefined based on the depth gradient direction, and the occlusion mark value within a range of 2 pixels on both sides of the boundary is smoothed by bilinear interpolation.

[0081] The pixel occlusion marker values ​​of all depth-determining sub-regions are statistically analyzed to generate an occlusion relationship marker matrix with the same resolution as the interactive region image.

[0082] Adaptive mask layer generation module:

[0083] Based on the occlusion relationship marking matrix, pixels with occlusion marking values ​​greater than 0 are selected pixel by pixel and marked as the effective occlusion area of ​​the user's limb;

[0084] Perform morphological opening and closing operations sequentially on the effective occlusion region;

[0085] Morphological opening operation: A 3×3 circular structural element is used to perform erosion and then expansion on the effective occlusion area to eliminate isolated noise areas, preserve the outline of the user's limbs, and avoid invalid spots in the mask layer caused by minor noise.

[0086] Morphological closing operation: Based on morphological opening operation, a 5×5 circular structural element is used to perform expansion and then erosion on the effective occlusion area to fill the hole area inside the user's limb contour and ensure the integrity of the effective occlusion area.

[0087] For the occlusion effective area after morphological processing, extract the outer contour boundary to generate the boundary pixel set of the user's limb;

[0088] Through formula The boundary weight values ​​are calculated, where, Indicates pixels within the effective occlusion area The minimum Euclidean distance to all boundary pixels in the boundary pixel set. The boundary radius parameter is pre-calibrated manually. Represents the natural exponential function;

[0089] The transparency channel value is calculated based on the occlusion marker value and boundary weight value, combined with the occlusion type differentiation. The specific calculation formula is as follows:

[0090] Foreground occlusion of label pixels: ;

[0091] Coplanar transition label pixels: ;

[0092] The system acquires real color and texture images of the user's limbs that are time-aligned with the depth data from the RGB image acquisition channel of the infrared structured light depth sensor, and fuses them with the transparency channel values ​​to generate an initial four-channel basic mask layer image.

[0093] The Lucas-Kanade optical flow method is used to estimate the two-dimensional motion vector field of the user's limbs between adjacent frames. Based on the two-dimensional motion vector field, the base mask layer of the previous frame is projected onto the predicted position of the current frame to generate the prediction mask layer.

[0094] Through formula The deviation of the boundary position between the actual masking layer and the predicted masking layer is calculated; where, Indicates the boundary pixel coordinates of the base mask layer in the current frame. This indicates the predicted boundary pixel coordinates of the mask layer. Indicates the total number of boundary pixels. Represents the set of boundary pixels;

[0095] If the boundary position deviation is less than the preset threshold, it indicates that the user's limbs are moving smoothly. A weighted average temporal smoothing is then performed on the boundary position and transparency channel value of the mask layer.

[0096] If the boundary position deviation is greater than the preset threshold, it indicates that the user's limbs are moving quickly or entering or leaving the interaction area, and the current frame's mask layer is directly used for output;

[0097] Temporal smoothing is performed on the alpha channel values ​​of all pixels using the formula. Output based on time-smoothed pixels transparency channel value ,in, Represents the filter coefficients. Represents pixels The opacity channel value of the base mask layer in the current frame. Represents pixels The opacity channel value of the base mask layer in the previous frame;

[0098] The occlusion relationship marker matrix of the last 5 frames is stored in a preset circular buffer area, and the occlusion type label of each pixel is monitored in real time. When the label is detected to switch from foreground occlusion to background penetration, a linear gradient transition of 3 consecutive frames is performed on the transparency channel value of the current pixel to avoid visual abruptness caused by instantaneous switching of occlusion or penetration effects.

[0099] The opacity channel values, after time-smooth processing, are re-fused with the RGB channels of the initial four-channel base mask layer image to generate the final four-channel dynamic mask layer image.

[0100] Occlusion effect blending display module:

[0101] Obtain the number of viewpoints of the naked-eye 3D display device and the horizontal viewing angle offset sequence corresponding to each viewpoint. The number of viewpoints is a positive integer greater than or equal to 2 and is consistent with the number of viewpoints divided by the cylindrical lens of the naked-eye 3D display device.

[0102] The naked-eye 3D rendering engine obtains a sequence of virtual object images from multiple viewpoints with corresponding viewpoint number parameters, and each viewpoint image corresponds to a specific viewing angle;

[0103] The specific viewing angle is the horizontal viewing angle of the physical viewpoint pre-calibrated by the naked-eye 3D display device based on the cylindrical lens optical structure;

[0104] For the Virtual object images from each viewpoint, based on the horizontal viewpoint offset corresponding to the current viewpoint. Horizontal viewing angle offset refers to the angular offset in the horizontal direction relative to the reference viewing angle directly in front of the device, characterizing the physical viewing angle difference between different viewpoints divided by the cylindrical lens of the device itself; through the formula... The horizontal projection offset is calculated.

[0105] Perform a horizontal translation transformation on the dynamic masking layer based on the horizontal projection offset to generate the first... Each viewpoint has its own viewpoint-specific masking layer;

[0106] Overlay the viewpoint-specific mask layer onto the first... On a virtual object image from a given viewpoint, for pixels with an alpha channel value greater than 0 in the viewpoint-specific mask layer, the formula is used to... Calculations are performed to obtain the first Pixels from each viewpoint The merged pixel values, where, Indicates the first Pixels in the viewpoint-specific mask layer corresponding to each viewpoint The transparency channel value, Represents pixels The pixel values ​​of the real color texture image of the user's limbs. Indicates the first Pixels from each viewpoint The original image pixel values ​​of the virtual object;

[0107] For pixels with an alpha channel value of 0, the original pixel values ​​of the virtual object image at the viewpoint are preserved.

[0108] After completing the image fusion of all viewpoints, a fused virtual object image sequence is obtained. Each image achieves the correct spatial occlusion between the user's limbs and the virtual object from the corresponding viewpoint.

[0109] The fused multi-view image sequence is then pixel-level synthesized based on the sub-pixel interlacing arrangement rules of the naked-eye 3D display device to generate the final naked-eye 3D interlaced output image.

[0110] Based on the synthesized naked-eye 3D interlaced output image, the naked-eye 3D interlaced output image is sent to the naked-eye 3D display device for display at a preset frame rate, so as to realize the real-time naked-eye 3D visual presentation of the correct spatial occlusion relationship between the user's limbs and virtual objects.

[0111] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A naked-eye 3D augmented reality interactive display system, characterized in that, include: Multi-source depth data acquisition module: Infrared structured light depth sensors are deployed on the upper and lower bezels of the naked-eye 3D display device to acquire, denoise and fuse the original depth images, correct the depth values ​​of edge occlusion areas, and generate time-aligned physical depth images of user limbs and parallax depth images of virtual objects. Parallax physical depth mapping module: acquires optical imaging parameters, constructs a positive and inverse mapping relationship between depth and parallax, corrects the viewing angle by combining eye-tracking data, and completes coordinate transformation and image registration through a spatial calibration matrix; Occlusion relationship determination module: Divide the depth determination sub-region, determine the occlusion type and assign occlusion mark value, smooth the boundary of adjacent regions and generate an occlusion relationship mark matrix; Adaptive mask layer generation module: Filters the effective occlusion area and generates a dynamic mask layer based on morphological processing, transparency calculation, and time-series smoothing; Occlusion effect fusion display module: It overlays and synthesizes the mask layer with multi-viewpoint virtual images, and sends them to the display device based on a preset frame rate to achieve real-time naked-eye 3D interactive display.

2. The naked-eye 3D augmented reality interactive display system according to claim 1, characterized in that, The specific operation steps of the multi-source depth data acquisition module are as follows: First and second infrared structured light depth sensors are deployed on the upper and lower bezels of the display screen of the naked-eye 3D display device. The optical axes of the first and second sensors intersect at a preset interactive distance in front of the center of the display area, forming a stereoscopic depth acquisition field of view covering the interactive area. Two sets of original depth images are acquired separately and bilateral filtering is performed to denoise them. Then, pixel-level fusion is performed to obtain the joint depth value. When the absolute value of the difference between the denoised depth values ​​of the first and second infrared structured light depth sensors exceeds the preset depth difference threshold, the corresponding pixel is determined to be in the edge occlusion area, and the minimum value of the denoised depth values ​​of the first and second infrared structured light depth sensors is taken as the corrected final physical depth value. Based on the final physical depth values ​​of all pixels, a time-aligned physical depth image of the user's limbs is generated frame by frame. The parallax depth image of the virtual object is read from the depth buffer of the naked-eye 3D rendering engine. The falling edge of the vertical synchronization signal output by the naked-eye 3D rendering engine is used as the trigger source to trigger the sensor to synchronously acquire data and generate the time-aligned physical depth image and parallax depth image.

3. The naked-eye 3D augmented reality interactive display system according to claim 1, characterized in that, The specific operation steps of the parallax physical depth mapping module are as follows: Obtain the optical imaging parameters of the naked-eye 3D display device, including the lenticular grid spacing, optimal viewing distance, standard human eye spacing, and display sub-pixel spacing; construct the positive mapping relationship between parallax and equivalent out-of-screen depth, and the inverse mapping relationship between physical depth and equivalent parallax. The real-time spatial coordinates of the user's eyes are obtained by an eye-tracking unit deployed on the display device. Combined with the reference coordinates of the center of the display area, the user's viewing offset in the horizontal and vertical directions is calculated. Based on preset horizontal and vertical parallax correction coefficients, combined with viewing offset and optimal viewing distance, the equivalent parallax depth value is corrected from the perspective. The spatial calibration of the sensor and display device is completed in advance using a calibration board to obtain a spatial calibration matrix containing translation and rotation parameters; Based on the spatial calibration matrix, coordinate transformation is performed on the corrected user limb equivalent parallax depth image, so that each pixel of the user limb equivalent parallax depth image corresponds one-to-one with the physical pixel of the display screen and the pixel coordinate of the virtual object parallax depth image, and is registered to the same image coordinate system, thus completing the unified mapping of parallax and physical depth.

4. The naked-eye 3D augmented reality interactive display system according to claim 1, characterized in that, The specific operation steps of the occlusion relationship determination module are as follows: Under a unified image coordinate system, the depth image covering the interactive area is divided into... The pixel has several depth determination sub-regions, and the pixel coordinates of the sub-regions correspond one-to-one with the pixel coordinates of the corrected user limb equivalent parallax depth image and the virtual object parallax depth image. Traverse all pixels in each sub-region, extract the equivalent disparity depth value of the user's limb after correction and the disparity depth value of the virtual object for each pixel, and calculate the arithmetic mean of each value to obtain the mean disparity depth of the user's limb and the virtual object. Combined with the preset depth similarity judgment threshold, assign occlusion labels such as foreground occlusion, background penetration or coplanar transition to the depth judgment sub-region. Based on the labels, occlusion marker values ​​are assigned to pixels. For adjacent sub-regions with different labels, the depth gradient direction is extracted to redefine the occlusion boundary. Bilinear interpolation is used to smooth the marker values ​​of two pixels on each side of the boundary. The pixel occlusion marker values ​​of all depth-determined sub-regions are counted to generate an occlusion relationship marker matrix with the same resolution as the interactive region image.

5. The naked-eye 3D augmented reality interactive display system according to claim 1, characterized in that, The specific operation steps of the adaptive mask layer generation module are as follows: The effective occlusion area of ​​the user's limbs is obtained by filtering pixels with occlusion marker values ​​greater than 0 on a pixel-by-pixel basis using the occlusion relationship marker matrix. Perform morphological opening and closing operations on the effective occlusion region; Extract the outer contour boundary of the effective occlusion region to generate a boundary pixel set; The boundary weight value is obtained based on the minimum Euclidean distance from the pixel in the effective occlusion area to all pixels in the boundary pixel set. The transparency channel value is then calculated by combining the occlusion marker value and the occlusion type difference. Acquire real color and texture images of the user's limbs that are time-aligned with depth data, and fuse them with the transparency channel values ​​to generate an initial four-channel base mask layer; The Lucas-Kanade optical flow method is used to estimate the two-dimensional motion vector field of the user's limbs between adjacent frames, and a prediction masking layer is generated. The boundary position deviation is obtained based on the boundary pixel coordinates of the base mask layer and the predicted mask layer. The masking layer is either subjected to weighted average temporal smoothing based on the boundary position deviation, or the current frame masking layer is directly output. Temporal smoothing is performed on the alpha channel values ​​of all pixels; the temporally smoothed alpha channel values ​​are then re-fused with the RGB channels of the initial four-channel base mask layer image to generate the final four-channel dynamic mask layer image.

6. The naked-eye 3D augmented reality interactive display system according to claim 1, characterized in that, The specific operation steps of the occlusion effect fusion display module are as follows: Obtain the number of viewpoints of the naked-eye 3D display device and the horizontal viewing angle offset corresponding to each viewpoint. The number of viewpoints is a positive integer not less than 2 and is consistent with the number of viewpoints divided by the cylindrical lens of the device. The naked-eye 3D rendering engine obtains a sequence of virtual object images from multiple viewpoints that are consistent with the number of viewpoints divided by the cylindrical lens of the naked-eye 3D display device. Each viewpoint image corresponds to the horizontal viewing angle of the physical viewpoint pre-calibrated by the naked-eye 3D display device based on the cylindrical lens optical structure. For each viewpoint, calculate the horizontal projection offset based on the corresponding horizontal viewing angle offset, and perform a horizontal translation transformation on the dynamic masking layer to generate a viewpoint-specific masking layer; The exclusive mask layer is superimposed on the virtual object image of the corresponding viewpoint. Pixels with an alpha channel value greater than 0 are merged with the pixel values ​​of the user's real body color and texture image, while pixels with an alpha channel value equal to 0 retain the original pixel values ​​of the virtual object. After all viewpoint images are fused, pixel-level synthesis is performed based on the sub-pixel interlacing arrangement rules of the naked-eye 3D display device to generate naked-eye 3D interlaced output images, which are then sent to the naked-eye 3D display device at a preset frame rate to achieve real-time naked-eye 3D visual presentation of the correct spatial occlusion relationship between the user's limbs and virtual objects.

Images