Naked-eye three-dimensional display method and device based on double-layer display structure

Abstract

This invention provides a glasses-free 3D display method and apparatus based on a dual-layer liquid crystal display, applied in the field of 3D display technology. The method includes acquiring a target binocular image to be displayed and human eye position data; inputting the target binocular image and human eye position data into a pre-trained deep learning model to obtain a learnable pattern for a first display panel in a dual-layer display structure, and target display data for a second display panel in the dual-layer display structure, wherein the first display panel is located on the light-emitting side of the second display panel; displaying the target display data using the second display panel, and displaying the learnable pattern using the first display panel. By using the deep learning model to output target display data and learnable patterns for the actual human eye position, combined with the dual-layer display structure, 3D display tailored to the actual human eye position is achieved, solving the problem of fixed viewing position in existing technologies, reducing crosstalk, and thus improving the 3D display effect.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional display technology, and in particular to a naked-eye three-dimensional display method and device based on a dual-layer display structure. Background Technology

[0002] Existing glasses-free 3D display technology mainly achieves a 3D viewing effect by using different light field modulation methods to create binocular parallax.

[0003] In practical implementation, a common approach is to set a fixed slit grating in front of the two-dimensional liquid crystal panel. The slit grating controls the passage and blocking of light from each pixel in the two-dimensional liquid crystal panel, directing the light emitted from different pixels to the positions of the eyes respectively. This allows light from the left eye image pixels to enter the left eye, and light from the right eye image pixels to enter the right eye, so that the left and right eyes see different parallax maps, enabling three-dimensional vision to be generated without the need to wear glasses.

[0004] However, with the above method, since the position of the slit grating is fixed, its light path is also fixed. This requires the viewer to view from a fixed position to achieve 3D display. If the viewer's position changes, light rays from both eyes may fall into the same eye after passing through the slit grating. For example, light rays from pixels in the left and right eyes may both fall into the left eye or both into the right eye, causing visual crosstalk and resulting in poor 3D display effect. Summary of the Invention

[0005] This invention provides a naked-eye 3D display method and device based on a dual-layer display structure to solve the problem of fixed viewing position in the prior art, thereby improving the 3D display effect.

[0006] This invention provides a naked-eye 3D display method based on a dual-layer display structure, comprising the following steps: acquiring a target binocular image to be displayed and human eye position data; inputting the target binocular image and human eye position data into a pre-trained deep learning model to obtain a learnable pattern of a first display panel in the dual-layer display structure and target display data of a second display panel in the dual-layer display structure, wherein the first display panel is located on the light-emitting side of the second display panel; displaying the target display data using the second display panel and displaying the learnable pattern using the first display panel.

[0007] According to the present invention, a naked-eye 3D display method based on a dual-layer display structure is provided, wherein the learnable pattern is a learnable non-periodic slit image, a learnable grayscale image, or a learnable RGB image.

[0008] According to the present invention, a naked-eye 3D display method based on a dual-layer display structure is provided. The method inputs the target binocular image and human eye position data into a pre-trained deep learning model to obtain a learnable pattern for the first display panel in the dual-layer display structure and target display data for the second display panel in the dual-layer display structure. The method includes: encoding and fusing the target binocular image and human eye position data using the pre-trained deep learning model to obtain first feature data; decoding the first feature data using the deep learning model to obtain multi-channel display data; and splitting the multi-channel display data according to channels to obtain a learnable pattern for the first display panel in the dual-layer display structure and target display data for the second display panel in the dual-layer display structure.

[0009] According to the present invention, a naked-eye 3D display method based on a dual-layer display structure is provided. The method involves encoding and fusing features of the target binocular image and the human eye position data using a pre-trained deep learning model to obtain first feature data. The method includes: encoding the target binocular image using a pre-trained deep learning model to obtain second feature data; encoding and performing dimensional expansion and repetition on the human eye position data using the deep learning model to obtain third feature data; and fusing the second feature data and the third feature data to obtain the first feature data.

[0010] According to the present invention, a naked-eye 3D display method based on a dual-layer display structure is provided. The method involves encoding the target binocular image using a pre-trained deep learning model to obtain second feature data. This includes: stitching together the target left-eye image and the target right-eye image in the target binocular image to obtain a stitched image, wherein the dimensions of the stitched image are [1, 6, H1, W1], where 1 is the number of images in the stitched image, 6 is the number of channels in the stitched image, H1 is the height of the stitched image, and W1 is the width of the stitched image; and encoding the stitched image using the encoding layer of the pre-trained deep learning model to obtain second feature data, wherein the dimensions of the second feature data are [1, C, H2, W2], where 1 is the number of images in the second feature data, C is the number of channels in the second feature data, H2 is the height of the second feature data, and W2 is the width of the second feature data.

[0011] According to the present invention, a naked-eye 3D display method based on a dual-layer display structure is provided. The method involves encoding, dimensional expansion, and repetition of the human eye position data using the deep learning model to obtain third feature data. This includes: converting the human eye position data into positional encoding features using a lightweight multilayer perceptron of the deep learning model; wherein the dimension of the human eye position data is [1, 3], where 1 represents the number of human eye position data points and 3 represents the number of channels; the dimension of the positional encoding features is [1, C], where 1 represents the number of positional encoding features and C represents the number of channels; and dimensional expansion and repetition of the positional encoding features are performed to obtain third feature data, wherein the dimension of the third feature data is [1, C, H2, W2], where 1 is the number of images in the third feature data, C is the number of channels in the third feature data, H2 is the height of the third feature data, and W2 is the width of the third feature data.

[0012] According to the present invention, a naked-eye 3D display method based on a dual-layer display structure is provided. The step of splitting the multi-channel display data according to channels to obtain a learnable pattern for a first display panel in the dual-layer display structure and target display data for a second display panel in the dual-layer display structure includes: splitting the data corresponding to the first half of the multi-channel display data into first display data, and splitting the data corresponding to the second half of the multi-channel display data into second display data; splitting the first display data into a preset number of display data segments according to channels to obtain a learnable pattern for the first display panel; and splitting the second display data into a preset number of display data segments according to channels to obtain target display data for the second display panel.

[0013] According to the present invention, a naked-eye 3D display method based on a dual-layer display structure is provided. The dimensions of the multi-channel display data are [1, 6×N, H3, W3], where 1 is the number of images in the multi-channel display data, 6×N is the number of channels in the multi-channel display data, H3 is the height of the multi-channel display data, W3 is the width of the multi-channel display data, and N is the preset quantity. The dimensions of the first display data and the second display data are both [1, 3×N, H3, W3]. The dimensions of the learnable pattern of the first display panel are [N, 3, H3, W3], and the dimensions of the target display data of the second display panel are [N, 3, H3, W3].

[0014] According to the present invention, a naked-eye 3D display method based on a dual-layer display structure is provided, wherein the refresh rate of the dual-layer display structure ranges from 120Hz to 480Hz, and the preset quantity ranges from 4 to 8.

[0015] According to the present invention, a naked-eye 3D display method based on a dual-layer display structure is provided. The deep learning model is trained in the following manner: acquiring sample binocular images and sample human eye positions; inputting the sample binocular images and sample human eye position data into the deep learning model to obtain a predicted learnable pattern of the first display panel in the dual-layer display structure and predicted display data of the second display panel in the dual-layer display structure; displaying the predicted display data using the second display panel and displaying the predicted learnable pattern using the first display panel, and acquiring the actual binocular image displayed by the dual-layer display structure at the position indicated by the sample human eye position data; calculating the model loss of the deep learning model based on the actual binocular image and the sample binocular image; and adjusting the parameters of the deep learning model according to the model loss.

[0016] According to the present invention, a naked-eye 3D display method based on a dual-layer display structure is provided. The sample binocular image includes a sample left-eye image and a sample right-eye image, and the actual binocular image includes an actual left-eye image and an actual right-eye image. The same training batch is divided into a left-eye loss part and a right-eye loss part. During the training process of the left-eye loss part of the deep learning model, the model loss is an L2 loss calculated based on a preset brightness adjustment coefficient, the actual left-eye image, and the sample left-eye image. During the training process of the right-eye loss part of the deep learning model, the model loss is an L2 loss calculated based on a preset brightness adjustment coefficient, the actual right-eye image, and the sample right-eye image.

[0017] According to the present invention, a naked-eye 3D display method based on a dual-layer display structure is provided, wherein the learnable pattern is a learnable non-periodic slit image. During the training process of the deep learning model, the predicted learnable pattern displayed by the first display panel is initialized as a preset number of complementary slit patterns.

[0018] The present invention also provides a naked-eye 3D display device based on a dual-layer display structure, comprising the following modules: The eye-tracking module is used to acquire the viewer's eye position data; The control and processing module is used to input the target binocular image and human eye position data into a pre-trained deep learning model to obtain the learnable pattern of the first display panel in the dual-layer display structure and the target display data of the second display panel in the dual-layer display structure. The dual-layer display structure is used to display the target display data using the second display panel and to display the learnable pattern using the first display panel, wherein the first display panel is located on the light-emitting side of the second display panel.

[0019] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the naked-eye 3D display method based on a dual-layer display structure as described above.

[0020] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the naked-eye 3D display method based on a dual-layer display structure as described above.

[0021] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the naked-eye 3D display method based on a dual-layer display structure as described above.

[0022] The naked-eye 3D display method and device based on a dual-layer display structure provided by this invention uses human eye position data as input to a deep learning model, and uses the deep learning model to output target display data and learnable patterns for the actual human eye position. Combined with a dual-layer display structure, it realizes 3D display for the actual human eye position, which can solve the problem of fixed viewing position in the prior art, reduce crosstalk, and thus improve the 3D display effect. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0024] Figure 1 This is one of the flowcharts illustrating the naked-eye 3D display method based on a dual-layer display structure provided by the present invention; Figure 2 This is the second flowchart of the naked-eye 3D display method based on a dual-layer display structure provided by the present invention; Figure 3 This is a schematic diagram of a possible implementation of step S1021 in the naked-eye 3D display method based on a dual-layer display structure provided by the present invention; Figure 4 This is the third flowchart of the naked-eye 3D display method based on a dual-layer display structure provided by the present invention; Figure 5 This is a flowchart illustrating the deep learning model training method provided by the present invention; Figure 6 This is a schematic diagram of the naked-eye 3D display device based on a dual-layer display structure provided by the present invention; Figure 7 This is a possible optical path diagram of the dual-layer display structure provided by the present invention; Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0026] In existing glasses-free 3D display technologies, a common approach is to place a fixed slit grating in front of a 2D LCD panel. This slit grating allows the viewer's left and right eyes to see different parallax maps. Since the slit grating is fixed in position, the viewer can only view from a fixed location. If the viewer's position changes, light rays from both eyes may fall on the same eye after passing through the slit grating, resulting in visual crosstalk and a poor 3D display effect.

[0027] Furthermore, using the aforementioned fixed slit grating method, because the two-dimensional LCD panel needs to display both left-eye and right-eye images simultaneously, the upper limit of the image resolution ultimately seen by the viewer is only half the LCD screen resolution. To further improve resolution, related technologies propose dividing a single-viewpoint high-resolution image into N low-resolution images based on the slit grating (see patent document CN119865600A). Then, by using time-division multiplexing, leveraging the persistence of vision in the human eye, N images are refreshed in a short time, thereby improving the resolution viewed by the human eye. Although this method improves display resolution by utilizing the persistence of vision, each frame of the displayed image is averaged over the black areas of the other N-1 frames, resulting in an actual display brightness of 1 / N of the original, significantly reducing the viewing experience.

[0028] To address at least one of the aforementioned problems, this invention provides a naked-eye 3D display method and apparatus based on a dual-layer display structure, which will be described below in conjunction with... Figures 1 to 8 Please provide a detailed explanation.

[0029] Figure 1 This is one of the flowcharts illustrating the naked-eye 3D display method based on a dual-layer display structure provided by the present invention, such as... Figure 1 As shown, the method includes the following steps: Step S101: Obtain the target binocular image to be displayed and the human eye position data.

[0030] The target binocular image includes the target left-eye image and the target right-eye image. The target left-eye image and the target right-eye image are disparity maps of each other, used to simulate the disparity map seen by human binoculars. The human eye position data represents the position of the human eyes. The human eye position data can be the coordinates of the center position of the two eyes; or the human eye position data can be the coordinates of the left eye and / or the right eye, etc.

[0031] Step S102: Input the target binocular image and human eye position data into a pre-trained deep learning model to obtain the learnable pattern of the first display panel in the dual-layer display structure and the target display data of the second display panel in the dual-layer display structure, wherein the first display panel is located on the light-emitting side of the second display panel.

[0032] The dual-layer display structure includes two display panels: a first display panel and a second display panel. The first display panel is located on the light-emitting side of the second display panel. The first and second display panels work together to achieve glasses-free 3D display. The target display data of the second display panel can be an RGB image, and the learnable pattern of the first display panel can be a learnable aperiodic slit image, a learnable grayscale image, or a learnable RGB image.

[0033] In one example, the second display panel can be an RGB color display panel, and the first display panel can use a learnable slit grading method to cooperate with the second display panel to achieve 3D display. The opening and closing of the liquid crystals of each pixel in the first display panel can be controlled according to the learnable non-periodic slit image. This allows light from the left-eye image pixels in the second display panel to reach the left eye position through the light-transmitting area, and light from the right-eye image pixels in the second display panel to reach the right eye position through the light-transmitting area, while the light in the corresponding obstructed areas is blocked, thereby achieving spatial separation of the binocular parallax image.

[0034] In one example, the second display panel can be an RGB color display panel, and the first display panel can use a learnable grayscale map to work with the second display panel to achieve 3D display. The first display panel is a light-transmitting liquid crystal panel capable of displaying a learnable grayscale image. The grayscale values ​​of each pixel in the learnable grayscale image are used to modulate the light intensity of the RGB image displayed on the second display panel, thereby achieving 3D display.

[0035] In one example, the second display panel can be an RGB color display panel, and the first display panel can use a fully learnable RGB image to work in conjunction with the second display panel to achieve 3D display. The first display panel is a light-transmitting liquid crystal panel, and the learnable RGB image displayed by the first display panel and the RGB image displayed by the second display panel are optically superimposed more flexibly to form a 3D display image.

[0036] Step S103: Display the target display data using the second display panel, and display the learnable pattern using the first display panel.

[0037] The first display panel in the dual-layer display structure displays a learnable pattern, and the second display panel in the dual-layer display structure displays the target display data, thereby achieving three-dimensional display.

[0038] In this embodiment of the invention, flexible modulation based on the actual viewing position is achieved. The human eye position data is used as the input of a deep learning model, and the deep learning model outputs target display data and learnable patterns based on the actual human eye position. Combined with a dual-layer display structure, three-dimensional display based on the actual human eye position is realized. This can solve the problem of fixed viewing position in the prior art, reduce crosstalk, and thus improve the three-dimensional display effect.

[0039] exist Figure 1 Based on the illustrated embodiments, in one possible implementation, see [link to relevant documentation]. Figure 2 Step S102 involves inputting the target binocular image and human eye position data into a pre-trained deep learning model to obtain a learnable pattern for the first display panel in the dual-layer display structure, and target display data for the second display panel in the dual-layer display structure, including: Step S1021: Encode and fuse the target binocular image and the human eye position data using a pre-trained deep learning model to obtain the first feature data.

[0040] The order of encoding and feature fusion is not specified here. For example, a deep learning model can be used to encode the target binocular image and the human eye position data separately, and then the encoded data can be fused to obtain the first feature data. Alternatively, the target binocular image and the human eye position data can be fused first, and then the fused features can be encoded using a deep learning model to obtain the first feature data. Specifically, the three-dimensional coordinates of the human eye position data can be repeatedly expanded into a three-channel coordinate graph with the same size as the target left / right eye images. The expanded data is then concatenated with the three-channel target left and right eye images along the channel dimension to form a nine-channel tensor, which is then input into the deep learning model for encoding to obtain the first feature data.

[0041] Step S1022: Decode the first feature data using the deep learning model to obtain multi-channel display data.

[0042] Step S1023: The multi-channel display data is split according to the channel to obtain the learnable pattern of the first display panel in the dual-layer display structure and the target display data of the second display panel in the dual-layer display structure.

[0043] The number of channels in the multi-channel display data is 2M, where M is a positive integer. The data from the first to the Mth channel in the multi-channel display data is extracted to obtain the target display data for one display panel; the remaining data (the data from the (M+1)th to the 2Mth channel) in the multi-channel display data is extracted to obtain the learnable pattern for another display panel.

[0044] In this embodiment of the invention, the encoded features of the target binocular image and the human eye position data are fused through feature fusion, achieving collaborative processing of the target binocular image and the human eye position data. This allows the deep learning model to simultaneously use both sets of data to obtain the target display data. Furthermore, by splitting the multi-channel display data into target display data for each of the two display panels according to channels, it supports the independent display of each of the two display panels in a dual-layer display structure, thereby achieving three-dimensional display based on the actual human eye position.

[0045] exist Figure 2 Based on the illustrated embodiments, in one possible implementation, see [link to relevant documentation]. Figure 3 Step S1021, which involves encoding and fusing features of the target binocular image and the human eye position data using a pre-trained deep learning model to obtain first feature data, includes: Step S10211: Encode the target binocular image using a pre-trained deep learning model to obtain second feature data.

[0046] The target binocular image can first be converted into a multi-channel matrix form to facilitate processing by the deep learning model. Then, it can be encoded using the encoding layer of the deep learning model to obtain the second feature data. In one example, the target binocular image includes a target left-eye image and a target right-eye image, both with dimensions [1, 3, H1, W1], where 1 represents the number of images, 3 represents the number of image channels (i.e., RGB channels), H1 represents the height of the image, and W1 represents the width of the image. The target left-eye image and the target right-eye image can be stitched together to obtain a stitched image with dimensions [1, 6, H1, W1], where 1 is the number of images in the stitched image, 6 is the number of channels in the stitched image, H1 is the height of the stitched image, and W1 is the width of the stitched image. The stitched image is then encoded using the encoding layer of a pre-trained deep learning model to obtain the second feature data. In one example, the dimensions of the second feature data are [1, C, H2, W2], where 1 is the number of images of the second feature data, C is the number of channels of the second feature data, H2 is the height of the second feature data, and W2 is the width of the second feature data.

[0047] Step S10212: Encode the human eye position data and perform dimensional expansion and repetition using the deep learning model to obtain the third feature data.

[0048] Human eye position data can be represented by coordinates. Encoding this data and then performing dimensional expansion and repetition yields the third feature data. To facilitate the fusion of the third and second feature data, their dimensions must be identical, meaning the height and width of their images must be the same. Specifically, the lightweight multilayer perceptron of the deep learning model can be used to convert the human eye position data into position-encoded features; these features are then dimensionally expanded and repeated to obtain the third feature data.

[0049] In one example, the dimension of the human eye position data is [1, 3], where 1 represents the number of human eye position data points and 3 represents the number of channels (i.e., the three dimensions of the three-dimensional spatial coordinates). The reason the number of human eye position data points is 1 is that the three-dimensional coordinates of the midpoint between the two eyes are used to represent the actual position of the human eye, and the interocular distance is considered a default value (e.g., 70mm). Thus, the three-dimensional coordinates of the midpoint between the two eyes can be used to represent the positions of the left and right eyes. When using the three-dimensional coordinates of the left and right eyes respectively to represent the actual position of the human eye, the dimension of the human eye position data can be [2, 3] or [1, 6]. The dimension of the position encoding feature can be [1, C]. The dimension of the third feature data can be [1, C, H2, W2], where 1 is the number of images of the third feature data, C is the number of channels of the third feature data, H2 is the height of the third feature data, and W2 is the width of the third feature data.

[0050] Step S10213: The second feature data and the third feature data are fused to obtain the first feature data.

[0051] In one example, the second feature data and the third feature data can be directly concatenated to obtain the first feature data.

[0052] In this embodiment of the invention, the encoded human eye position data is dimensionally expanded and repeated so that the size of the third feature data is the same as the size of the second feature data, thereby facilitating the fusion of the second and third feature data.

[0053] Given the limited resolution of dual-layer display structures, time-division multiplexing can be used to improve the resolution of 3D displays. Figure 2 Based on the illustrated embodiments, in one possible implementation, see [link to relevant documentation]. Figure 4 Step S1023, which involves splitting the multi-channel display data according to channels to obtain the learnable pattern of the first display panel in the dual-layer display structure and the target display data of the second display panel in the dual-layer display structure, includes: Step S10231: Split the data corresponding to the first half of the multi-channel display data into first display data, and split the data corresponding to the second half of the multi-channel display data into second display data.

[0054] The number of channels in the multi-channel display data is 2M, where M is a positive integer. The data from the first to the Mth channel in the multi-channel display data is split to obtain the first display data; the remaining data (the data from the (M+1)th to the 2Mth channel) is split to obtain the second display data. In one example, the dimensions of the multi-channel display data are [1, 6×N, H3, W3], where 1 is the number of images in the multi-channel display data, 6×N is the number of channels in the multi-channel display data, H3 is the height of the multi-channel display data, W3 is the width of the multi-channel display data, and N is the preset quantity. Therefore, the dimensions of both the first and second display data are [1, 3×N, H3, W3].

[0055] Step S10232: The first display data is split into a preset number of display data segments according to the channel to obtain the learnable pattern of the first display panel; the second display data is split into a preset number of display data segments according to the channel to obtain the target display data of the second display panel.

[0056] The preset quantity can be set according to the refresh rate of the dual-layer display structure. The higher the refresh rate of the dual-layer display structure, the larger the preset quantity can be. In one example, the refresh rate of the dual-layer display structure ranges from 120Hz to 480Hz, and the preset quantity ranges from 4 to 8. The larger the preset quantity, the higher the resolution of the 3D image actually perceived by the human eye. When the resolution of the dual-layer display structure is X, the resolution of the 3D image actually perceived by the human eye can reach a maximum of X.

[0057] The first display data is divided into N display data pieces in channel order to obtain the learnable pattern of the first display panel. The second display data is divided into N display data pieces in channel order to obtain the target display data of the second display panel. In one example, the dimensions of the learnable pattern of the first display panel are [N, 3, H3, W3], and the dimensions of the target display data of the second display panel are [N, 3, H3, W3].

[0058] After obtaining the learnable patterns on the first display panel and the target display data of each of the second display panels, the learnable patterns are displayed using the first display panel, that is, the N learnable patterns of the first display panel with dimensions [3, H3, W3] are refreshed sequentially; and the target display data are displayed using the second display panel, that is, the N target display data of the second display panel with dimensions [3, H3, W3] are refreshed sequentially, thereby realizing the three-dimensional display of the target binocular image.

[0059] In this embodiment of the invention, the first display data and the second display data are respectively divided into a preset number of images according to the channel. Thus, during the display of a target binocular image, both the first display panel and the second display panel refresh the preset number of images. By using time-division multiplexing, the target left-eye image and the target right-eye image can be divided into N sub-images for display. Based on the characteristic of human visual persistence, refreshing these N sub-images in a short time can improve the resolution of the three-dimensional display.

[0060] The following section introduces the training methods for deep learning models. (See also...) Figure 5 ,include: Step S501: Obtain the sample binocular image and the position of the sample human eye.

[0061] The sample binocular images include a sample left-eye image and a sample right-eye image. The sample left-eye image and the sample right-eye image are disparity maps of each other, used to simulate the disparity map seen by human binoculars. The sample human eye position indicates the location of the human eye. In one example, a camera can be designed at the location of the left and right eyes respectively to facilitate the subsequent acquisition of actual binocular images.

[0062] Step S502: Input the sample binocular image and the sample human eye position data into the deep learning model to obtain the predicted learnable pattern of the first display panel in the dual-layer display structure and the predicted display data of the second display panel in the dual-layer display structure.

[0063] The deep learning model in this embodiment of the invention can be customized according to the actual situation. For example, it can be an image processing domain model with any encoder-decoder structure, such as the U-Net model, the Convolutional Neural Network (CNN) model, the Vision Transformer (ViT) model, or the Shifted Window Transformer (Swin-Transformer) model.

[0064] The process by which a deep learning model outputs predicted display data and predicted learnable patterns based on sample binocular images and sample human eye positions can be found in the above embodiment: the process by which a deep learning model outputs target display data and learnable patterns based on target binocular images and sample human eye position data.

[0065] In one example, the input data for the deep learning model includes: the sample binocular images have dimensions [2, 3, H1, W1], representing two 3-channel images with a resolution of H1×W1 (the sample left and right eye images); the sample eye position has dimensions [1, 3], representing one 3D coordinate (it is not necessary to input each eye separately, only the 3D coordinate of the midpoint between the eyes needs to be input, and the binocular distance can be fixed at 70mm by default). The predicted display data and predictable learnable patterns output by the deep learning model include: multiple frames of display images for each of the two display panels, with dimensions [2, N, 3, H3, W3], representing N 3-channel images with a resolution of H3×W3 for each of the two eyes. Here, H3×W3 is less than H1×W1, meaning the sample left and right eye images are each split into N images for display.

[0066] In one example, during a single forward pass, the sample binocular images are stitched together along the channel dimension to form a single 6-channel image: [1, 6, H1, W1]. After passing through the encoding layer of the deep learning model, high-dimensional image features (second feature data) are obtained: [1, C, H2, W2], where C is the channel dimension of the second feature data, and H2 and W2 are the height and width of the high-dimensional image features. The sample eye position: [1, 3] is converted into position-encoded features: [1, C] through a lightweight multilayer perceptron, and then transformed into third feature data: [1, C, H2, W2] through dimensional expansion and repetition. Finally, the second and third feature data are added to achieve feature fusion, resulting in the first feature data: [1, 2C, H2, W2]. The first feature data, after being decoded by the deep learning model, yields a single 6×N channel image [1, 6×N, H3, W3]. This image is then divided by channel; the first half represents the display data for the first display panel, and the second half represents the display data for the second display panel, denoted as [2, 3×N, H3, W3]. This represents the 3×N channel images with a resolution of H3×W3 for both display panels. The display data for the first display panel is then further divided by channel into N frames of predictable learnable patterns, denoted as [N, 3, H3, W3]. Similarly, the display data for the second display panel is divided into N frames of predicted display data, denoted as [N, 3, H3, W3], representing N 3-channel images with a resolution of H3×W3. This process yields the N frames of display content for each of the two display panels.

[0067] In one example, when the second display panel is an RGB color display panel, and the first display panel uses a learnable slit method to cooperate with the second display panel to achieve 3D display, it is necessary to average each predicted learnable pattern corresponding to the first display panel according to the channel and H3 dimension, and then repeatedly expand it back to the original shape. This ensures that the channel direction and the vertical direction maintain the same value, forming a slit pattern. In another example, when the second display panel is an RGB color display panel, and the first display panel uses a learnable grayscale image to cooperate with the second display panel to achieve 3D display, each predicted learnable pattern corresponding to the first display panel needs to average the channel dimension and then expand it. In yet another example, when the second display panel is an RGB color display panel, and the first display panel can use a fully learnable RGB image to cooperate with the second display panel to achieve 3D display, each predicted learnable pattern corresponding to the first display panel requires no other additional operations.

[0068] Step S503: Display the predicted display data using the second display panel and display the predicted learnable pattern using the first display panel, and acquire the actual binocular image displayed by the dual-layer display structure at the position indicated by the sample human eye position data.

[0069] At the left and right eye positions corresponding to the sample's eye position, images of the actual displayed dual-layer display structure are acquired using an image acquisition device to obtain the actual binocular images. Besides using an image acquisition device, actual binocular images can also be acquired through optical path simulation.

[0070] Step S504: Calculate the model loss of the deep learning model based on the actual binocular image and the sample binocular image.

[0071] Actual binocular images include actual left-eye and actual right-eye images, while sample binocular images include sample left-eye and sample right-eye images. The loss for each of the left and right eye images can be calculated separately and then summed as the model loss. Alternatively, the process of "calculating monocular loss + backpropagation update" can be performed alternately on the left and right eye images.

[0072] In one example, the first loss can be obtained by calculating the loss between the sample left-eye image and the actual left-eye image, and the second loss can be obtained by calculating the loss between the sample right-eye image and the actual right-eye image. The model loss is obtained by weighted summation of the first loss and the second loss.

[0073] In one example, the same training batch is divided into a left-eye loss part and a right-eye loss part. During the training of the left-eye loss part of the deep learning model, the model loss is the L2 loss of the actual left-eye image in the actual binocular image and the sample left-eye image in the sample binocular image; during the training of the right-eye loss part of the deep learning model, the model loss is the L2 loss of the actual right-eye image in the actual binocular image and the sample right-eye image in the sample binocular image.

[0074] While time-division multiplexing can improve the resolution of 3D displays, it reduces display brightness. Therefore, a preset brightness adjustment coefficient can be used to compensate for the brightness reduction caused by time-division multiplexing. In one example, when calculating the model loss of a deep learning model, the preset brightness adjustment coefficient can be multiplied by the brightness of the sample left-eye image to obtain the brightness-adjusted sample left-eye image; the preset brightness adjustment coefficient can be multiplied by the brightness of the sample right-eye image to obtain the brightness-adjusted sample right-eye image; then, the L2 loss between the brightness-adjusted sample left-eye image and the actual left-eye image, and / or the L2 loss between the brightness-adjusted sample right-eye image and the actual right-eye image can be calculated to obtain the model loss.

[0075] Alternatively, equivalently, the brightness of the actual left-eye image is divided by a preset brightness adjustment coefficient to obtain the brightness-adjusted actual left-eye image; the brightness of the actual right-eye image is divided by the preset brightness adjustment coefficient to obtain the brightness-adjusted actual right-eye image. Then, the L2 loss between the brightness-adjusted actual left-eye image and the sample left-eye image is calculated, and / or the L2 loss between the brightness-adjusted actual right-eye image and the sample right-eye image is calculated, thereby obtaining the model loss. The preset brightness adjustment coefficient ranges from 1 to N, where N is the preset number. In one example, the preset brightness adjustment coefficient can be set to N / 2. By introducing the preset brightness adjustment coefficient into the calculation process of the model loss, the deep learning model is forced to fit a brighter dual-layer display structure during gradient descent optimization. This ensures that after N frames of time-division multiplexing, the average perceived brightness of the image by the human eye is close to the brightness of the original target binocular image, thus solving the problem of decreased display brightness while maintaining improved resolution.

[0076] Step S505: Adjust the parameters of the deep learning model according to the model loss.

[0077] The parameters of the deep learning model are adjusted using gradient descent based on the model loss.

[0078] Repeat steps S501-S505 until the preset termination condition is met to obtain the trained deep learning model. The preset termination condition can be set according to the actual situation, such as the model loss converging or reaching a preset number of training iterations.

[0079] exist Figure 5 Based on the illustrated embodiment, in order to increase the model convergence speed, in one possible implementation, the learnable pattern is a learnable aperiodic slit image; during the training of the deep learning model, the predicted learnable pattern displayed on the first display panel is initialized as a preset number of complementary slit patterns.

[0080] When the first display panel uses a learnable slit method to separate the left-eye and right-eye images, each pixel in the first display panel has only two modes: fully open (represented by 1) and fully closed (represented by 0). The preset number is N. Therefore, the initial value of the predictable learnable pattern displayed by the first display panel is N complementary slit patterns. Taking N=4 as an example, in the first frame image of the predictable learnable pattern displayed by the first display panel, each row is arranged in a repeating pattern of 10001000…; in the second frame image, each row is arranged in a repeating pattern of 01000100…; in the third frame image, each row is arranged in a repeating pattern of 00100010…; and in the fourth frame image, each row is arranged in a repeating pattern of 00010001… After summing the four frames, each pixel position is 1, thus forming a complementary structure.

[0081] In this embodiment of the invention, when the first display panel uses a learnable slit method to achieve three-dimensional display, during the training process of the deep learning model, by initializing the predicted learnable pattern displayed on the first display panel as a preset number of complementary slit patterns, convergence learning can be accelerated and the training speed of the deep learning model can be improved.

[0082] The naked-eye 3D display device based on a dual-layer display structure provided by the present invention is described below. The naked-eye 3D display device based on a dual-layer display structure described below and the naked-eye 3D display method based on a dual-layer display structure described above can be referred to in correspondence with each other.

[0083] See Figure 6 , Figure 6 This is a schematic diagram of a naked-eye 3D display device based on a dual-layer display structure according to an embodiment of the present invention, comprising: The human eye tracking module 601 is used to acquire the viewer's eye position data.

[0084] In one example, the eye-tracking module 601 may include a camera and a 3D position calculation unit; the camera is used to acquire image data including the viewer's eyes; the 3D position calculation unit is used to determine the viewer's eye position data based on the image data. The camera may be a depth camera capable of acquiring depth information, such as a binocular camera or a time-of-flight (TOF) camera. To ensure real-time eye tracking, in one example, the camera's sampling frequency needs to be greater than or equal to 60Hz. The 3D position calculation unit obtains the viewer's eye position data based on the eye region in the image data.

[0085] The control and processing module 602 is used to input the target binocular image and human eye position data into a pre-trained deep learning model to obtain the learnable pattern of the first display panel in the dual-layer display structure and the target display data of the second display panel in the dual-layer display structure.

[0086] The control and processing module 602 can be a computing unit with data processing capabilities, such as a System-on-Chip (SoC), a Microcontroller Unit (MCU), a Graphics Processing Unit (GPU), or a Field-Programmable Gate Array (FPGA). The control and processing module 602 is used to run the deep learning model, taking the target binocular image and human eye position data as input, and outputting the target display data / learnable image for each layer of the dual-layer display structure 603. The learnable pattern can be a learnable aperiodic slit image, a learnable grayscale image, or a learnable RGB image.

[0087] A dual-layer display structure 603 is used to display the target display data using the second display panel and to display the learnable pattern using the first display panel, wherein the first display panel is located on the light-emitting side of the second display panel.

[0088] In one example, such as Figure 7As shown, the dual-layer display structure 603 includes a first display panel 6031 and a second display panel 6032. The first display panel 6031 is located on the light-emitting side of the second display panel 6032, that is, the first display panel 6031 is located between the second display panel 6032 and the viewer's eyes. The second display panel 6032 can be a color display panel, for example, it can be an organic light-emitting diode (OLED) panel, a mini light-emitting diode (Mini-LED) panel, or a liquid crystal display (LCD) panel. The first display panel 6031 can be a grayscale liquid crystal panel or a color display panel. Wherein, 701 represents the light rays of the right eye image, 702 represents the light rays of the left eye image, 703 represents the viewer's right eye, and 704 represents the viewer's left eye.

[0089] In this embodiment of the invention, flexible modulation based on the actual viewing position is achieved. The human eye position data is used as the input of a deep learning model, and the deep learning model outputs target display data and learnable patterns based on the actual human eye position. Combined with a dual-layer display structure, three-dimensional display based on the actual human eye position is realized. This can solve the problem of fixed viewing position in the prior art, reduce crosstalk, and thus improve the three-dimensional display effect.

[0090] In one possible implementation, the control and processing module 602 is specifically used to: encode and fuse the target binocular image and the human eye position data using a pre-trained deep learning model to obtain first feature data; decode the first feature data using the deep learning model to obtain multi-channel display data; and split the multi-channel display data according to the channels to obtain a learnable pattern of the first display panel in the dual-layer display structure and target display data of the second display panel in the dual-layer display structure.

[0091] In one possible implementation, the control and processing module 602 is specifically used to: encode the target binocular image using a pre-trained deep learning model to obtain second feature data; encode and perform dimensional expansion and repetition on the human eye position data using the deep learning model to obtain third feature data; and fuse the second feature data and the third feature data to obtain first feature data.

[0092] In one possible implementation, the control and processing module 602 is specifically used to: stitch together the target left-eye image and the target right-eye image in the target binocular image to obtain a stitched image, wherein the dimensions of the stitched image are [1, 6, H1, W1], where 1 is the number of images in the stitched image, 6 is the number of channels in the stitched image, H1 is the height of the stitched image, and W1 is the width of the stitched image; and encode the stitched image using the encoding layer of a pre-trained deep learning model to obtain second feature data, wherein the dimensions of the second feature data are [1, C, H2, W2], where 1 is the number of images in the second feature data, C is the number of channels in the second feature data, H2 is the height of the second feature data, and W2 is the width of the second feature data.

[0093] In one possible implementation, the control and processing module 602 is specifically used to: convert the human eye position data into positional encoding features using the lightweight multilayer perceptron of the deep learning model, wherein the dimension of the human eye position data is [1, 3], where 1 represents the number of human eye position data and 3 represents the number of channels; the dimension of the positional encoding features is [1, C], where 1 represents the number of positional encoding features and C represents the number of channels; and the positional encoding features are dimensionally expanded and repeated to obtain third feature data, wherein the dimension of the third feature data is [1, C, H2, W2], where 1 is the number of images of the third feature data, C is the number of channels of the third feature data, H2 is the height of the third feature data, and W2 is the width of the third feature data.

[0094] In one possible implementation, the control and processing module 602 is specifically used to: split the data corresponding to the first half of the multi-channel display data into first display data, and split the data corresponding to the second half of the multi-channel display data into second display data; split the first display data into a preset number of display data according to the channels to obtain a learnable pattern for the first display panel; and split the second display data into a preset number of display data according to the channels to obtain target display data for the second display panel.

[0095] In one possible implementation, the dimensions of the multi-channel display data are [1, 6×N, H3, W3], where 1 is the number of images in the multi-channel display data, 6×N is the number of channels in the multi-channel display data, H3 is the height of the multi-channel display data, W3 is the width of the multi-channel display data, and N is the preset quantity; the dimensions of the first display data and the second display data are both [1, 3×N, H3, W3]; the dimensions of the learnable pattern of the first display panel are [N, 3, H3, W3], and the dimensions of the target display data of the second display panel are [N, 3, H3, W3].

[0096] In one possible implementation, the refresh rate of the dual-layer display structure ranges from 120Hz to 480Hz, and the preset quantity ranges from 4 to 8.

[0097] In one possible implementation, the control and processing module 602 is further configured to: acquire sample binocular images and sample human eye positions; input the sample binocular images and sample human eye position data into a deep learning model to obtain a predicted learnable pattern for the first display panel in the dual-layer display structure, and predicted display data for the second display panel in the dual-layer display structure; display the predicted display data using the second display panel, and display the predicted learnable pattern using the first display panel, and acquire the actual binocular image displayed by the dual-layer display structure at the position indicated by the sample human eye position data; calculate the model loss of the deep learning model based on the actual binocular image and the sample binocular image; and adjust the parameters of the deep learning model according to the model loss.

[0098] In one possible implementation, the sample binocular image includes a sample left-eye image and a sample right-eye image, and the actual binocular image includes an actual left-eye image and an actual right-eye image. The same training batch is divided into a left-eye loss portion and a right-eye loss portion. During the training of the left-eye loss portion of the deep learning model, the model loss is an L2 loss calculated based on a preset brightness adjustment coefficient, the actual left-eye image, and the sample left-eye image. During the training of the right-eye loss portion of the deep learning model, the model loss is an L2 loss calculated based on a preset brightness adjustment coefficient, the actual right-eye image, and the sample right-eye image.

[0099] In one possible implementation, the learnable pattern is a learnable aperiodic slit image; during the training of the deep learning model, the predicted learnable pattern displayed on the first display panel is initialized as a preset number of complementary slit patterns.

[0100] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8 As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute any of the aforementioned glasses-free 3D display methods based on a dual-layer display structure.

[0101] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0102] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute any of the above-mentioned naked-eye 3D display methods based on a dual-layer display structure.

[0103] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described naked-eye 3D display methods based on a dual-layer display structure.

[0104] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0105] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A naked-eye 3D display method based on a dual-layer display structure, characterized in that, include: Acquire the target binocular image to be displayed and the human eye position data; The target binocular image and the human eye position data are encoded and feature fused using a pre-trained deep learning model to obtain the first feature data; The deep learning model is used to decode the first feature data to obtain multi-channel display data; The data corresponding to the first half of the multi-channel display data is split into first display data, and the data corresponding to the second half of the multi-channel display data is split into second display data; The first display data is divided into a preset number of display data segments according to the channel to obtain a learnable pattern for the first display panel; the second display data is divided into a preset number of display data segments according to the channel to obtain target display data for the second display panel, wherein the first display panel is located on the light-emitting side of the second display panel; The target display data is displayed using the second display panel, and the learnable pattern is displayed using the first display panel.

2. The method according to claim 1, characterized in that, The learnable pattern can be a learnable aperiodic slit image, a learnable grayscale image, or a learnable RGB image.

3. The method according to claim 1, characterized in that, The first feature data is obtained by encoding and fusing features of the target binocular image and the human eye position data using a pre-trained deep learning model, including: The target binocular image is encoded using a pre-trained deep learning model to obtain second feature data; The deep learning model is used to encode, expand, and repeat the human eye position data to obtain the third feature data. The second feature data and the third feature data are fused to obtain the first feature data.

4. The method according to claim 3, characterized in that, The second feature data obtained by encoding the target binocular image using a pre-trained deep learning model includes: The left eye image and the right eye image of the target in the target binocular image are stitched together to obtain a stitched image, wherein the dimensions of the stitched image are [1, 6, H1, W1], 1 is the number of images in the stitched image, 6 is the number of channels in the stitched image, H1 is the height of the stitched image, and W1 is the width of the stitched image. The stitched image is encoded using the encoding layer of a pre-trained deep learning model to obtain second feature data, wherein the dimensions of the second feature data are [1, C, H2, W2], where 1 is the number of images in the second feature data, C is the number of channels in the second feature data, H2 is the height of the second feature data, and W2 is the width of the second feature data.

5. The method according to claim 4, characterized in that, The process of encoding, expanding, and repeating the human eye position data using the deep learning model to obtain third feature data includes: The human eye position data is converted into positional encoding features using the lightweight multilayer perceptron of the deep learning model. The human eye position data has a dimension of [1, 3], where 1 represents the number of human eye position data and 3 represents the number of channels. The positional encoding features have a dimension of [1, C], where 1 represents the number of positional encoding features and C represents the number of channels. The location encoding features are expanded and repeated in dimensions to obtain third feature data, wherein the dimensions of the third feature data are [1, C, H2, W2], where 1 is the number of images of the third feature data, C is the number of channels of the third feature data, H2 is the height of the third feature data, and W2 is the width of the third feature data.

6. The method according to claim 1, characterized in that, The dimensions of the multi-channel display data are [1, 6×N, H3, W3], where 1 is the number of images in the multi-channel display data, 6×N is the number of channels in the multi-channel display data, H3 is the height of the multi-channel display data, W3 is the width of the multi-channel display data, and N is the preset quantity; the dimensions of the first display data and the second display data are both [1, 3×N, H3, W3]; the dimensions of the learnable pattern of the first display panel are [N, 3, H3, W3], and the dimensions of the target display data of the second display panel are [N, 3, H3, W3].

7. The method according to claim 5, characterized in that, The refresh rate of the dual-layer display structure ranges from 120Hz to 480Hz, and the preset quantity ranges from 4 to 8.

8. The method according to claim 1, characterized in that, The deep learning model is trained in the following manner: Acquire sample binocular images and the position of the human eyes in the samples; The sample binocular images and the sample human eye position data are input into a deep learning model to obtain the predicted learnable pattern of the first display panel in the dual-layer display structure and the predicted display data of the second display panel in the dual-layer display structure. The predicted display data is displayed using the second display panel, and the predicted learnable pattern is displayed using the first display panel. The actual binocular image displayed by the dual-layer display structure is acquired at the position indicated by the sample human eye position data. The model loss of the deep learning model is calculated based on the actual binocular image and the sample binocular image; The parameters of the deep learning model are adjusted according to the model loss described above.

9. The method according to claim 8, characterized in that, The learnable pattern is a learnable non-periodic slit image. During the training process of the deep learning model, the predicted learnable pattern displayed on the first display panel is initialized as a preset number of complementary slit patterns. And / or, the sample binocular image includes a sample left-eye image and a sample right-eye image, and the actual binocular image includes an actual left-eye image and an actual right-eye image; The same training batch is divided into a left-eye loss part and a right-eye loss part. During the training of the left-eye loss part of the deep learning model, the model loss is the L2 loss calculated based on the preset brightness adjustment coefficient, the actual left-eye image and the sample left-eye image. During the training of the right eye loss part of the deep learning model, the model loss is the L2 loss calculated based on the preset brightness adjustment coefficient, the actual right eye image, and the sample right eye image.

10. A naked-eye 3D display device based on a dual-layer display structure, characterized in that, include: The eye-tracking module is used to acquire the viewer's eye position data; The control and processing module is used to acquire the target binocular image to be displayed, and to encode and fuse the target binocular image and the human eye position data using a pre-trained deep learning model to obtain the first feature data. The deep learning model is used to decode the first feature data to obtain multi-channel display data; the data corresponding to the first half of the multi-channel display data is split into first display data, and the data corresponding to the second half of the multi-channel display data is split into second display data; the first display data is split into a preset number of display data according to the channels to obtain a learnable pattern for the first display panel; The second display data is divided into a preset number of display data segments according to the channel to obtain the target display data of the second display panel; The dual-layer display structure is used to display the target display data using the second display panel and to display the learnable pattern using the first display panel, wherein the first display panel is located on the light-emitting side of the second display panel.

Images