Light field display method and system and storage medium
The light field display method addresses the limitation of two-dimensional displays by collecting and processing interaction data to render high-definition and low-definition areas, improving three-dimensional content display efficiency and user experience.
Patent Information
- Application Number
- CN202310665056.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-06
- Publication Date
- 2024-12-06
AI Technical Summary
Traditional display solutions fail to fully express three-dimensional visual information, resulting in poor display effects as they can only present two-dimensional images.
A light field display method that collects original image data, identifies interaction data such as face, gaze point, and gesture positions, and renders high-definition and low-definition areas based on these interactions to create light field data for display.
Enables active interaction and efficient data processing, reducing latency and system resource occupation while enhancing the display of three-dimensional content.
Smart Images

Figure CN119088205A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of three-dimensional imaging, and in particular to an active interactive detection light field display method, an active interactive detection light field display system, and a storage medium. Background Art
[0002] In nature, the human eye can receive light emitted by objects and thus see three-dimensional objects. However, traditional display solutions can only present two-dimensional images, and the generation and display of content cannot fully express the visual information of each dimension, resulting in poor display effects.
[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention
[0004] According to a first aspect of the present disclosure, a light field display method is provided, which is applied to a light field display system, and the method includes: collecting original image data; identifying the original image data to obtain corresponding interaction data; wherein the interaction data includes: any one or a combination of any one of a face space position, a gaze point space position, a gesture space position and a gesture category; sending the interaction data to a content generation unit to obtain content to be displayed rendered by the content generation unit based on the interaction data; rearranging data according to the content to be displayed in combination with the interaction data to obtain light field data, and displaying the light field data.
[0005] In an exemplary embodiment of the present disclosure, a content generation unit renders the content to be displayed based on the interaction data, including: calculating the overlapping area according to the spatial position of the gesture and the position of the currently displayed content; when it is determined that there is an overlapping area, determining the updated content corresponding to the currently displayed content according to the gesture category; and rendering the updated content to obtain the content to be displayed.
[0006] In an exemplary embodiment of the present disclosure, the updated content is rendered to obtain the content to be displayed, including: determining a high-definition area and a low-definition area of the updated content according to the spatial position of the gaze point; and rendering the high-definition area and the low-definition area respectively to obtain the content to be displayed.
[0007] In an exemplary embodiment of the present disclosure, the original image data includes a face image; the identifying the original image data to obtain corresponding interaction data includes: identifying the face image to determine a face ROI area in the face image; determining the corresponding face spatial position based on the face ROI area; and updating the face ROI area according to preset rules, and performing face tracking based on the updated face ROI area.
[0008] In an exemplary embodiment of the present disclosure, the original image data includes an eye image; the identifying the original image data to obtain corresponding interaction data includes: performing image recognition on the eye image to determine an eye ROI area in the eye image; determining a corresponding gaze point spatial position based on the eye ROI area; and updating the eye ROI area according to preset rules, and using the updated eye ROI area to perform eye tracking.
[0009] In an exemplary embodiment of the present disclosure, the original image data includes a gesture image; the identifying the original image data to obtain corresponding interaction data includes: performing image recognition on the gesture image to determine a hand ROI area; determining a corresponding hand spatial position based on the hand ROI area; updating the hand ROI area according to preset rules, and using the updated hand ROI area to track the hand; and identifying the hand ROI area using a gesture recognition model to determine a corresponding gesture category.
[0010] In an exemplary embodiment of the present disclosure, the method further includes: in response to a trigger operation, configuring a corresponding ROI area for each sensor used to collect raw image data.
[0011] In an exemplary embodiment of the present disclosure, the method further includes: configuring a rendering mode of the content generating unit in response to a configuration instruction.
[0012] In an exemplary embodiment of the present disclosure, the collecting of raw image data includes: collecting facial images using a plurality of grayscale cameras, collecting eye images using an IR camera, and collecting gesture images using a depth camera; the plurality of grayscale cameras, IR cameras, and depth cameras are arranged on one side of the light field display unit.
[0013] According to one aspect of the present disclosure, a light field display system is provided, the system comprising:
[0014] An active detection unit, used for collecting raw image data;
[0015] An interaction information calculation unit is used to receive the original image data, identify the original image data to obtain corresponding interaction data; and send the interaction data to the content generation unit; wherein the interaction data includes: any one or a combination of any one of the face space position, gaze point space position, gesture space position and gesture category;
[0016] A display data processing unit, configured to receive the to-be-displayed content rendered by the content generation unit, and to rearrange the data in combination with the interaction data and the pixel arrangement map to obtain light field data;
[0017] A light field display unit is used to display the light field data.
[0018] In an exemplary embodiment of the present disclosure, the interaction information calculation unit configures the ROI area of each sensor in the active detection unit in response to the trigger information.
[0019] In an exemplary embodiment of the present disclosure, the display data processing unit sends a synchronization signal to the interaction information calculation unit, so that the interaction information calculation unit completes the calculation of the interaction data in response to the synchronization information.
[0020] In an exemplary embodiment of the present disclosure, the interactive active detection unit is arranged on one side of the light field display unit, and includes a plurality of grayscale cameras, an IR camera, and a depth camera; the grayscale camera is used to collect face images, the IR camera is used to collect eye images, and the depth camera is used to collect gesture images.
[0021] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the light field display method described in any one of the above is implemented.
[0022] According to the light field display method provided by the present disclosure, by collecting the original image and identifying the original image, the corresponding interaction data can be determined; and the content generation unit can be used to generate the content to be displayed based on the interaction data, and the data of the content to be displayed can be rearranged, so that the light field data can be displayed to realize active interaction.
[0023] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:
[0025] Figure 1 A schematic diagram schematically illustrates the composition of a light field display system according to an exemplary embodiment of the present disclosure;
[0026] Figure 2 A schematic diagram schematically showing the arrangement of cameras;
[0027] Figure 3 A schematic diagram schematically illustrates a light field display method according to an exemplary embodiment of the present disclosure;
[0028] Figure 4 A schematic diagram schematically illustrates a process of a method for calculating the spatial position of a face in an exemplary embodiment of the present disclosure;
[0029] Figure 5 A schematic diagram schematically illustrates a flow of a method for calculating a spatial position of a gaze point in an exemplary embodiment of the present disclosure;
[0030] Figure 6 A schematic diagram schematically illustrates a process of a method for calculating a spatial position of a human hand and a gesture type in an exemplary embodiment of the present disclosure;
[0031] Figure 7 A schematic diagram schematically illustrates a face ROI region marking in an exemplary embodiment of the present disclosure;
[0032] Figure 8 A schematic diagram schematically illustrates a human eye ROI region marking in an exemplary embodiment of the present disclosure;
[0033] Fig. 9 A schematic diagram schematically illustrates a hand ROI region marking in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION
[0034] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
[0035] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.
[0036] In view of the shortcomings and deficiencies of the prior art, this exemplary embodiment provides a light field display system 10, which can be used as a display terminal. Figure 1 As shown, the system 10 includes: an active detection unit 101, an interaction information calculation unit 102, a display data processing unit 103, and a light field display unit 104. The active detection unit 101 is connected to the interaction information calculation unit 102, the interaction information calculation unit 102 is connected to the display data processing unit 103, and the display data processing unit 103 is connected to the light field display unit 104. The active detection unit 101 can be used to collect original image data; wherein the original image data includes: face image, eye image, gesture image. The interaction information calculation unit 102 can be used to receive the original image data collected by the active detection unit, and identify the original image data to obtain corresponding interaction data; and send the interaction data to the content generation unit 201. The display data processing unit 103 can be used to receive the content to be displayed rendered by the content generation unit 201, and rearrange the data in combination with the interaction data and the pixel arrangement map to obtain light field data; and send it to the light field display unit 104. The light field display unit 104 can be used to display the light field data.
[0037] Among them, the active detection unit can receive ROI (range of interesting) control data, such as ROI coordinates (x, y, w, h) from the interactive information calculation unit through I2C (Inter-Integrated Circuit) / SPI (Serial Peripheral Interface) and the like; on the other hand, it can also send real-time collected raw image data (raw data) to the interactive information calculation unit through Mipi (Mobile Industry Processor Interface) / USB (Universal Serial Bus) and the like.
[0038] Specifically, the active detection unit may include a number of grayscale cameras, IR cameras, depth cameras and infrared LEDs. Among them, the grayscale camera can capture grayscale images, which can be used to detect the spatial position of the face relative to the screen, and the number is 2n, where n is a positive integer; the spatial position of the face relative to the screen can be determined by binocular ranging. Specifically, multiple groups of grayscale cameras can be set to achieve large-scale face tracking. The IR camera (infrared camera) can capture infrared images to accurately capture the movement of the human eye pupil, and then obtain the spatial gaze position of the human eye based on the movement of the human eye pupil; similarly, multiple groups of IR cameras can be set to obtain a larger detection range by splicing multiple groups. The depth camera (not limited to structured light or TOF camera) can be used to capture depth images, which are used to capture the movement of human hands using depth images and determine the corresponding gesture operations. The infrared LED is used to provide 850nm fill light for the IR camera to improve image quality. Reference Figure 2 As shown, the cameras included in the active detection unit 101 can be arranged at intervals. For example, an infrared LED 21, a depth camera 22, a group of IR cameras 23, and a group of grayscale cameras 24 are arranged in sequence in a certain direction to form a camera group. Multiple camera groups can be set.
[0039] The light field display unit can be used to receive data processed by the display data processing unit, and display it to restore the real light field information; the light field display unit can be composed of an ultra-high resolution display panel and an optical light control device; wherein the pixel arrangement of the display panel can be conventional RGB or a pixel island RGB arrangement; the optical light control device can be a physical lens or a liquid crystal lens for light control.
[0040] The light field display system can exchange data with the system end 20. For example, the system end can be a smart terminal device such as a computer or a tablet computer. The system end 20 can include a content generation unit 201, which is used to receive the interaction data transmitted by the interaction information calculation unit, and use the interaction data to calculate, generate the content required for the light field display, and render the display content. It can be based on the calculated coordinates of the face center / brow center as a reference, and 2 to n cameras can be set on the left and right according to the optical design to collect image content under multiple views; and then transmit it to the display data processing unit. Among them, the interaction data can include the spatial position of the face, the spatial position of the gaze point, the spatial position of the gesture, and the gesture category.
[0041] For example, the light field display system can be used as a display end. By setting an interactive information calculation unit and a display data processing unit, the collected image data can be processed at the display front end to obtain the ROI area in the image, thereby realizing efficient data processing and low-latency interaction. A display device with active detection function is provided.
[0042] This example embodiment provides a light field display method, which can be applied to the above-mentioned light field display system. Figure 3 As shown in , the above method may include:
[0043] Step S11, collecting original image data;
[0044] Step S12, identifying the original image data to obtain corresponding interaction data; wherein the interaction data includes: face space position, gaze point space position, gesture space position and gesture category;
[0045] Step S13, sending the interaction data to a content generation unit to obtain content to be displayed rendered by the content generation unit based on the interaction data;
[0046] Step S14: Rearrange data according to the content to be displayed and the interaction data to obtain light field data, and display the data.
[0047] The light field display method provided in this exemplary embodiment integrates these functions on the display end by realizing the collection, recognition, and data reshooting of original image data on the display end, without occupying system resources.
[0048] Below, each step of the light field display method in this example implementation will be described in more detail with reference to the accompanying drawings and embodiments.
[0049] In step S11, original image data is acquired.
[0050] In this example implementation, an active detection unit may be provided to collect raw image data; wherein the raw image data includes: face images, eye images, and gesture images. Figure 2 As shown, the active detection unit may include several grayscale cameras, IR cameras, depth cameras, and infrared LEDs. Among them, several grayscale cameras can be used to collect face images, IR cameras can be used to collect eye images, and depth cameras can be used to collect gesture images. For example, a naked-eye 3D display device can be provided, on which the above-mentioned light field display system is arranged; the active detection device can be arranged on one side of the display device. The user can maintain a certain spatial position with the display device, for example, standing in front of the display device; the user's face image, eye image, and gesture image are collected in real time by the active detection unit. In the initial state, the original image collected by each camera can be an image in the current area.
[0051] In step S12, the original image data is identified to obtain corresponding interaction data; wherein the interaction data includes: face spatial position, gaze point spatial position, gesture spatial position and gesture category.
[0052] In this example implementation, the above step S12 may include:
[0053] Step S21, identifying the face image to determine a face ROI region in the face image;
[0054] Step S22, determining the corresponding face spatial position based on the face ROI area; and
[0055] Step S23, updating the face ROI region according to a preset rule, and performing face tracking based on the updated face ROI region.
[0056] Specifically, an interactive information calculation unit can be provided, which can receive the original image data sent by the active detection unit and calculate the image data. When calculating the spatial position of the face, the interactive information calculation unit can receive the grayscale image captured by the grayscale camera as the face image, and use the binocular ranging algorithm to calculate the grayscale image to determine the spatial position of the face relative to the screen of the display device. When the active detection unit is arranged with multiple groups of grayscale cameras, multiple groups of grayscale images can be provided to the interactive information calculation unit; the interactive information calculation unit can calculate multiple spatial positions of faces based on the multiple groups of grayscale images.
[0057] In the initial state, the grayscale camera collects all the contents of the current field of view as the initial face image, and sends the initial face image to the interactive information calculation unit. The interactive information calculation unit can first perform image recognition on the initial face image to determine the face ROI area and the corresponding coordinate information (x, y, w, h); then use the binocular ranging algorithm to calculate the corresponding face spatial position according to the face ROI area. Specifically, the interactive information calculation unit can use a face detection model based on deep learning to perform face recognition on the collected grayscale image and determine the face ROI area in the grayscale image. Then, based on the binocular ranging algorithm, the face ROI area of the acquired initial face image can be calculated to determine the initial face spatial position. For example, the binocular ranging algorithm generally needs to calibrate the binocular camera first to obtain the internal and external parameters and homography matrix of the two cameras; then, the collected initial face image can be corrected according to the calibration result, and the two corrected images are located in the same plane and parallel to each other; the two corrected images are pixel matched to obtain a disparity image; the depth of each pixel is calculated according to the matching result to obtain a depth map; and then the spatial position of the face is determined. The spatial position calculation based on the binocular ranging algorithm can be obtained by conventional means, and the present disclosure does not specifically limit the specific steps of this algorithm. In addition, the interactive information calculation unit can also configure the ROI area information of the grayscale camera and the size of the bounding box, and push it to the active detection unit, so that the grayscale camera can collect the subsequent frames of small-sized grayscale images according to the configured ROI area information and bounding box size, thereby realizing the accurate collection of face images.
[0058] For example, for the collected initial face image, the interaction information calculation unit can use a bounding box to mark the face ROI area and mark the corresponding coordinate information (x, y, w, h) as the updated ROI area information; for example, refer to Figure 7As shown, a face ROI region is marked with a bounding box; and the updated ROI region information is sent to the active detection unit. The active detection unit receives the updated ROI region information, and when the next frame of face image is collected, the grayscale image of the next frame can be recollected according to the coordinate information in the updated ROI region information. In addition, at the same time, during the collection process of the subsequent frames, the size of the collected image can be reconfigured according to the updated ROI region information; for example, the size of the ROI region can be enlarged according to a preset rule to obtain an enlarged size, so that the grayscale image of the next frame can be recollected based on the updated coordinate information of the ROI region and the enlarged size, and then the next frame of grayscale image is sent to the interactive information calculation unit, and the face spatial position of the next frame of image is calculated; this process is repeated to achieve continuous tracking of the face image. During the tracking process of the face image, if the face tracking fails according to the image collected according to the updated ROI region information, all the content in the field of view of the grayscale camera can be recollected, and face recognition and ROI region update can be re-performed to achieve continuous tracking.
[0059] For example, refer to Figure 4 As shown, for the interactive information calculation unit, the face ROI area can be initialized first. In the initial state, face detection can be performed on the collected face area image to determine the corresponding face ROI area, and the face spatial position under the current grayscale image can be calculated using the binocular ranging method according to the calculation result of the face ROI area; at the same time, the coordinate information of the face ROI area can be updated using the face ROI area obtained above; the next frame of small-sized image is reassembled using the updated face ROI area information, and face recognition is performed to determine whether the tracking is successful; if the tracking is successful, the ROI area information is updated according to the face ROI area information in the frame image until the tracking is completed; or, if the tracking is judged to fail in the next frame image, re-initialization is performed, the whole face image of full size is re-collected, and face detection is performed again to re-execute the above method flow. At the same time, after the ROI area information is updated, the image acquisition position information of the grayscale camera is configured according to the updated face ROI area; so that the grayscale camera can acquire a smaller-sized face grayscale image. The spatial position of the face obtained by the above method may be the spatial position of the center point of the face, or the spatial position of the center of the eyebrows, the spatial position of the center of the eyes, etc.
[0060] In this example implementation, the above step S12 may further include:
[0061] Step S31, performing image recognition on the eye image to determine the eye ROI area in the eye image;
[0062] Step S32, determining a corresponding gaze point spatial position based on the eye ROI area; and
[0063] Step S33: updating the eye ROI region according to a preset rule, and performing eye tracking using the updated eye ROI region.
[0064] Specifically, the original image data includes an eye image. The corresponding interactive data may be the spatial position of the gaze point obtained based on the eye image. Specifically, the IR camera in the active detection unit may be used to collect the pupil image of the human eye and send it to the interactive information calculation unit; the interactive information calculation unit may use the received pupil image of the human eye to estimate the gaze point of the human eye, obtain the spatial position of the gaze point of the human eye, and track the eye ROI area.
[0065] For example, refer to Figure 5 As shown, for the interactive information calculation unit, the human eye ROI area can be initialized first. In the initial state, an initial image captured by the IR camera can be received, and the initial image can be an initial image corresponding to the entire field of view of the IR camera; the interactive information calculation unit can use the human eye detection model to directly perform human eye detection on the initial image, determine the human eye ROI area, and track the human eye ROI area; for example, refer to Figure 8 As shown, the eye ROI area is marked with a bounding box. Alternatively, in some embodiments, face detection may be first performed on the initial image to determine the face area; then the eye features may be accurately extracted and aligned on the face area image to achieve eye detection and determine the eye ROI area. Then, based on the determined eye ROI area, the trained gaze point estimation model may be used to calculate the gaze point spatial position to determine the gaze point spatial position information. At the same time, based on the currently determined eye ROI area information, the eye ROI area information may be updated. The updated eye ROI area information may be configured for the IR camera as the ROI area, so that the IR camera can accurately capture the eye image according to the configured ROI area information in subsequent image acquisition without the need to capture the complete scene image or face image.
[0066] In this example implementation, the above step S12 may further include:
[0067] Step S41, performing image recognition on the gesture image to determine a hand ROI area;
[0068] Step S42, determining a corresponding hand spatial position based on the hand ROI region pair;
[0069] Step S43, updating the hand ROI region according to a preset rule, and using the updated hand ROI region to perform hand tracking; and
[0070] Step S44: using a gesture recognition model to identify the hand ROI area to determine the corresponding gesture category.
[0071] Specifically, the original image data may also include a gesture image. For the interactive information calculation unit, the depth image collected in real time by the depth camera in the active detection unit may be received. Image recognition is performed on the depth image to determine the hand ROI area in the depth image and the corresponding gesture type. Based on the determined hand ROI area, the spatial position of the hand of the human hand may be calculated and the hand may be tracked.
[0072] For example, refer to Figure 6 As shown, for the interactive information calculation unit, it can be initialized first; in the initial state, receive the initial image captured by the depth camera in the active detection unit; wherein, the initial image can be a depth image captured by the depth camera of all contents within the current field of view. For the depth image, the trained recognition model can be used to perform human hand detection on the depth image to determine the hand ROI area in the depth image; for example, refer to Fig. 9 As shown, the hand ROI area is marked with a bounding box; based on the hand ROI area information and the corresponding depth information, the hand spatial position information can be calculated; at the same time, the trained gesture recognition model can be used to perform gesture recognition on the hand ROI area to determine the corresponding gesture type. For the currently acquired hand ROI area information, the ROI area can be updated to determine whether the hand area is tracked successfully in the current depth image. If the current tracking is normal, the updated ROI area can be used to collect the next frame of depth image; at the same time, the updated ROI area information can be used to configure the image acquisition range and position coordinates of the depth camera in the next frame of image. Alternatively, if it is determined that the current frame image tracking fails, that is, the hand information is not recognized in the depth image, it can be re-initialized, the initial depth image can be collected, and the above method flow can be re-executed to obtain the hand spatial position information.
[0073] In step S13, the interaction data is sent to a content generation unit to obtain content to be displayed rendered by the content generation unit based on the interaction data.
[0074] In this example implementation, the interaction information calculation unit receives a synchronization signal from the display data processing unit, and then transmits the calculated face spatial position (x1, y1, z1), gaze point spatial position (x2, y2, z2), gesture spatial position (x3, y3, z3) & gesture category to the content generation unit via USB / WIFI for content generation; at the same time, it can also be transmitted to the display data processing unit via I2C / SPI for content processing.
[0075] In this example implementation, the content generation unit renders the content to be displayed based on the interaction data, including: calculating the overlapping area according to the spatial position of the gesture and the position of the currently displayed content; when it is determined that there is an overlapping area, determining the updated content corresponding to the currently displayed content according to the gesture category; and rendering the updated content to obtain the content to be displayed.
[0076] Specifically, for the collected gesture images of the user, a user hand coordinate system is established, and the spatial position of the gesture is marked in the coordinate system; for the current display content on the screen, a screen coordinate system can be established, and the spatial position of the current display content can be marked; based on the coordinate system conversion matrix pre-calculated between the hand coordinate system and the screen coordinate system, the hand spatial position is converted to the screen coordinate system, so that the coordinate intersection between the user's hand and the currently displayed virtual 3D display content can be calculated; if there is an intersection of coordinates, it means that the two overlap, and the user has interactive behavior with the virtual object; if the positions of the two do not overlap, it is determined that the user's hand has not interacted with the virtual object. When there is interaction, the current display content can be updated according to the recognized user gesture category. For example, the user's gesture category can be single-click, double-click, rotate, etc.; correspondingly, the position and state of the currently displayed virtual 3D display can be updated according to preset rules to obtain updated content.
[0077] In this example implementation, the updated content is rendered to obtain the content to be displayed, including: determining a high-definition area and a low-definition area of the updated content according to the spatial position of the gaze point; and rendering the high-definition area and the low-definition area respectively to obtain the content to be displayed.
[0078] Specifically, for the virtual 3D display content currently displayed on the screen, n Views can be configured; where n is a positive integer. Based on the identified spatial position of the human eye, n / 2 viewpoints can be configured for the left eye and the right eye respectively. When using multi-viewpoint rendering, the focus area corresponding to the current display content can be calculated according to the gaze position of the human eye, and the focus area can be configured as a high-definition area for high-resolution rendering; for other areas, they can be used as low-definition areas and rendered with low definition. The high-definition area is then combined with the low-definition area to obtain the content to be displayed.
[0079] For example, the content generation unit can be an intelligent terminal device, such as a computer, a tablet computer, or a server that performs display content calculations. The content generation unit is mainly used to generate content required for light field display. Specifically, the content generation unit can obtain 3D display content by ray tracing. Generally speaking, the ray tracing method may include the following steps: a. Emitting multiple light rays from the center of the human eye / the center of the face / the center of the eyebrows; b. Extending the light to connect the center of the lens and the center of the sub-pixel; c. The light continues to extend through the sub-pixel until it collides with the object, and assigns the color and brightness information of the collided object to the corresponding pixel; d. Obtaining the 3D display content that matches the panel.
[0080] In this example implementation, the method further includes: configuring a rendering mode of the content generation unit in response to a configuration instruction.
[0081] Specifically, users can configure the rendering mode of the displayed content on the terminal device. Specifically, the rendering mode may include a low-definition rendering mode, a high-definition rendering mode, and a gaze point rendering mode; wherein the low-definition rendering mode may be configured with a relatively low resolution for the displayed content, thereby reducing rendering requirements and transmission bandwidth; the high-definition rendering mode may be configured with a relatively high resolution for the displayed content; the gaze point rendering mode may be based on the user's gaze point information, using a high-definition mode to render the gaze point area of the displayed content, and using a low-definition mode to render other areas.
[0082] In step S14, data is rearranged according to the content to be displayed in combination with the interaction data to obtain light field data, and the light field data is displayed.
[0083] In this example implementation, a display data processing unit may be provided. The display data processing unit, on the one hand, sends a synchronization signal to the interaction information calculation unit, and receives the interaction data sent by the interaction information calculation unit according to the synchronization timing, including: the spatial position of the face, the spatial position of the gaze point, and the spatial position of the gesture; on the other hand, receives the display content sent from the content processing unit; then rearranges the data according to the interaction information, content, and the pixel arrangement of the light field display to meet the data requirements of the light field display, and then sends the data to the light field display unit.
[0084] Among them, the light field display unit is used to receive the data processed by the display data processing unit, and display it to restore the real light field information; the light field display unit is composed of an ultra-high-resolution display panel and an optical light control device, wherein the panel pixel arrangement can be conventional RGB or pixel island RGB arrangement; the optical light control device can be a physical lens or a liquid crystal lens light control. Generally speaking, light field display is a technology that constructs 3D objects through ray tracing. Its principle can be expressed based on a full optical function containing 7 dimensional parameters, including the position of the eye (x, y, z), the horizontal angle θ of the light, the vertical angle φ of the light, and the wavelength λ of the light, which is used to indicate that the light intensity will change with time t.
[0085] For example, a multi-viewpoint fusion algorithm can be used to achieve multi-viewpoint naked-eye 3D display image fusion to obtain light field data. Taking 5 viewpoints as an example, the multi-viewpoint image fusion algorithm may include: 1) determining the tilt angle α, grating period D and sub-pixel width Dh of the cylindrical grating arrangement; calculating the number of sub-pixel points N covered by a grating period; 2) calculating the sub-pixel mapping table through the tilt angle α, the number of sub-pixel points N, and the sub-pixel row and column coordinates (k, l); 3) arranging the 5 viewpoint images vertically into a complete image according to the resolution of the display screen; 4) converting the image color space from YCrCb to RGB; 5) resizing the image and rotating it 90 degrees clockwise; 6) filling the RGB sub-pixel components of the 5 viewpoints into the corresponding RGB components of the multi-viewpoint stereo composite image according to the obtained sub-pixel mapping table. Generally speaking, for a grating naked-eye 3D display, the gratings are arranged at a certain tilt angle, and the grating pitch is set to D, which can cover N sub-pixel points in the horizontal direction. The multi-view sub-pixel mapping algorithm can be used to determine the corresponding components of the pixel point in the view from which the RGB components of the pixel point P at a certain position on a given display screen should be taken.
[0086] In addition, it is also possible to determine whether both eyes of the user are located at the best viewpoint based on the coordinates of the user's eyes and the coordinates of multiple viewpoints.
[0087] The light field display method provided in the embodiment of the present disclosure is applicable to the light field display system, and by integrating the active detection unit, the interactive information calculation unit, and the display data processing unit at the display end, the information processing and transmission efficiency is improved, the information transmission efficiency is improved, the delay is reduced, and the data processing pressure at the system end is reduced, thereby improving the user experience of 3D display.
[0088] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.
[0089] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.
[0090] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0091] The units involved in the embodiments of the present invention may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, limit the units themselves.
[0092] It should be noted that, as another aspect, the present disclosure further provides a storage medium, which may be included in an electronic device; or may exist independently without being assembled into the electronic device. The above storage medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiment.
[0093] It should be noted that the storage medium may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any storage medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.
[0094] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.
[0095] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.
Claims
1. A light field display method, characterized in that: Applied to a light field display system, the method comprises: Collecting raw image data; Identify the original image data to obtain corresponding interaction data; wherein the interaction data includes: any one or a combination of any two of the following: face spatial position, gaze point spatial position, gesture spatial position and gesture category; Sending the interaction data to a content generation unit to obtain content to be displayed rendered by the content generation unit based on the interaction data; Data is rearranged according to the content to be displayed in combination with the interaction data to obtain light field data, and the light field data is displayed.
2. The light field display method according to claim 1, characterized in that: The content generation unit renders the content to be displayed based on the interaction data, including: Calculate the overlapping area based on the gesture spatial position and the current display content position; When it is determined that there is an overlapping area, the update content corresponding to the currently displayed content is determined according to the gesture category; and the update content is rendered to obtain the content to be displayed.
3. The light field display method according to claim 2, characterized in that: The updating content is rendered to obtain the content to be displayed, including: Determining a high-definition area and a low-definition area of the updated content according to the spatial position of the gaze point; The high-definition area and the low-definition area are rendered respectively to obtain the content to be displayed.
4. The light field display method according to claim 1, characterized in that: The original image data includes a face image; The identifying the original image data to obtain corresponding interaction data includes: Recognize the face image to determine the face ROI area in the face image; Determine the corresponding face spatial position based on the face ROI area; and The face ROI area is updated according to preset rules, and face tracking is performed based on the updated face ROI area.
5. The light field display method according to claim 1, characterized in that: The original image data includes an eye image; The identifying the original image data to obtain corresponding interaction data includes: Performing image recognition on the eye image to determine an eye ROI region in the eye image; Determine the corresponding gaze point spatial position based on the eye ROI area; and The eye ROI region is updated according to a preset rule, and the updated eye ROI region is used for eye tracking.
6. The light field display method according to claim 1, characterized in that: The original image data includes a gesture image; The identifying the original image data to obtain corresponding interaction data includes: Performing image recognition on the gesture image to determine a hand ROI area; Determining a corresponding hand spatial position based on the hand ROI region pair; updating the hand ROI region according to a preset rule, and using the updated hand ROI region to perform hand tracking; and The hand ROI region is identified using a gesture recognition model to determine a corresponding gesture category.
7. The light field display method according to claim 1, characterized in that: The method further comprises: In response to the trigger operation, a corresponding ROI area is configured for each sensor used to collect raw image data.
8. The light field display method according to claim 1, characterized in that: The method further comprises: In response to the configuration instruction, a rendering mode of the content generation unit is configured.
9. The light field display method according to claim 1, characterized in that: The collecting of raw image data comprises: Use several grayscale cameras to collect face images, use IR cameras to collect eye images, and use depth cameras to collect gesture images; Several grayscale cameras, IR cameras, and depth cameras are arranged on one side of the light field display unit.
10. A light field display system, characterized in that: The system comprises: An active detection unit, used for collecting raw image data; An interaction information calculation unit is used to receive the original image data, identify the original image data to obtain corresponding interaction data; and send the interaction data to the content generation unit; wherein the interaction data includes: any one or a combination of any one of the face space position, gaze point space position, gesture space position and gesture category; A display data processing unit, configured to receive the to-be-displayed content rendered by the content generation unit, and to rearrange the data in combination with the interaction data and the pixel arrangement map to obtain light field data; A light field display unit is used to display the light field data.
11. The light field display system according to claim 10, characterized in that: The interaction information calculation unit configures the ROI area of each sensor in the active detection unit in response to the trigger information.
12. The light field display system according to claim 10, characterized in that: The display data processing unit sends a synchronization signal to the interaction information calculation unit, so that the interaction information calculation unit completes the calculation of the interaction data in response to the synchronization information.
13. The light field display system according to claim 10, characterized in that the interactive active detection unit is arranged on one side of the light field display unit, and comprises a plurality of grayscale cameras, IR cameras, and depth cameras; A grayscale camera is used to capture face images, an IR camera is used to capture eye images, and a depth camera is used to capture gesture images.
14. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the light field display method according to any one of claims 1 to 9 is implemented.
Citation Information
Cited By
Light field display method and system, and storage medium
EP4636545A1
