Method and apparatus for controlling multi-view video rendering in a multi-display environment

Abstract

A method and apparatus for intelligently expanding the range of stereoscopic image observation in a multi-view stereoscopic display system and user environment are disclosed. A method for intelligently expanding the range of stereoscopic image observation in a multi-view stereoscopic display system and user environment according to one embodiment of the present invention may include: a step of determining a primary user among at least one user viewing an image displayed on each multi-view display based on images captured by a plurality of cameras attached to a plurality of multi-view displays; a step of analyzing the primary user's viewing intention regarding the image displayed on each multi-view display based on information of at least one of the primary user's body, head, and eyes for each multi-view display; and a step of controlling the rendering of the image displayed on each multi-view display based on the analysis result of the primary user's viewing intention for each multi-view display.

Description

Technology Field

[0001] This relates to a method and device for controlling the rendering of multi-view images. Background Technology

[0002] Stereoscopic display technology provides 3D stereoscopic images, enabling users to have an immersive viewing experience. While this technology is utilized in various fields such as medicine, entertainment, and education, existing technologies are designed around a single user, making it difficult to provide an efficient viewing experience in environments where multiple users watch simultaneously.

[0003] One of the problems with existing technology is that due to limited viewing angles, stereoscopic images are distorted or cannot be seen properly unless the user is in a specific position. Furthermore, the lack of technology capable of dynamically adjusting the effective viewing area based on the user's position or viewing direction has made it difficult to provide an optimal viewing environment that considers each user's location and viewing orientation when multiple users watch stereoscopic images simultaneously.

[0004] In particular, in medical environments, there are often cases where accurate 3D images must be viewed in real time from various angles during surgery. However, existing 3D display systems can only provide accurate images from limited angles, so medical staff had to spend a lot of time adjusting the display or require complex equipment to view images from the desired angle. The problem to be solved

[0005] The present invention provides a method and apparatus for controlling the rendering of multi-view images in a multi-display environment. The invention is not limited to the technical problems described above, and other technical problems may be derived from the following description. means of solving the problem

[0006] A multi-view display rendering control method according to the present invention comprises: a step of determining a primary user among at least one user viewing an image displayed on each multi-view display based on images captured by a plurality of cameras attached to a plurality of multi-view displays; a step of analyzing the primary user's viewing intention regarding the image displayed on each multi-view display based on information of at least one of the primary user's body, head, and eyes for each multi-view display; and a step of controlling the rendering of the image displayed on each multi-view display based on the analysis result of the primary user's viewing intention for each multi-view display.

[0007] The step of determining the primary user may include: tracking the eye position and gaze direction of each of at least one user viewing an image displayed on one of the multiple multi-view displays using a captured image from one of the multiple multi-view displays attached to one of the multiple multi-view displays; setting a priority among at least one user viewing an image displayed on one of the multiple multi-view displays based on the tracked eye position and gaze direction of each of the at least one user; and selecting a primary user among at least one user viewing an image displayed on one of the multiple multi-view displays by comparing the priority among at least one user viewing an image displayed on one of the multiple multi-view displays.

[0008] The step of tracking the eye position and gaze direction of each of the at least one user can track the eye position and gaze direction of each of the at least one user by inputting each of the plurality of image frames extracted from the captured image of any one camera into the first artificial intelligence model, thereby obtaining the eye position and gaze direction of each of the at least one user represented by each of the plurality of image frames from the first artificial intelligence model.

[0009] The step of setting priority among at least one user may determine whether the gaze direction of each of the tracked at least one user is a gaze direction toward any one multi-view display, assign the same rank score to at least one user having a gaze direction toward any one multi-view display, and additionally assign an additive rank score to at least one user assigned the same rank score according to the distance between the eye position of each of the at least one user assigned the same rank score and any one multi-view display.

[0010] The step of analyzing the viewing intention of the primary user can be performed by inputting each of the plurality of video frames extracted from the captured video of any one camera into the second artificial intelligence model to obtain body position, head position, eye position, and posture information of the primary user of any one multi-view display from the second artificial intelligence model, and analyzing the viewing intention of the primary user of any one multi-view display based on the obtained body position, head position, eye position, and posture information of the primary user.

[0011] The step of controlling the rendering of an image displayed on the multi-view display can control the rendering of the image displayed on the multi-view display so that the specific object is reduced or enlarged if the analysis result of the primary user's viewing intention of the multi-view display indicates an intention to reduce or enlarge a specific object within the image displayed on the multi-view display.

[0012] The step of controlling the rendering of an image displayed on the multi-view display can control the rendering of the image displayed on the multi-view display so that the specific object is rotated if the result of the analysis of the viewing intention of the primary user of the multi-view display indicates an intention to rotate the specific object within the image displayed on the multi-view display.

[0013] A multi-view display rendering control device according to the present invention comprises: a primary user determination unit that determines a primary user among at least one user viewing an image displayed on each multi-view display based on images captured by a plurality of cameras attached to a plurality of multi-view displays; a rendering control unit that analyzes the viewing intention of the primary user regarding the image displayed on each multi-view display based on information of at least one of the body, head, and eye of the primary user for each multi-view display; and a step of controlling the rendering of the image displayed on each multi-view display based on the analysis result of the viewing intention of the primary user for each multi-view display. Effects of the invention

[0014] Based on images captured by multiple cameras attached to multiple multi-view displays, a primary user is identified among at least one user viewing an image displayed on each multi-view display; the primary user's viewing intention regarding the image displayed on each multi-view display is analyzed based on information regarding at least one of the primary user's body, head, or eyes; and the rendering of the image displayed on each multi-view display is controlled based on the analysis results regarding the primary user's viewing intention, thereby expanding the viewing range of stereoscopic images and minimizing interference by other users in a multi-user environment by reflecting the primary user's viewing intention. This multi-view display rendering control method according to the present invention improves the quality of stereoscopic images by reflecting the primary user's viewing intention in the medical field and enables non-contact control with minimal head movement.

[0015] The effects are not limited to those described above, and other effects may be derived from the following description. Brief explanation of the drawing

[0016] FIG. 1 is a configuration diagram of a multi-view display rendering control device according to one embodiment of the present invention. FIG. 2 is a flowchart of a multi-view display rendering control method according to an embodiment of the present invention. FIG. 3 is an example diagram for explaining the primary user identification criteria of the primary user identification unit (20) shown in FIG. 1. FIG. 4 is a diagram showing an embodiment of the viewing intention analysis unit (30) and rendering control unit (40) illustrated in FIG. 1, which analyzes the viewing intention of a primary user and controls a three-dimensional object according to the analyzed intention. Specific details for implementing the invention

[0017] Embodiments of the present invention will be described in detail below with reference to the drawings. The embodiments of the present invention described below relate to an apparatus and method for expanding the viewing range of stereoscopic images in a multi-view stereoscopic display and a multi-user environment. Hereinafter, such a method and apparatus will be briefly referred to as a “multi-view display rendering control method” and a “multi-view display rendering control apparatus.”

[0018] FIG. 1 is a configuration diagram of a multi-view display rendering control device according to an embodiment of the present invention. Referring to FIG. 1, the multi-view display rendering control device according to the present embodiment is composed of a primary user identification unit (20), a first artificial intelligence model (25), a viewing intention analysis unit (30), a second artificial intelligence model (35), and a rendering control unit (40). FIG. 1 illustrates the essential components of the present embodiment so that the present embodiment can be easily understood without obscuring the features of the present embodiment. A person skilled in the art to which the present embodiment belongs will understand that other components may be added in addition to the components shown in FIG. 1.

[0019] FIG. 2 is a flowchart of a multi-view display rendering control method according to an embodiment of the present invention. Referring to FIG. 2, the multi-view display rendering control method according to the present embodiment consists of the following, which are executed by the multi-view display rendering control device illustrated in FIG. 1.

[0020] Steps 61 to 63 involve the main user identification unit (20) identifying the main user among at least one user who is watching the video displayed on each multi-view display (51 to 53) based on the captured video of a plurality of cameras (11 to 13) attached to a plurality of multi-view displays (51 to 53).

[0021] In step 61, the primary user identification unit (20) tracks the eye position and gaze direction of each at least one user who is watching an image displayed on one of the multiple multi-view displays (51-53) by using a captured image from one of the multiple multi-view displays (51-53) attached to one of the multiple multi-view displays (51-53). The primary user identification unit (20) tracks the eye position and gaze direction of each at least one user by inputting each of the multiple image frames extracted from the captured image of one of the cameras (11-13) into the first artificial intelligence model (25), thereby obtaining the eye position and gaze direction of each at least one user represented by each of the multiple image frames from the first artificial intelligence model (25).

[0022] In step 62, the primary user identification unit (20) sets a priority among at least one user watching an image displayed on one of the multi-view displays (51-53) based on the eye position and gaze direction of each of the at least one user tracked in step 61. The primary user identification unit (20) determines whether the gaze direction of each of the at least one tracked user is a gaze direction toward one of the multi-view displays (51-53), assigns the same rank score to at least one user having a gaze direction toward one of the multi-view displays (51-53), and additionally assigns an additional rank score to at least one user assigned the same rank score according to the distance between the eye position of each of the at least one user assigned the same rank score and one of the multi-view displays (51-53).

[0023] In step 63, the primary user identification unit (20) selects a primary user among at least one user watching a video displayed on one of the multi-view displays (51-53) by comparing the priority among at least one user watching a video displayed on one of the multi-view displays (51-53) based on the priority set in step 62. FIG. 3 is an example diagram for explaining the primary user identification criteria of the primary user identification unit (20) illustrated in FIG. 1. As shown in FIG. 3 (a), the primary user identification unit (20) searches for people within the field of view of the camera (11) and identifies a primary user by tracking the distance (location recognition-based) and gaze direction (gaze point recognition-based) of the person detected from one of the multi-view displays (51).

[0024] Among the users detected by the camera (11) in FIG. 3 (a), the person at a distance of 0.7m is the closest position but is not the primary user because he is not looking at the single multi-view display (51). Instead, the primary user determination unit (20) recognizes the person at a distance of 1.0m who is looking at the single multi-view display (51) as the primary user. When multiple users existing within the field of view of the camera (11) are looking at the single multi-view display (51), the primary user determination unit (20) selects the closest user among the multiple users as the primary user.

[0025] Referring to FIG. 3(b), the primary user identification unit (20) has one RGB-D camera (11) corresponding to the camera (11) installed on one multi-view display (51), and a dynamic effective viewing area centered on the primary user is formed. Here, the dynamic effective viewing area refers to a viewing area that changes in real time depending on the location of the primary user. The primary user identification unit (20) can obtain temporary primary user authority by having a secondary user look at another multi-view display (52) other than the one multi-view display (51) that the primary user is looking at. Here, a secondary user is a user other than the primary user, and refers to a user who can obtain primary user authority when the primary user does not look at one multi-view display (51) or looks at another multi-view display (52) different from the primary user.

[0026] In step 64, the viewing intention analysis unit (30) analyzes the viewing intention of the main user for the images displayed on each multi-view display (51-53) based on information of at least one of the main user's body, head, and eyes for each multi-view display (51-53). Specifically, the viewing intention analysis unit (30) receives as input data the direction angle data (pitch, roll, yaw) of the main user's head looking at the multi-view display and the coordinates of the main user's posture information data (coordinates of the center of both eyes, mouth, left shoulder, right shoulder, torso), and analyzes the data. The viewing intention analysis unit (30) inputs each of the multiple video frames extracted from the captured video of one camera (11-13) into the second artificial intelligence model (35) to obtain body position, head position, eye position, and posture information of the main user of one multi-view display (51-53) from the second artificial intelligence model (35), and analyzes the viewing intention of the main user of one multi-view display (51-53) based on the obtained body position, head position, eye position, and posture information of the main user.

[0027] In step 65, the rendering control unit (40) controls the rendering of the video displayed on each multi-view display (51-53) based on the analysis results of the main user's viewing intention for each multi-view display (51-53) by the viewing intention analysis unit (30) in step 64.

[0028] The rendering control unit (40) controls the rendering of the image displayed on the multi-view display (51-53) so that the specific object is reduced or enlarged when the analysis result of the main user's viewing intention of one multi-view display (51-53) indicates an intention to reduce or enlarge a specific object within the image displayed on the multi-view display (51-53).

[0029] The rendering control unit (40) controls the rendering of the image displayed on the multi-view display (51-53) so that the specific object is rotated when the result of the analysis of the viewing intention of the main user of the multi-view display (51-53) indicates an intention to rotate the specific object within the image displayed on the multi-view display (51-53).

[0030] FIG. 4 is a diagram illustrating an embodiment of the viewing intention analysis unit (30) and rendering control unit (40) shown in FIG. 1, showing the process of analyzing the viewing intention of a primary user and controlling a 3D object according to the analyzed intention. According to FIG. 4 (a), one RGB-D camera (11) is installed above one multi-view display (51), and one RGB-D camera (11) captures an image of a primary user looking at one multi-view display (51). Depending on the position where the primary user gazes at one multi-view display (51), it is divided into four areas: A, B, C, and D, and the viewing intention analysis unit (30) and the rendering control unit (40) operate accordingly. When the user's eyes are located at A and the primary user gazes at the 3D object with the intention of wanting to see the 3D object larger, the viewing intention analysis unit (30) analyzes this intention, and the rendering control unit (40) performs the function of enlarging the 3D object.

[0031] When the user's eyes are positioned at B and the user gazes at the 3D object with the intention of looking at the left side of the 3D object, the viewing intention analysis unit (30) analyzes this intention, and the rendering control unit (40) performs the function of rotating the 3D object to the right. When the user's eyes are positioned at C and the main user gazes at the 3D object with the intention of looking at the right side of the 3D object, the viewing intention analysis unit (30) analyzes this intention, and the rendering control unit (40) performs the function of rotating the 3D object to the left. When the user's eyes are positioned at D and the main user gazes at the 3D object with the intention of looking at the 3D object in its entirety, the viewing intention analysis unit (30) analyzes this intention, and the rendering control unit (40) performs the function of shrinking the 3D object.

[0032] FIG. 4(b) shows a front view of a single RGB-D camera (11), and the rendering control unit (40) adjusts the rotation of a 3D object according to the position of the main user's eyes relative to the height center of a single multi-view display (51). When the user's eyes are positioned at height E and the user gazes at the 3D object with the intention of looking at the upper part of the 3D object, the viewing intention analysis unit (30) analyzes this intention, and the rendering control unit (40) performs the function of rotating the 3D object downward. When the user's eyes are positioned at height F and the user gazes at the 3D object with the intention of looking at the lower part of the 3D object, the viewing intention analysis unit (30) analyzes this intention, and the rendering control unit (40) performs the function of rotating the 3D object upward.

[0033] The present invention has been described above with reference to its preferred embodiments. Those skilled in the art will understand that the present invention may be embodied in modified forms without departing from the essential characteristics of the invention. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the invention is defined by the claims, not by the foregoing description, and all variations within the scope of the claims should be interpreted as being included in the invention. Explanation of the symbols

[0034] 11 ... camera 12 ... camera 13 ... camera 20 ... Primary User Identification Unit 25 ... 1st Artificial Intelligence Model 30 ... Viewing Intention Analysis Department 35 ... 2nd Artificial Intelligence Model 40 ... Rendering control unit 51 ... multi-view display 52 ... multi-view display 53 ... multi-view display

Claims

Claim 1 A multi-view display rendering control method comprising: a step of determining a primary user among at least one user viewing an image displayed on each multi-view display based on images captured by a plurality of cameras attached to a plurality of multi-view displays; a step of analyzing the primary user's viewing intention regarding the image displayed on each multi-view display based on information of at least one of the primary user's body, head, and eyes for each multi-view display; and a step of controlling the rendering of the image displayed on each multi-view display based on the analysis result of the primary user's viewing intention for each multi-view display, wherein the step of analyzing the primary user's viewing intention is characterized by obtaining body position, head position, eye position, and posture information of the primary user of one multi-view display from the second artificial intelligence model by inputting each of a plurality of image frames extracted from the images captured by one of the cameras into the second artificial intelligence model, and analyzing the primary user's viewing intention of one multi-view display based on the obtained body position, head position, eye position, and posture information of the primary user. Claim 2 A multi-view display rendering control method according to claim 1, wherein the step of determining the primary user comprises: tracking the eye position and gaze direction of each of at least one user viewing an image displayed on one of the multiple multi-view displays using a captured image of one camera attached to one of the multiple multi-view displays; setting a priority among at least one user viewing an image displayed on one of the multiple multi-view displays based on the tracked eye position and gaze direction of each of the at least one user; and selecting a primary user among at least one user viewing an image displayed on one of the multiple multi-view displays by comparing the priority among at least one user viewing an image displayed on one of the multiple multi-view displays. Claim 3 In claim 2, the step of tracking the eye position and gaze direction of each of the at least one user is characterized by tracking the eye position and gaze direction of each of the at least one user by inputting each of the plurality of image frames extracted from the captured image of any one camera into the first artificial intelligence model, thereby obtaining the eye position and gaze direction of each of the at least one user represented by each of the plurality of image frames from the first artificial intelligence model. Claim 4 In claim 2, the step of setting priority among at least one user is characterized by determining whether the gaze direction of each of the tracked at least one user is a gaze direction toward any one multi-view display, assigning the same rank score to at least one user having a gaze direction toward any one multi-view display, and additionally assigning an additive rank score to at least one user assigned the same rank score according to the distance between the eye position of each of the at least one user assigned the same rank score and the any one multi-view display. Claim 5 delete Claim 6 A method for controlling rendering of a multi-view display according to claim 1, wherein the step of controlling the rendering of an image displayed on the multi-view display is characterized by controlling the rendering of the image displayed on the multi-view display such that the specific object is reduced or enlarged when the analysis result of the viewing intention of the primary user of the multi-view display indicates an intention to reduce or enlarge the specific object within the image displayed on the multi-view display. Claim 7 A method for controlling rendering of a multi-view display according to claim 1, wherein the step of controlling the rendering of an image displayed on the multi-view display is characterized by controlling the rendering of an image displayed on the multi-view display such that the specific object is rotated when the result of analyzing the viewing intention of the primary user of the multi-view display indicates an intention to rotate the specific object within the image displayed on the multi-view display. Claim 8 A multi-view display rendering control device comprising: a primary user determination unit that determines a primary user among at least one user viewing an image displayed on each multi-view display based on images captured by a plurality of cameras attached to a plurality of multi-view displays; a viewing intention analysis unit that analyzes the viewing intention of the primary user regarding the image displayed on each multi-view display based on information of at least one of the body, head, and eye of the primary user for each multi-view display; and a rendering control unit that controls the rendering of the image displayed on each multi-view display based on the analysis result of the viewing intention of the primary user for each multi-view display, wherein the viewing intention analysis unit acquires body position, head position, eye position, and posture information of the primary user of one multi-view display from the second artificial intelligence model by inputting each of a plurality of image frames extracted from the images captured by any one camera into the second artificial intelligence model, and analyzes the viewing intention of the primary user of one multi-view display based on the acquired body position, head position, eye position, and posture information of the primary user.

Images