Stereoscopic image display device and control method thereof

Abstract

A stereoscopic image display device and a control method thereof may be provided. Specifically, a stereoscopic image display device and a control method thereof may be provided, wherein the stereoscopic image display device obtains a depth map from a planar image, obtains feature information associated with a first depth from the depth map, obtains context information of the planar image, calculates an estimated fatigue of a user on the basis of the depth map, the feature information, and the context information, the estimated fatigue indicating the visual comfort (VC) the user feels when viewing a first stereoscopic image, corrects the depth map on the basis of the estimated fatigue and a target fatigue, generates a left-eye image by correcting the planar image on the basis of the corrected depth map, generates a right-eye image by warping the left-eye image, and displays a second stereoscopic image by outputting the left-eye image and the right-eye image.

Description

Stereoscopic image display device and control method thereof

[0001] Embodiments of the present disclosure relate to a stereoscopic image display device and a control method thereof.

[0002] A stereoscopic display device can be a device that converts a flat image into a stereoscopic image by utilizing binocular parallax. Stereoscopic images can provide a greater sense of immersion than flat images by providing a sense of depth. Stereoscopic display devices are widely used in applications such as head-mounted displays (HMDs), movies, and exhibition hall videos.

[0003] Stereoscopic images can provide users with a sense of depth or disparity, depending on the degree of depth or disparity. The greater the depth of a stereoscopic image, the easier it is for users to perceive its depth. Stereoscopic images can also cause visual comfort (VC) depending on the degree of depth. As the depth of a stereoscopic image increases, visual fatigue may also increase. There can be a trade-off between stereoscopic effects and visual fatigue.

[0004] Existing stereoscopic display devices estimate the depth of a flat image and generate a stereoscopic image based on a specified baseline, regardless of visual fatigue. Consequently, stereoscopic images displayed on existing stereoscopic display devices can be visually tiring for users.

[0005] According to one embodiment of the present disclosure, a stereoscopic image display device includes a display; a memory storing at least one instruction; and at least one processor, wherein the at least one instruction, when executed by the at least one processor, causes the at least one processor to obtain a depth map from a planar image, obtain characteristic information related to a first depth from the depth map, obtain context information of the planar image, calculate an estimated fatigue of a user based on the depth map, the characteristic information, and the context information, wherein the estimated fatigue represents visual comfort (VC) felt by the user when viewing a first stereoscopic image, correct the depth map based on the estimated fatigue and a target fatigue (target VC), correct the planar image based on the corrected depth map to generate a left-eye image, warp the left-eye image to generate a right-eye image, and output the left-eye image and the right-eye image so that the display displays a second stereoscopic image.

[0006] A method for controlling a stereoscopic image display device according to one embodiment of the present disclosure may include: obtaining a depth map from a planar image; obtaining characteristic information related to a first depth from the depth map; obtaining context information of the planar image; calculating an estimated fatigue of a user based on the depth map, the characteristic information, and the context information, the estimated fatigue indicating visual fatigue felt by the user when viewing a first stereoscopic image; correcting the depth map based on the estimated fatigue and the target fatigue; correcting the planar image based on the corrected depth map to generate a left-eye image; warping the left-eye image to generate a right-eye image; and outputting the left-eye image and the right-eye image to display a second stereoscopic image.

[0007] FIG. 1 is a drawing showing a stereoscopic image display device according to one embodiment of the present disclosure.

[0008] FIG. 2 is a block diagram illustrating a stereoscopic image display device according to one embodiment of the present disclosure.

[0009] FIG. 3 is a flowchart illustrating a control method of a stereoscopic image display device according to one embodiment of the present disclosure.

[0010] FIG. 4 is a block diagram showing a processor of a stereoscopic image display device according to one embodiment of the present disclosure as a functional unit.

[0011] FIG. 5 is a diagram illustrating an inference module and a target fatigue score network of a stereoscopic image display device according to one embodiment of the present disclosure.

[0012] FIG. 6 is a drawing showing a stereoscopic image display device according to one embodiment of the present disclosure generating a stereoscopic image based on a flat image.

[0013] FIG. 7 is a drawing showing a stereoscopic image displayed by a stereoscopic image display device according to one embodiment of the present disclosure while taking visual fatigue into consideration.

[0014] FIG. 8 is a block diagram showing functional units for depth control in consideration of a user's gaze included in a processor of a stereoscopic image display device according to one embodiment of the present disclosure.

[0015] FIG. 9 is a drawing showing a stereoscopic image displayed by a stereoscopic image display device according to one embodiment of the present disclosure while taking into consideration the user's line of sight.

[0016] FIG. 10 is a flowchart illustrating a method for a stereoscopic image display device to display a stereoscopic image by taking into account a user's line of sight according to an embodiment of the present disclosure.

[0017] FIG. 11 is a flowchart illustrating a method for correcting the color of a stereoscopic image by taking into account the user's line of sight in a stereoscopic image display device according to one embodiment of the present disclosure.

[0018] The terms used in this disclosure will be briefly explained, and one embodiment of the present disclosure will be specifically described.

[0019] The terms used in this disclosure are selected from widely used, current terms, taking into account the functions of one embodiment of the disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant embodiments of the disclosure. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on the meanings of the terms and the overall content of the disclosure.

[0020] Throughout this disclosure, when a part is said to "include" a component, this does not exclude other components, but rather implies the inclusion of other components, unless otherwise specifically stated. Furthermore, terms such as "part," "module," and the like described herein refer to a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software.

[0021] Below, with reference to the attached drawings, embodiments of the present disclosure are described in detail so that those skilled in the art can easily implement the present disclosure. However, one embodiment of the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts irrelevant to the description are omitted to clearly describe one embodiment of the present disclosure, and similar parts are designated with similar drawing reference numerals throughout the present disclosure.

[0022] The present disclosure provides a technology for generating a stereoscopic image with low visual fatigue by adjusting depth based on a visual comfort (VC) estimation model when generating a stereoscopic image by inputting a flat image.

[0023] FIG. 1 is a drawing showing a stereoscopic image display device (100) according to one embodiment of the present disclosure.

[0024] A stereoscopic image display device (100) may be a device that converts a flat image into a stereoscopic image and displays it by utilizing binocular disparity. The stereoscopic image display device (100) may output a left-eye image and a right-eye image to a user (200). The stereoscopic image display device (100) may provide a left-eye image to the left eye (210) of the user (200). The stereoscopic image display device (100) may provide a right-eye image to the right eye (220) of the user (200). The stereoscopic image display device (100) may enable the user (200) to view a stereoscopic image according to binocular disparity.

[0025] Stereoscopic images can provide users with a sense of three-dimensionality, providing a higher level of immersion than flat images. Stereoscopic images can generate a sense of three-dimensionality through the difference between the left-eye and right-eye images. A stereoscopic image display device (100) may include a head-mounted display (HMD), a movie projection device, or an exhibition hall image display device. The stereoscopic image display device (100) can be widely used in situations where a spatial sensation is desired to be provided to the user.

[0026] Stereoscopic images can provide users with a sense of three-dimensionality based on the degree of depth or disparity. Depth can refer to the depth of an object included in a stereoscopic image. An object can refer to any object excluding the background of the stereoscopic image. Disparity can refer to the difference between the left-eye and right-eye images. As the three-dimensionality of a stereoscopic image increases, the user can easily perceive the depth of the stereoscopic image. Stereoscopic images can have visual comfort (VC) depending on the degree of three-dimensionality. Visual fatigue can refer to the fatigue felt by a user viewing a stereoscopic image. As the three-dimensionality of a stereoscopic image increases, visual fatigue can increase. Stereoscopic effect and visual fatigue can have a trade-off relationship.

[0027] FIG. 2 is a block diagram illustrating a stereoscopic image display device (100) according to one embodiment of the present disclosure. The stereoscopic image display device (100) according to one embodiment may include a display (110), a memory (120), and a processor (130).

[0028] In one embodiment, the display (110) can display a stereoscopic image to the outside of the stereoscopic image display device (100). The display (110) can receive data for displaying a stereoscopic image from the processor (130). The display (110) can display a stereoscopic image by utilizing the user's binocular parallax. The display (110) can output a left-eye image and a right-eye image so that the user can view the stereoscopic image. The display (110) can include a display panel that outputs the left-eye image and the right-eye image and a display driver integrated circuit (DDI) for driving the display panel. For example, the display (110) can be at least one of a liquid crystal display (LCD), an organic light emitting display (OLED), a quantum dot display (QD display), and a micro light emitting diode (Micro LED) display.

[0029] In one embodiment, the memory (120) can store at least one instruction. The at least one instruction can control the overall operation of the stereoscopic image display device (100). The memory (120) can store planar image data related to a planar image acquired by the stereoscopic image display device (100). The at least one instruction can cause the processor (130) to generate stereoscopic image data based on the planar image data. The memory (120) can include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD memory, XD memory), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk.

[0030] In one embodiment, the processor (130) may control the overall operation of the stereoscopic image display device (100). The processor (130) may execute at least one instruction stored in the memory (120). The at least one instruction, when executed by the processor (130), may cause the processor (130) to generate stereoscopic image data based on planar image data. The processor (130) may receive the planar image data from the memory (120). The processor (130) may generate stereoscopic image data based on the planar image data. The processor (130) may transmit the stereoscopic image data to the display (110). The processor (130) may include at least one of a central processing unit (CPU), an application processor (AP), a graphic processing unit (GPU), and an image signal processor (ISP).

[0031] In one embodiment, the processor (130) can obtain a depth map from a planar image. The depth map may be information indicating the depth of each pixel constituting the planar image. The processor (130) can obtain the depth map by extracting depth information of each pixel constituting the planar image. For example, the processor (130) can obtain the depth map by extracting depth information from planar image data stored in the memory (120). For example, the processor (130) can obtain the depth map by extracting depth information through a network that estimates the depth of the planar image.

[0032] In one embodiment, the processor (130) may obtain depth-related feature information from a depth map. The feature information may include weighting values ​​that contribute to visual fatigue estimation according to the depth of pixels included in a planar image. The feature information may include depth information indicating the depth of each region of the planar image and a confidence value that estimates the degree of depth error. The feature information may include factors important for visual fatigue estimation.

[0033] In one embodiment, the processor (130) may obtain context information of a planar image. Context information of a planar image may be defined as information representing features and situations extracted from the planar image. Context information of a planar image may be referred to as feature information of the planar image, feature values ​​of the planar image, or feature extraction results of the planar image.

[0034] In one embodiment, context information may include an object included in a planar image, the relative size of the object, and the location of the object. An object included in a planar image may refer to any object excluding the background of the planar image. An object may refer to an object that has the greatest influence on the characteristics of the planar image as measured by saliency. An object may be a feature value expressed in the form of a map. The relative size of an object may refer to a ratio of the sizes of each object included in the planar image. The location of an object may be a part of an object recognized as most important when estimating visual fatigue.

[0035] In one embodiment, the context information may include motion information related to the motion of the planar image and edge information for compensating for visual fatigue estimated based on an error in the depth of the planar image.

[0036] In one embodiment, the processor (130) may calculate an estimated user fatigue based on a depth map, feature information, and context information. The estimated fatigue may represent the visual fatigue felt by the user when viewing a stereoscopic image. For example, the estimated user fatigue may be the level of visual fatigue felt by the user when viewing a stereoscopic image. The processor (130) may extract factors important for fatigue estimation from a planar image and an estimated depth. The processor (130) may calculate the estimated user fatigue based on the extracted factors important for fatigue estimation. The processor (130) may estimate the fatigue level based on a weight that contributes to the visual fatigue estimation. The processor (130) may estimate the visual fatigue by simultaneously considering the importance of context information of the planar image and the depth of the planar image when estimating the visual fatigue. Accordingly, the processor (130) may estimate the visual fatigue in a manner that is robust to errors that occur when estimating depth from a planar image.

[0037] In one embodiment, the processor (130) can calibrate the depth map based on the estimated fatigue level and the target fatigue level (target VC). The target fatigue level may be a threshold fatigue level determined so that the user does not experience fatigue above a certain level. The processor (130) can receive the target fatigue level included in the user input. The processor (130) can adjust the stereoscopic effect of the stereoscopic image so that the estimated fatigue level corresponds to the target fatigue level. The processor (130) can calibrate the depth map so that the adjusted stereoscopic effect corresponds to the adjusted stereoscopic effect.

[0038] In one embodiment, the processor (130) may generate a left-eye image by correcting a planar image based on a corrected depth map. The processor (130) may generate a left-eye image from the planar image corrected to correspond to the target fatigue.

[0039] In one embodiment, the processor (130) may warp the left-eye image to generate a right-eye image. Warping may be a process of manipulating an image so that at least one object included in the image is displayed deformed. Warping may be utilized in image processing, image manipulation, or stereoscopic image generation. For example, warping may include a process of digitally manipulating an image so that at least one object included in a planar image is displayed in a distorted manner so that the planar image has a three-dimensional effect. For example, warping may include a process of performing at least one operation among an X-axis shift, a Y-axis shift, an X-axis-based rotation, a Y-axis-based rotation, and a scaling on at least one object included in the planar image so that the at least one object has a three-dimensional effect. The processor (130) may warp the left-eye image to generate a right-eye image so as to correspond to a target fatigue.

[0040] In one embodiment, the processor (130) may output a left-eye image and a right-eye image to enable the display (110) to display a stereoscopic image. The processor (130) may control the display (110) so that the left-eye image faces the user's left eye and the right-eye image faces the user's right eye. The processor (130) may control the display (110) so that the user can view the stereoscopic image by outputting the left-eye image and the right-eye image.

[0041] FIG. 3 is a flowchart illustrating a control method of a stereoscopic image display device (100) according to one embodiment of the present disclosure.

[0042] In operation 310, a stereoscopic image display device (100) according to one embodiment can obtain a depth map from a planar image. The stereoscopic image display device (100) can obtain a depth map by extracting depth information of each pixel constituting the planar image.

[0043] In operation 320, a stereoscopic image display device (100) according to an embodiment can obtain depth-related characteristic information from a depth map. The stereoscopic image display device (100) can obtain a weight that contributes to visual fatigue estimation according to the depth of a pixel included in a planar image. The stereoscopic image display device (100) can obtain depth information indicating the depth of each region of the planar image and a confidence value that estimates the degree of error in the depth. The stereoscopic image display device (100) can obtain an important factor for visual fatigue estimation.

[0044] In operation 330, a stereoscopic image display device (100) according to an embodiment may obtain context information of a planar image. The context information of a planar image may be defined as information indicating features and situations extracted from the planar image. The context information of a planar image may be referred to as feature information of the planar image, a feature value of the planar image, or a feature extraction result of the planar image. The stereoscopic image display device (100) may obtain an object included in the planar image, a relative size of the object, and a position of the object. An object may refer to an object that has the greatest influence on the characteristics of the planar image as a result of measuring saliency. An object may be a feature value expressed in the form of a map. The relative size of an object may refer to a ratio of the sizes of each object included in the planar image. The position of an object may be a part of an object recognized as the most important when estimating visual fatigue. The stereoscopic image display device (100) may obtain edge information for correcting visual fatigue estimated based on motion information related to the motion of the planar image and an error in the depth of the planar image.

[0045] In operation 340, a stereoscopic image display device (100) according to an embodiment can calculate an estimated user fatigue level based on a depth map, characteristic information, and context information. The stereoscopic image display device (100) can calculate a level of visual fatigue felt by the user when viewing a stereoscopic image. The stereoscopic image display device (100) can extract factors important for estimating visual fatigue from a planar image and an estimated depth. The stereoscopic image display device (100) can calculate the estimated user fatigue level based on the extracted factors important for estimating visual fatigue. The stereoscopic image display device (100) can estimate the fatigue level based on a weight that contributes to estimating visual fatigue. The stereoscopic image display device (100) can estimate the fatigue level by simultaneously considering the importance of context information of a planar image and depth of the planar image when estimating visual fatigue. Accordingly, the stereoscopic image display device (100) can estimate the visual fatigue level in a manner that is robust to errors that occur when estimating depth from a planar image.

[0046] In operation 350, a stereoscopic image display device (100) according to an embodiment can correct a depth map based on an estimated fatigue level and a target fatigue level. The stereoscopic image display device (100) can input a threshold fatigue level that prevents the user from feeling fatigue above a corresponding level as a target fatigue level. The stereoscopic image display device (100) can adjust the stereoscopic effect of a stereoscopic image so that the estimated fatigue level corresponds to the target fatigue level. The stereoscopic image display device (100) can correct the depth map so that the adjusted stereoscopic effect corresponds to the adjusted stereoscopic effect.

[0047] In one embodiment, a stereoscopic image display device (100) according to an embodiment of the present invention can generate a left-eye image by correcting a flat image based on a corrected depth map in motion 360. The stereoscopic image display device (100) can generate a left-eye image from a flat image corrected to correspond to a target fatigue.

[0048] In operation 370, a stereoscopic image display device (100) according to an embodiment may warp a left-eye image to generate a right-eye image. Warping may be a process of manipulating an image so that at least one object included in the image is displayed in a deformed manner. Warping may be utilized in image processing, image manipulation, or stereoscopic image generation. For example, warping may include a process of digitally manipulating an image so that at least one object included in a planar image is displayed in a distorted manner so that the planar image has a three-dimensional effect. For example, warping may include a process of performing at least one operation among an X-axis shift, a Y-axis shift, an X-axis-based rotation, a Y-axis-based rotation, and a scaling on at least one object included in a planar image so that the at least one object has a three-dimensional effect. The stereoscopic image display device (100) may generate a right-eye image by warping the left-eye image to correspond to a target image.

[0049] In operation 380, a stereoscopic image display device (100) according to one embodiment can display a stereoscopic image by outputting a left-eye image and a right-eye image. The stereoscopic image display device (100) can output the left-eye image so that it faces the user's left eye, and the right-eye image so that it faces the user's right eye. The stereoscopic image display device (100) can output the left-eye image and the right-eye image so that the user can view the stereoscopic image.

[0050] FIG. 4 is a block diagram illustrating a processor (130) of a stereoscopic image display device (100) according to one embodiment of the present disclosure as a functional unit. The processor (130) according to one embodiment may include a depth estimation unit (410), a depth importance extraction unit (420), an image context extraction unit (430), a fatigue estimation unit (440), a stereoscopic effect adjustment unit (450), and a warping unit (460).

[0051] The depth estimation unit (410) can acquire a planar image. The depth estimation unit (410) can estimate the depth of the acquired planar image. The depth estimation unit (410) can extract a depth map of the acquired planar image. For example, the depth estimation unit (410) can extract a depth map of planar image data stored in the memory (120). For example, the depth estimation unit (410) can extract the depth map through a network that estimates the depth of a single planar image.

[0052] The depth importance extraction unit (420) can obtain a depth map. The depth importance extraction unit (420) can extract depth-related feature information from the depth map. The feature information can include weights that contribute to visual fatigue estimation according to the depth of pixels included in a planar image. The feature information can include depth information indicating the depth of each region of the planar image and a confidence value that estimates the degree of error in the depth. The depth importance extraction unit (420) can extract the importance of depth by extracting the weights and confidence values.

[0053] The depth importance extraction unit (420) according to one embodiment can extract at least one of a weight and a confidence value included in the characteristic information. For example, the depth importance extraction unit (420) can determine a main object of a planar image based on elements such as color, texture, and motion. For example, the depth importance extraction unit (420) can assign a higher weight to an area where the depth of the object is large, or closer to the depth of the main object in the planar image. For example, the depth importance extraction unit (420) can determine that the degree of error in the depth estimated from the planar image is proportional to the estimated confidence value.

[0054] The depth importance extraction unit (420) can transfer the extracted characteristic information to the fatigue estimation unit (440). The depth importance extraction unit (420) can transfer at least one of the extracted weights and confidence values ​​to the fatigue estimation unit (440).

[0055] The image context extraction unit (430) can obtain a planar image. The image context extraction unit (430) can extract context information from the planar image. The context information can include objects included in the planar image, relative sizes of the objects, and positions of the objects.

[0056] According to one embodiment, an object may refer to an object that has the greatest influence on the characteristics of a planar image as a result of measuring saliency. For example, an object may refer to an object that protrudes the most toward the front side among objects included in a planar image. For example, an object may refer to an object that has the greatest influence on the characteristics of a planar image as a result of measuring saliency. The image context extraction unit (430) may measure the saliency of each object included in a planar image. The image context extraction unit (430) may determine an object that has the greatest influence on the characteristics of a planar image as an object, as a result of measuring saliency. For example, the image context extraction unit (430) may determine an object that protrudes the most toward the front or the object that has the greatest influence on the characteristics of a planar image as a result of measuring saliency.

[0057] According to one embodiment, an object may be a feature value expressed in the form of a map. An object may be a value containing at least one variable, rather than a specified constant value. An object may be a concept distinct from an index used to identify the object. Each object can be extracted and utilized through a network. Accordingly, when an object is not input as a map-type input to the image context extraction unit (430), the network itself can determine the object by determining whether it has a major influence.

[0058] The relative size of an object according to one embodiment may refer to the ratio of the sizes of each object included in a planar image. The relative size of an object may be a value distinct from a size calculated based on a specific standard. The image context extraction unit (430) may calculate the relative size of an object by comparing the ratio of the sizes of each object included in a planar image. The image context extraction unit (430) may determine that as the ratio of the calculated relative sizes of the objects increases, the corresponding object has a greater influence on visual fatigue.

[0059] According to one embodiment, the location of an object may be a part of the object that is recognized as most important when estimating visual fatigue. In the present disclosure, the location of an object may be distinguished from the location or coordinate values ​​of a specific object. The location of an object may be a part of the object that the image context extraction unit (430) determines to have the greatest influence when estimating visual fatigue. For example, the location of an object may be a part of the object that has the greatest influence when estimating visual fatigue by comprehensively considering the coordinate values ​​of a central area including the object, the boundary of the object, the shape of the object, and the saliency of the object. The image context extraction unit (430) may recognize the most important part of an object when estimating visual fatigue through network learning.

[0060] Context information according to one embodiment may include motion information related to the motion of a planar image and edge information for compensating for visual fatigue estimated based on an error in depth of the planar image.

[0061] An image context extraction unit (430) according to one embodiment can detect important information such as a main object included in a planar image, the relative size of the main object, and the position of the main object, movements that affect visual fatigue due to rapid movements, and edge information for correcting visual fatigue incorrectly estimated due to depth errors. The image context extraction unit (430) can transmit the extracted context information to a fatigue estimation unit (440).

[0062] The fatigue estimation unit (440) can receive a depth map, feature information, and context information. The fatigue estimation unit (440) can calculate an estimated fatigue level of the user based on the depth map, feature information, and context information. The estimated fatigue level may be a level of visual fatigue that the user is expected to feel when viewing a stereoscopic image based on a planar image. The fatigue estimation unit (440) can estimate a fatigue level based on the depth map, a context extracted from the planar image, and a weight related to depth. For example, the fatigue estimation unit (440) can estimate a numerical level of fatigue felt by the user by applying a feature extraction method to the depth map. The feature extraction method may be a method of converting raw data of the depth map into numerical features that can be processed by the fatigue estimation unit (440). The numerical features converted by the feature extraction method may have information included in the raw data of the depth map. For example, the fatigue estimation unit (440) can estimate the numerical level of fatigue felt by the user by applying a feature extraction method to the RGB color information of the planar image. For example, the fatigue estimation unit (440) can implement the feature extraction method by using a network that processes characteristic information related to the depth of the planar image. The fatigue estimation unit (440) can transmit the calculated estimated fatigue to the three-dimensional effect control unit (450).

[0063] The three-dimensional effect control unit (450) can receive a depth map from the depth estimation unit (410). The three-dimensional effect control unit (450) can receive an estimated fatigue from the fatigue estimation unit (440). The three-dimensional effect control unit (450) can obtain a target fatigue. The target fatigue may be a threshold fatigue determined so that the user does not feel fatigue above a corresponding level. The target fatigue may include a plurality of levels. For example, the target fatigue may include a plurality of score levels. As the score level of the target fatigue increases, visual fatigue may decrease. For example, the target fatigue may include score level 0, score level 1, score level 2, score level 3, and score level 4. As the target fatigue approaches score level 4, visual fatigue felt by the user may decrease. As the target fatigue approaches score level 4, a stereoscopic image similar to a planar image may be generated.

[0064] A three-dimensional effect control unit (450) according to one embodiment can correct a depth map based on an estimated degree of fatigue and an acquired target degree of fatigue. The three-dimensional effect control unit (450) can receive a target degree of fatigue included in a user input. The three-dimensional effect control unit (450) can compare the estimated degree of fatigue and the target degree of fatigue. The three-dimensional effect control unit (450) can adjust the three-dimensional effect based on the comparison result of the estimated degree of fatigue and the target degree of fatigue. For example, the three-dimensional effect control unit (450) can adjust the depth range of the depth map based on the comparison result of the estimated degree of fatigue and the target degree of fatigue. The depth range can refer to the degree of three-dimensional effect that occurs when changing a planar image into a three-dimensional image having three-dimensional effect. The three-dimensional effect control unit (450) can transmit a depth map with adjusted three-dimensional effect to the warping unit (460).

[0065] The warping unit (460) can receive a depth map with adjusted three-dimensionality from the three-dimensionality control unit (450). The warping unit (460) can correct a planar image based on the depth map with adjusted three-dimensionality to generate a left-eye image. The warping unit (460) can apply a value output from the three-dimensionality control unit to the depth and convert it according to the degree of incongruity. The warping unit (460) can warp the left-eye image to generate a right-eye image. The warping unit (460) can apply the degree of incongruity to the left-eye image to generate a right-eye image. The degree of incongruity may refer to the visual difference between the left-eye image and the right-eye image. As the visual difference between the left-eye image and the right-eye image increases, the three-dimensionality of the stereoscopic image may increase. As the visual difference between the left-eye image and the right-eye image increases, visual fatigue of the stereoscopic image may increase. The warping unit (460) can output the left-eye image and the right-eye image to display the stereoscopic image.

[0066] The processor (130) of the stereoscopic image display device (100) according to one embodiment can extract factors important for estimating visual fatigue from a planar image and an estimated depth, and adjust the stereoscopic effect to a target fatigue level. The stereoscopic image display device (100) according to one embodiment can provide a visual fatigue prediction structure that is robust to errors accompanying depth estimation from a planar image. The processor (130) of the stereoscopic image display device (100) according to one embodiment can estimate visual fatigue by simultaneously considering the importance of context information and depth, which are essential elements in estimating visual fatigue.

[0067] FIG. 5 is a diagram illustrating an inference module (500) and a target fatigue score network (560) of a stereoscopic image display device (100) according to one embodiment of the present disclosure. The inference module (500) according to one embodiment may be included in the processor (130). The inference module (500) according to one embodiment may include a depth and uncertainty estimation unit (510), an edge detection unit (520), a depth control unit (530), a depth heterogeneity unit (540), and a scale adjustment unit (550).

[0068] The depth and uncertainty estimation unit (510) can receive a planar image. The depth and uncertainty estimation unit (510) can estimate the depth of the planar image. The depth and uncertainty estimation unit (510) can obtain a depth map based on the estimated depth of the planar image. The depth and uncertainty estimation unit (510) can estimate uncertainty regarding the estimated depth. The depth and uncertainty estimation unit (510) can extract elements related to uncertainty among the importance of depth.

[0069] A depth and uncertainty estimation unit (510) according to one embodiment can obtain uncertainty information related to the uncertainty of a depth map. The depth and uncertainty estimation unit (510) can transmit the uncertainty information to a scale adjustment unit (550).

[0070] The edge detection unit (520) can receive a planar image. The edge detection unit (520) can extract edges of the planar image. The edge detection unit (520) can extract elements related to edges among the importance of depth.

[0071] The depth control unit (530) can receive depth-related information from the depth and uncertainty estimation unit (510). The depth control unit (530) can receive edge-related information from the edge detection unit (520). The depth control unit (530) can control the depth based on the depth-related information and the edge-related information. The depth control unit (530) can transmit the control result to the depth heterogeneity unit (540).

[0072] The depth heterogeneity unit (540) can obtain a planar image. The depth heterogeneity unit (540) can receive a control result of the depth control unit (530). The depth heterogeneity unit (540) can receive a target fatigue. The depth heterogeneity unit (540) can receive an adjustment result of the scale adjustment unit (550). The depth heterogeneity unit (540) can predict heterogeneity at depth. The depth heterogeneity unit (540) can transmit the predicted heterogeneity to the target fatigue score network (560). The depth heterogeneity unit (540) can be referred to as Depth2Disp.

[0073] The scale adjustment unit (550) can receive uncertainty information from the depth and uncertainty estimation unit (510). The scale adjustment unit (550) can adjust the scale of the planar image based on the uncertainty information. The scale adjustment unit (550) can control the depth heterogeneity unit (540) to correct the depth map based on the uncertainty information.

[0074] A scale adjustment unit (550) according to one embodiment may apply a local scale to an area with high uncertainty and a large negative disparity. The negative disparity may refer to a visual difference that gives the illusion of a three-dimensional image when an object appears in front of the screen of a display (110) when a stereoscopic image display device (100) displays a stereoscopic image. The scale adjustment unit (550) may apply a smoothing filter to the uncertainty map to reduce noise.

[0075] In one embodiment, the scale adjustment unit (550) may calculate a first threshold value from the result of applying a smoothing filter to reduce noise. The first threshold value may be a threshold value for extracting an uncertainty map. The scale adjustment unit (550) may extract an area in the uncertainty map where the uncertainty is higher than the first threshold value.

[0076] In one embodiment, the scale adjustment unit (550) may apply non-linear mapping to an area in which the negative disparity is higher than a second threshold among areas in which the uncertainty is higher than a first threshold. The second threshold may be a threshold for the negative disparity that has a major impact on visual fatigue. The non-linear mapping may be a mapping method that estimates a predicted disparity map with multiple values. The scale adjustment unit (550) may calculate the predicted disparity by applying at least one specific value to the depth map. If the scale adjustment unit (550) applies a single specific value to the depth map, it may not be able to take uncertainty into account. Accordingly, the scale adjustment unit (550) may calculate the predicted disparity with multiple values ​​corresponding to the uncertainty map by applying non-linear mapping.

[0077] The scale adjustment unit (550) according to one embodiment can map different values ​​to areas having negative heterogeneity. For example, the scale adjustment unit (550) can apply nonlinear mapping to areas having negative heterogeneity in which the negative heterogeneity is greater than a second threshold value in the uncertainty map among areas having negative heterogeneity. Accordingly, the stereoscopic image display device (100) according to one embodiment can reduce visual fatigue caused by negative heterogeneity that can be felt in a specific area of ​​a stereoscopic image, while maintaining the stereoscopic effect in the remaining areas of the stereoscopic image.

[0078] The target fatigue score network (560) can receive the predicted heterogeneity from the depth heterogeneity unit (540). The target fatigue score network (560) can receive the target fatigue score. The target fatigue score network (560) can estimate the estimated fatigue from the input heterogeneity. The target fatigue score network (560) can be used to learn a model that estimates the estimated fatigue.

[0079] The inference module (500) of the stereoscopic image display device (100) according to one embodiment can reduce visual fatigue of a stereoscopic image by using not only depth but also edge components of a planar image and depth uncertainty information. The inference module (500) can reduce errors that occur when estimating depth from a planar image. The inference module (500) can correct at least a portion of a depth map corresponding to an area where an error exceeds a specified threshold value based on edge and depth uncertainty information extracted from a planar image. For example, the inference module (500) can extract an edge component by applying a Sobel filter that detects a change in brightness values ​​of an image to distinguish boundaries between different objects. For example, the inference module (500) can generate an uncertainty map that expresses pixel-by-pixel probability for uncertainty of the depth map. For example, if there is an area in the uncertainty map where the uncertainty probability exceeds a specified threshold value, the inference module (500) can correct the area in the depth map where the uncertainty probability exceeds the threshold value.

[0080] The inference module (500) of the stereoscopic image display device (100) according to one embodiment can obtain loss information related to the heterogeneity between the planar image and the left-eye image. The loss information can include geometric loss, re-projection loss, and visual fatigue regression loss (VC regression loss).

[0081] In one embodiment, the geometric loss may be a value that compares the predicted heterogeneity with the ground truth (GT) heterogeneity. For example, the geometric loss may have a first loss value (L1), which is a loss value resulting from the difference between the ground truth heterogeneity and the predicted heterogeneity.

[0082] In one embodiment, the re-projection loss may be a loss value between a warped stereoscopic image and a planar image. The re-projection loss may be a loss value calculated by applying a mask to the warped stereoscopic image and the planar image, excluding an occlusion area. The occlusion area may be an area that is not visible in the left-eye image but is visible only in the right-eye image. Occlusion may be a phenomenon that occurs due to binocular parallax. The mask may be a binary mask. The image of the occlusion area may be inaccurate. In order to prevent an error from occurring when calculating the re-projection loss due to an image of an inaccurate occlusion area, the inference module (500) may assume the occlusion area to be 0 when calculating the re-projection loss. To assume the occlusion area to be 0, the inference module (500) may apply a binary mask to the warped stereoscopic image and the planar image. For example, the reprojection loss may have a first loss value (L1) that is the same as the geometric loss.

[0083] The visual fatigue regression loss according to one embodiment may be a value measuring the fatigue score of the generated heterogeneity.

[0084] According to one embodiment, the inference module (500) can learn by utilizing loss information during fatigue estimation learning. The inference module (500) can be trained to adjust the sense of heterogeneity so as to minimize visual fatigue while simultaneously reflecting the original stereoscopic image's inherent three-dimensionality and image quality through geometric loss and re-projection loss.

[0085] FIG. 6 is a diagram illustrating a stereoscopic image display device (100) according to one embodiment of the present disclosure generating a stereoscopic image based on a planar image. The stereoscopic image display device (100) according to one embodiment may include a depth and uncertainty estimation unit (510), an edge detection unit (520), a depth control unit (530), and a depth heterogeneity unit (540). The depth control unit (530) according to one embodiment may include a depth refinement unit (610) and a depth adjustment unit (620).

[0086] A depth and uncertainty estimation unit (510) according to one embodiment can receive a planar RGB image. The planar RGB image can be a planar image including RGB values. The depth and uncertainty estimation unit (510) can extract the depth of the planar RGB image. The depth and uncertainty estimation unit (510) can transmit the extracted depth to a depth refinement unit (610).

[0087] An edge detection unit (520) according to one embodiment can receive a planar RGB image. The edge detection unit (520) can extract an edge of the planar RGB image. The edge detection unit (520) can transmit the extracted edge to a depth refinement unit (610).

[0088] The depth refinement unit (610) can receive the depth from the depth and uncertainty estimation unit (510). The depth refinement unit (610) can receive the edge from the edge detection unit (520). The depth refinement unit (610) can clarify the edge of the planar RGB image based on the edge. The depth refinement unit (610) can separate the object and the background of the planar RGB image. The depth refinement unit (610) can increase the three-dimensionality of the planar RGB image. The depth refinement unit (610) can reduce the phenomenon in which errors are propagated and fatigue increases when generating a three-dimensional image due to blurry edges at the boundary of the object. The depth refinement unit (610) can reduce the fatigue of the planar RGB image.

[0089] The depth adjustment unit (620) can analyze the depth range of the planar RGB image. The depth range of the planar RGB image can be set by the distance from the camera when the planar RGB image is acquired. The planar RGB image acquired by the stereoscopic image display device (100) can be acquired by the camera of the stereoscopic image display device (100) or an external camera. Among the objects included in the planar RGB image, the closer the object is to the camera, the greater the influence it can have on visual fatigue. The depth adjustment unit (620) can adjust the depth of the depth map to reduce visual fatigue.

[0090] A depth adjustment unit (620) according to one embodiment can reduce the stereoscopic effect of a critical region whose depth is greater than or equal to a specified threshold depth in a depth map. The depth adjustment unit (620) can reduce the stereoscopic effect of a critical region whose depth is greater than or equal to a specified threshold depth in a depth map while maintaining the stereoscopic effect of the remaining portions of a stereoscopic image generated based on a planar RGB image. Accordingly, the depth adjustment unit (620) can reduce visual fatigue occurring in a critical region in a stereoscopic image that particularly strongly causes visual fatigue.

[0091] The depth disparity unit (540) can calculate scale and offset by inputting a planar RGB image and depth to convert from depth to disparity. The depth disparity unit (540) can calculate disparity as depth * scale + offset. The offset can affect the zero plane. The zero plane can be an area where there is no visual difference between the left-eye image and the right-eye image. The scale can affect the visual difference between the left-eye image and the right-eye image. The reference plane and visual difference can be generated differently for each screen of the stereoscopic image by the two elements of scale and offset. The depth disparity unit (540) can transmit the generated disparity to the target fatigue score network (560). When training the depth heterogeneity unit (540) through the target fatigue score network (560), the depth heterogeneity unit (540) can adjust the heterogeneity based on the feedback received from the target fatigue score network (560). The depth heterogeneity unit (540) can generate values ​​related to the heterogeneity without the information received from the target fatigue score network (560) when the inference module (500) performs inference.

[0092] FIG. 7 is a drawing showing a stereoscopic image displayed by a stereoscopic image display device (100) according to one embodiment of the present disclosure while taking visual fatigue into consideration.

[0093] The first stereoscopic image (710) may be a stereoscopic image before visual fatigue is reduced. The first stereoscopic image (710) may be a stereoscopic image before depth map edge enhancement and depth map nonlinear mapping using an uncertainty map are applied. The first object (711) and the second object (712) of the first stereoscopic image (710) are each spaced apart from each other to the left and right by a first distance (d1) to generate a visual difference and provide a sense of three-dimensionality. The first distance (d1) may be greater than a critical distance, which may cause visual fatigue.

[0094] The second stereoscopic image (720) may be a stereoscopic image after visual fatigue has been reduced to the first degree. For example, the second stereoscopic image (720) may be a stereoscopic image to which a target fatigue score level of 3.0 has been applied. The first object (721) and the second object (722) of the second stereoscopic image (720) may be spaced apart from each other left and right by a second distance (d2) to generate a visual difference and provide a sense of three-dimensionality. The second distance (d2) may be less than a critical distance, thereby reducing visual fatigue.

[0095] The third stereoscopic image (730) may be a stereoscopic image after visual fatigue has been reduced by a second degree. The third stereoscopic image (730) may have a higher target fatigue score level than the second stereoscopic image (720). For example, the third stereoscopic image (730) may be a stereoscopic image to which a target fatigue score level of 4.0 has been applied. The distance between the first object (731) and the second object (732) of the third stereoscopic image (730) on the left and right may be a very small distance that is difficult to perceive with the naked eye. The third stereoscopic image (730) may further reduce visual fatigue.

[0096] According to one embodiment, a stereoscopic image display device (100) can generate a stereoscopic image by setting a target fatigue level between 2.0 and 4.0. For example, as the target fatigue score level set by the stereoscopic image display device (100) decreases, the possibility of visual fatigue occurring due to negative foreignness may increase. For example, as the target fatigue score level set by the stereoscopic image display device (100) increases, the three-dimensional effect of the stereoscopic image may decrease, but the overall foreignness range may decrease, thereby reducing visual fatigue.

[0097] FIG. 8 is a block diagram illustrating functional units for depth control in consideration of a user's gaze included in a processor (130) of a stereoscopic image display device (100) according to one embodiment of the present disclosure. The processor (130) of the stereoscopic image display device (100) may include a gaze tracking unit (810), a context acquisition unit (820), an environmental feature acquisition unit (830), a depth control unit (840), and a target fatigue setting unit (850). The depth control unit (840) may include a stereoscopic colorization unit (841) and a depth adjustment unit (842).

[0098] The gaze tracking unit (810) can obtain a face image. The face image can be an image including the user's face. The face image can be an image showing the direction of the user's gaze. The gaze tracking unit (810) can track the user's gaze from the face image. The gaze tracking unit (810) can obtain gaze information from the face image. The gaze information can include information related to an area where the user's gaze is directed in the stereoscopic image. The gaze information can include information related to the viewer's characteristics. The gaze tracking unit (810) can transmit the gaze information to the depth control unit (840).

[0099] The context acquisition unit (820) can acquire a left-eye image. The left-eye image may be an image input to the user's left eye. The context acquisition unit (820) can acquire context from the left-eye image. The context may include information related to features of a depth map corresponding to the left-eye image. The context may include information related to features of content included in the left-eye image. The context acquisition unit (820) can transmit the context to the depth control unit (840).

[0100] The environmental feature acquisition unit (830) can acquire viewing environment information. The viewing environment information can include information related to the specifications of the video being viewed by the user. For example, the viewing environment information can include at least one of the color tone of the video, the illuminance of the video, the contrast of the video, and the display size of the video. The environmental feature acquisition unit (830) can acquire environmental features from the viewing environment information. The environmental features can include at least one of the color tone of the video, the illuminance of the video, the contrast of the video, and the display size of the video. The environmental feature acquisition unit (830) can transmit the environmental features to the depth control unit (840).

[0101] The depth control unit (840) can obtain gaze information from the gaze tracking unit (810). The depth control unit (840) can obtain context from the context acquisition unit (820). The depth control unit (840) can obtain environmental features from the environmental feature acquisition unit (830). The depth control unit (840) can correct a depth map based on the gaze information, the context, and the environmental features. The depth map can include information related to the depth of a planar image.

[0102] The depth control unit (840) can transmit the corrected depth map to the target fatigue setting unit (850). The depth control unit (840) can receive the target fatigue from the target fatigue setting unit (850). The depth control unit (840) can adjust the depth of the planar image based on the gaze information, context, environmental features, and the target fatigue. The depth control unit (840) can adjust the depth of the planar image to generate a stereoscopic image. The depth control unit (840) can output a stereoscopic image.

[0103] The target fatigue setting unit (850) can receive user input. The user input can include information related to the target fatigue. For example, the user input can include the target fatigue score level of the stereoscopic image. The target fatigue setting unit (850) can receive the depth map corrected by the depth control unit (840). The target fatigue setting unit (850) can transmit the target fatigue to the depth control unit (840).

[0104] A stereoscopic image display device (100) according to one embodiment can track an area where a user's gaze lingers using gaze tracking technology. The stereoscopic image display device (100) can extract the depth of the area where the user's gaze actually lingers, the characteristics of the content, and the movement of the object. The stereoscopic image display device (100) can utilize the extraction results to predict fatigue. The stereoscopic image display device (100) can calculate visual fatigue by utilizing information on the content of the area where the gaze actually lingers. For example, the stereoscopic image display device (100) can recognize information and characteristics of the content of the area where the user's gaze lingers and calculate a region-based visual fatigue. The stereoscopic image display device (100) can weight and use information on the area where the user mainly looks. For example, the stereoscopic image display device (100) can perform different visual fatigue and depth adjustments depending on the location where the user's gaze lingers, even for the same flat image.

[0105] A stereoscopic image display device (100) according to one embodiment can predict visual fatigue by considering the area where the user's gaze lingers, thereby improving the accuracy of the prediction result. In particular, when the stereoscopic image display device (100) is a device that applies Virtual Studio Technology (VST), the stereoscopic image display device (100) according to one embodiment can further improve the accuracy of extracting the area where the user's gaze lingers.

[0106] A stereoscopic image display device (100) according to one embodiment can proactively reduce visual fatigue in an area where a user's gaze lingers in a stereoscopic image through learning. The stereoscopic image display device (100) can learn information about the area and objects where the user's gaze lingers. When learning the area where the user's gaze lingers, the stereoscopic image display device (100) can learn gaze movement based on the characteristics of the content and gaze movement based on the user's viewing characteristics and preferences. For example, the stereoscopic image display device (100) can learn the content of interest to the user, the user's preferred actor, and gaze movement based on subtitles.

[0107] According to one embodiment, a stereoscopic image display device (100) can acquire a user's viewing habits based on learning results. The stereoscopic image display device (100) can predict the area where the user's gaze will be directed based on the user's viewing habits. The stereoscopic image display device (100) can use information related to the user's viewing habits by linking it to a personal account. For example, the stereoscopic image display device (100) can store and update information related to the user's viewing habits on a personalized device or personal account.

[0108] A stereoscopic image display device (100) according to one embodiment can provide a stereoscopic image with low visual fatigue by pre-adjusting the depth of an area where the user's gaze is expected to be directed. For example, the stereoscopic image display device (100) can limit the depth range of an area where the user's gaze is expected to remain within a specific threshold range. For example, the stereoscopic image display device (100) can map the depth of an area where the user's gaze is expected to remain to 0. Accordingly, the stereoscopic image display device (100) according to one embodiment can preemptively generate a stereoscopic image with low visual fatigue.

[0109] A stereoscopic image display device (100) according to one embodiment can adjust visual fatigue by considering a stereoscopic image and depth. The stereoscopic image display device (100) can reduce visual fatigue through stereoscopic image optimization and depth scaling. The stereoscopic image display device (100) can apply at least one of stereoscopic image optimization and depth scaling depending on the visual fatigue of the stereoscopic image. For example, the stereoscopic image display device (100) can apply stereoscopic image optimization when the depth range of the stereoscopic image is within a specified range. The stereoscopic image display device (100) can apply stereoscopic image optimization to a stereoscopic image including content that can be perceived as stereoscopic even if the depth is small. For example, the stereoscopic image display device (100) can apply stereoscopic image optimization to a stereoscopic image in which a single object is moving in front or in which a sense of perspective is prominent.

[0110] The present disclosure can provide an algorithm for predicting visual fatigue by considering the characteristics of each stereoscopic image and each user. Furthermore, the present disclosure can provide an algorithm pipeline that predicts the cumulative value of a user's visual fatigue by observing the user's biosignals while watching a stereoscopic image and reflects this in real-time stereoscopic adjustments. Accordingly, the present disclosure can provide an objective indicator-based fatigue prediction algorithm for predicting and reflecting discomfort in real-time.

[0111] FIG. 9 is a drawing showing a stereoscopic image displayed by a stereoscopic image display device (100) according to one embodiment of the present disclosure while taking into consideration the user's line of sight.

[0112] The fourth stereoscopic image (910) may be a stereoscopic image prior to applying gaze information. The fourth stereoscopic image (910) may display the first object (911) and the second object (912) with the same three-dimensional effect. Each of the first object (911) and the second object (912) of the fourth stereoscopic image (910) may be focused to the same degree.

[0113] The fifth stereoscopic image (920) may be a stereoscopic image that applies gaze information in which the gaze moves to the second object (922). The fifth stereoscopic image (920) may display the second object (922) to have a greater sense of three-dimensionality compared to the first object (921). The fifth stereoscopic image (920) may display the second object (922) to have a greater focus compared to the first object (921). The fifth stereoscopic image (920) may have the first object (921) blurred.

[0114] The sixth stereoscopic image (930) may be a stereoscopic image that applies gaze information in which the gaze moves to the first object (931). The sixth stereoscopic image (930) may display the first object (931) to have a greater sense of three-dimensionality compared to the second object (932). The sixth stereoscopic image (930) may display the first object (931) to have a more focused image compared to the second object (932). The second object (932) may be blurred in the sixth stereoscopic image (930).

[0115] A stereoscopic image display device (100) according to one embodiment of the present disclosure can predict visual fatigue based on bio-signals. For example, the stereoscopic image display device (100) can include a gaze tracking sensor module. The stereoscopic image display device (100) can obtain gaze information using the gaze tracking sensor module. The stereoscopic image display device (100) can predict visual fatigue based on the gaze information.

[0116] A stereoscopic image display device (100) according to one embodiment can obtain user profile information before the user views a stereoscopic image. The stereoscopic image display device (100) can obtain at least one of user age information, user pupil size information, and user interpupillary distance information. For example, the stereoscopic image display device (100) can obtain user age using user input or pre-stored user information. For example, the stereoscopic image display device (100) can obtain user pupil size and user interpupillary distance using user input or facial information.

[0117] A stereoscopic image display device (100) according to one embodiment can obtain viewing environment information before a user views a stereoscopic image. For example, the stereoscopic image display device (100) can obtain at least one of ambient illuminance information, current time information, and viewing distance information.

[0118] A stereoscopic image display device (100) according to one embodiment can obtain gaze information while a user is watching a stereoscopic image. For example, the stereoscopic image display device (100) can obtain bio-signals such as the user's eye movement speed and the user's eye blinking. For example, the stereoscopic image display device (100) can obtain the user's facial expression information to predict whether the user feels visual discomfort. For example, the stereoscopic image display device (100) can determine whether the user has difficulty focusing by measuring the presence of gaze fixation and the saccadic movement of the user's eyes. For example, the stereoscopic image display device (100) can detect a protruding area where the user's gaze lingers in a stereoscopic image.

[0119] A stereoscopic image display device (100) according to one embodiment can predict visual fatigue based on the content of a stereoscopic image while a user is watching the stereoscopic image. For example, the stereoscopic image display device (100) can predict visual fatigue by acquiring the quality of the stereoscopic image, the genre of the stereoscopic image, the complexity of the stereoscopic image, the amount of motion on the plane of the stereoscopic image, and the amount of motion in the depth direction of the stereoscopic image.

[0120] A stereoscopic image display device (100) according to one embodiment can output depth according to a desired visual fatigue score by taking into account a three-dimensional effect and visual comfort using a depth control network. The depth control network can reduce visual fatigue based on an eye tracking map.

[0121] FIG. 10 is a flowchart illustrating a method for a stereoscopic image display device (100) according to one embodiment of the present disclosure to display a stereoscopic image by taking into account the user's line of sight.

[0122] In operation 1010, a stereoscopic image display device (100) according to an embodiment can acquire a face image. The stereoscopic image display device (100) can include a camera module. The stereoscopic image display device (100) can capture a user's face using the camera module.

[0123] In operation 1020, a stereoscopic image display device (100) according to an embodiment can track a user's gaze in a face image. The stereoscopic image display device (100) can track the user's gaze to obtain information about the user's gaze. For example, the stereoscopic image display device (100) can track the user's gaze to obtain information about the direction in which the user's gaze is directed, a map showing where the user's gaze is directed, or coordinate information on a stereoscopic image to which the user's gaze is directed. The stereoscopic image display device (100) can track the user's gaze to obtain the user's facial expression characteristics. The stereoscopic image display device (100) can track the user's gaze to obtain information about the user's eye movement. The stereoscopic image display device (100) can obtain the user's gaze information in real time.

[0124] In operation 1030, a stereoscopic image display device (100) according to an embodiment may extract a region of interest (ROI) from a planar image based on a tracking result. The ROI may include an area where the user's gaze lingers and an area where the user's gaze primarily directs. The stereoscopic image display device (100) may accumulate information related to a location where the gaze is directed within the stereoscopic image. The stereoscopic image display device (100) may extract the ROI based on the accumulated information. For example, the stereoscopic image display device (100) may extract the ROI by combining main object information and gaze information of the stereoscopic image to determine whether the user is observing the main object of the stereoscopic image. For example, the stereoscopic image display device (100) may extract the ROI by weighting the vergence-accommodation conflict (VAC) of the area where the user's gaze lingers.

[0125] In operation 1040, a stereoscopic image display device (100) according to an embodiment can obtain a depth map from a planar image. The stereoscopic image display device (100) can obtain depth in each region of the planar image. The stereoscopic image display device (100) can obtain a depth map using the obtained depth for each region.

[0126] In operation 1050, a stereoscopic image display device (100) according to an embodiment may acquire depth-related characteristic information based on a depth map and a region of interest. The characteristic information may include information indicating the depth of the region of interest in more detail. For example, the characteristic information may include at least one of a depth range, a depth gradient map, and positive / negative depth information.

[0127] According to one embodiment, a stereoscopic image display device (100) can more precisely acquire depth-related characteristic information from a region of interest to correct the region of interest. For example, the stereoscopic image display device (100) can acquire characteristic information by determining the depth of an object in the region of interest as a reference height. For example, the stereoscopic image display device (100) can adjust the depth centered on the region of interest.

[0128] In operation 1060, a stereoscopic image display device (100) according to an embodiment can display a stereoscopic image generated based on characteristic information. The stereoscopic image display device (100) can adjust visual fatigue of a region of interest based on the characteristic information. For example, the stereoscopic image display device (100) can limit a depth adjustment value centered on the region of interest to a threshold value. Accordingly, the stereoscopic image display device (100) can reduce visual fatigue by reducing a phenomenon in which abrupt changes in depth or three-dimensionality occur centered on the region of interest.

[0129] FIG. 11 is a flowchart illustrating a method for correcting the color of a stereoscopic image by taking into account the user's line of sight in a stereoscopic image display device (100) according to one embodiment of the present disclosure.

[0130] In operation 1110, a stereoscopic image display device (100) according to an embodiment can acquire a face image. The stereoscopic image display device (100) can include a camera module. The stereoscopic image display device (100) can capture a user's face using the camera module.

[0131] In operation 1120, a stereoscopic image display device (100) according to an embodiment can track a user's gaze in a face image. The stereoscopic image display device (100) can include a gaze tracking sensor module. The stereoscopic image display device (100) can obtain information related to the user's eye movements using the gaze tracking sensor module. For example, the stereoscopic image display device (100) can track at least one of the user's eye movements of blink, eye fixation, saccade, scan path length, and vergence using the gaze tracking sensor module.

[0132] A stereoscopic image display device (100) according to one embodiment can measure user-specific characteristic information using a gaze tracking sensor module. For example, the stereoscopic image display device (100) can measure information related to at least one of the user's age, the user's pupil size, the user's interpupillary distance, the area where the user's eyes remain, and the user's facial expression using the gaze tracking sensor module.

[0133] In operation 1130, a stereoscopic image display device (100) according to an embodiment may extract a region of interest from a planar image based on a tracking result. The region of interest may include an area where the user's gaze lingers and an area where the user's gaze is primarily directed. The stereoscopic image display device (100) may accumulate information related to a position where the gaze is directed within the stereoscopic image. The stereoscopic image display device (100) may extract a region of interest based on the accumulated information. For example, the stereoscopic image display device (100) may combine main object information and gaze information of the stereoscopic image to determine whether the user is observing a main object of the stereoscopic image, thereby extracting a region of interest.

[0134] In operation 1140, a stereoscopic image display device (100) according to an embodiment can obtain a depth map from a planar image. The stereoscopic image display device (100) can obtain depth in each region of the planar image. The stereoscopic image display device (100) can obtain a depth map using the obtained depth for each region.

[0135] In operation 1150, a stereoscopic image display device (100) according to an embodiment can correct a depth map based on a target fatigue level. The stereoscopic image display device (100) can correct the depth of a stereoscopic image to satisfy the target fatigue level using a depth control network. For example, the stereoscopic image display device (100) can correct the depth of a stereoscopic image to be below a threshold depth set according to the target fatigue level. The stereoscopic image display device (100) can correct the depth of a stereoscopic image based on a reference plane set by the depth control network.

[0136] In operation 1160, a stereoscopic image display device (100) according to an embodiment can correct the color of a region of interest based on a depth map and a region of interest. The stereoscopic image display device (100) can include a colorization module. The stereoscopic image display device (100) can perform optimal colorization for a stereoscopic image using the colorization module. The stereoscopic image display device (100) can adjust the three-dimensional effect of a stereoscopic image through colorization. The stereoscopic image display device (100) can perform colorization of a region of interest based on a depth value in a depth map. The stereoscopic image display device (100) can determine the degree of color correction of a region of interest based on a size of the region of interest, a position of the region of interest, and a depth value in the region of interest.

[0137] A stereoscopic image display device (100) according to one embodiment can adjust the visual fatigue and three-dimensionality of a stereoscopic image by using at least one of colorization and depth control. For example, the stereoscopic image display device (100) can output a flat image with a depth of 0 by using only colorization. For example, the stereoscopic image display device (100) can output a stereoscopic image with a more three-dimensional effect at the same depth by using colorization after depth control. For example, the stereoscopic image display device (100) can output a stereoscopic image that reduces visual fatigue while providing the same three-dimensional effect by using colorization and depth control.

[0138] A stereoscopic image display device (100) according to one embodiment can perform colorization considering a region of interest. For example, the stereoscopic image display device (100) can improve the image quality of a stereoscopic image by performing colorization on a user's region of interest. For example, the stereoscopic image display device (100) can enhance the three-dimensional effect of a stereoscopic image by performing colorization on a user's region of interest.

[0139] In operation 1170, a stereoscopic image display device (100) according to an embodiment can display a stereoscopic image based on a corrected region of interest. The stereoscopic image display device (100) can adjust the visual fatigue of the region of interest based on characteristic information. For example, the stereoscopic image display device (100) can limit the color change value centered on the region of interest to a threshold value. Accordingly, the stereoscopic image display device (100) can reduce the phenomenon of abrupt color change or three-dimensional change centered on the region of interest, thereby reducing visual fatigue.

[0140] A stereoscopic image display device according to the present disclosure may include a display, a memory storing at least one instruction, and at least one processor. In one embodiment, at least one instruction, when executed by the at least one processor, causes the at least one processor to obtain a depth map from a planar image, obtain characteristic information related to a first depth from the depth map, obtain context information of the planar image, calculate an estimated fatigue of a user based on the depth map, the characteristic information, and the context information, wherein the estimated fatigue represents visual comfort (VC) felt by the user when viewing a first stereoscopic image, correct the depth map based on the estimated fatigue and the target VC, correct the planar image based on the corrected depth map to generate a left-eye image, warp the left-eye image to generate a right-eye image, and output the left-eye image and the right-eye image so that the display displays a second stereoscopic image.

[0141] The characteristic information according to one embodiment may include a weighting value that contributes to estimating visual fatigue according to a second depth of a pixel included in a planar image.

[0142] The characteristic information according to one embodiment may include depth information indicating multiple depths for each of multiple regions of a planar image and a confidence value estimating the degree of error of the multiple depths.

[0143] Context information according to one embodiment may include objects included in a planar image, relative sizes of the objects, and positions of the objects.

[0144] Context information according to one embodiment may include motion information related to the motion of a planar image and edge information for correcting visual fatigue estimated based on an error in a second depth of the planar image.

[0145] At least one instruction according to one embodiment, when executed by at least one processor, may cause the at least one processor to obtain uncertainty information related to uncertainty of a depth map and to correct the depth map based on the uncertainty information.

[0146] At least one instruction according to one embodiment, when executed by at least one processor, may cause the at least one processor to obtain loss information related to disparity between a planar image and a left-eye image.

[0147] At least one instruction according to one embodiment, when executed by at least one processor, may cause the at least one processor to reduce the depth of a critical region in a depth map having a depth greater than a specified threshold depth.

[0148] At least one instruction according to one embodiment, when executed by at least one processor, may cause the at least one processor to extract a region of interest from a planar image based on a result of tracking a user's gaze, and to obtain feature information based on the depth map and the region of interest.

[0149] At least one instruction according to one embodiment, when executed by at least one processor, may cause the at least one processor to correct a color of a region of interest based on a depth map and a region of interest.

[0150] A method for controlling a stereoscopic image display device according to the present disclosure may include an operation of obtaining a depth map from a planar image, an operation of obtaining characteristic information related to a first depth from the depth map, an operation of obtaining context information of the planar image, an operation of calculating an estimated fatigue of a user based on the depth map, the characteristic information, and the context information, an operation of correcting the depth map based on the estimated fatigue and a target fatigue (target VC), wherein the estimated fatigue represents visual comfort (VC) felt by the user when viewing a first stereoscopic image, an operation of correcting the planar image based on the corrected depth map to generate a left-eye image, an operation of warping the left-eye image to generate a right-eye image, and an operation of outputting the left-eye image and the right-eye image to display a second stereoscopic image.

[0151] The characteristic information according to one embodiment may include a weighting value that contributes to estimating visual fatigue according to a second depth of a pixel included in a planar image.

[0152] The characteristic information according to one embodiment may include depth information indicating multiple depths for each of multiple regions of a planar image and a confidence value estimating the degree of error in the depth.

[0153] Context information according to one embodiment may include objects included in a planar image, relative sizes of the objects, and positions of the objects.

[0154] Context information according to one embodiment may include motion information related to the motion of a planar image and edge information for correcting visual fatigue estimated based on an error in a second depth of the planar image.

[0155] A method for controlling a stereoscopic image display device according to one embodiment may include an operation of obtaining uncertainty information related to uncertainty of a depth map, and an operation of correcting the depth map based on the uncertainty information.

[0156] A method for controlling a stereoscopic image display device according to one embodiment may include an operation of obtaining loss information related to disparity between a planar image and a left-eye image.

[0157] A method for controlling a stereoscopic image display device according to one embodiment may include an operation for reducing the stereoscopic effect of a threshold area whose depth is greater than a specified threshold depth in a depth map.

[0158] A method for controlling a stereoscopic image display device according to one embodiment may include an operation of extracting a region of interest from a planar image based on a result of tracking a user's gaze, and an operation of obtaining characteristic information based on a depth map and the region of interest.

[0159] A method for controlling a stereoscopic image display device according to one embodiment may include an operation of correcting a color of a region of interest based on a depth map and a region of interest.

[0160] A stereoscopic image display device and a control method thereof according to the present disclosure have the effect of generating a stereoscopic image with low visual fatigue by estimating visual fatigue when generating a stereoscopic image by receiving a flat image as input and adjusting the stereoscopic effect so that the estimated visual fatigue corresponds to a target fatigue.

[0161] A method according to an embodiment of the present disclosure may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the present disclosure or may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0162] Some embodiments of the present disclosure may also be implemented in the form of a recording medium containing computer-executable instructions, such as program modules, executed by a computer. Computer-readable media may be any available media that can be accessed by a computer, and include both volatile and nonvolatile media, removable and non-removable media. Furthermore, computer-readable media may include both computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. Communication media typically includes computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave, or other transport mechanism, and includes any information delivery media. Furthermore, some embodiments of the present disclosure may also be implemented as a computer program or computer program product containing computer-executable instructions, such as a computer program that is executed by a computer.

[0163] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0164] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

Claims

1. In a stereoscopic image display device, display; memory for storing at least one instruction; and comprising at least one processor, The at least one instruction, when executed by the at least one processor, causes the at least one processor to: Obtain a depth map from a flat image, Obtaining feature information related to the first depth from the above depth map, Obtain context information of the above flat image, The estimated fatigue of the user is calculated based on the depth map, the characteristic information, and the context information, and the estimated fatigue represents the visual comfort (VC) felt by the user when viewing the first stereoscopic image. The depth map is corrected based on the estimated fatigue and target fatigue (target VC), Generate a left eye image by correcting the flat image based on the corrected depth map, The left eye image is warped to create a right eye image, A stereoscopic image display device that outputs the left-eye image and the right-eye image so that the display displays a second stereoscopic image.

2. In paragraph 1, The above characteristic information is, A stereoscopic image display device including a weighting value that contributes to the estimation of visual fatigue according to the second depth of a pixel included in the flat image.

3. In either of paragraphs 1 and 2, The above characteristic information is, A stereoscopic image display device including depth information representing a plurality of depths for a plurality of areas of the above-mentioned flat image and a confidence value estimating the degree of error of the plurality of depths.

4. In any one of paragraphs 1 to 3, The above context information is, A stereoscopic image display device, comprising an object included in the flat image, a relative size of the object, and a position of the object.

5. In any one of paragraphs 1 to 4, The above context information is, A stereoscopic image display device including motion information related to the motion of the flat image and edge information for correcting the visual fatigue estimated based on an error in the second depth of the flat image.

6. In any one of paragraphs 1 to 5, The at least one instruction, when executed by the at least one processor, causes the at least one processor to: Obtain uncertainty information related to the uncertainty of the above depth map, A stereoscopic image display device that corrects the depth map based on the uncertainty information.

7. In any one of paragraphs 1 to 6, The at least one instruction, when executed by the at least one processor, causes the at least one processor to: A stereoscopic image display device that obtains loss information related to disparity between the above-mentioned flat image and the above-mentioned left-eye image.

8. In any one of paragraphs 1 to 7, The at least one instruction, when executed by the at least one processor, causes the at least one processor to: A stereoscopic image display device that reduces the three-dimensionality of a critical area whose depth is greater than a specified critical depth in the depth map.

9. In any one of paragraphs 1 to 8, The at least one instruction, when executed by the at least one processor, causes the at least one processor to: Based on the results of tracking the user's gaze, the region of interest is extracted from the flat image, A stereoscopic image display device that obtains the characteristic information based on the depth map and the region of interest.

10. In paragraph 9, The at least one instruction, when executed by the at least one processor, causes the at least one processor to: A stereoscopic image display device that corrects the color of the region of interest based on the depth map and the region of interest.

11. A method for controlling a stereoscopic image display device, The act of obtaining a depth map from a flat image; An operation of obtaining feature information related to a first depth from the above depth map; An operation of obtaining context information of the above flat image; An operation of calculating an estimated fatigue level of a user based on the depth map, the characteristic information, and the context information, wherein the estimated fatigue level represents visual comfort (VC) felt by the user when viewing a first stereoscopic image; An operation of correcting the depth map based on the estimated fatigue and target fatigue (target VC); An operation of generating a left eye image by correcting the flat image based on the corrected depth map; An operation of warping the left-eye image to generate a right-eye image; and A method including an operation of displaying a second stereoscopic image by outputting the left eye image and the right eye image.

12. In paragraph 11, The above characteristic information is, A method comprising a weighting value contributing to the estimation of visual fatigue according to a second depth of a pixel included in the above-mentioned flat image.

13. In any one of paragraphs 11 and 12, The above characteristic information is, A method comprising depth information representing multiple depths for each of multiple regions of the above-mentioned plane image and a confidence value estimating the degree of error of the multiple depths.

14. In any one of paragraphs 11 to 13, The above context information is, A method comprising an object included in the above plane image, a relative size of the object, and a position of the object.

15. In any one of paragraphs 11 to 14, The above context information is, A method comprising edge information for correcting the visual fatigue estimated based on motion information related to the motion of the planar image and an error in the second depth of the planar image.

Images