Electronic device, and method for providing 3D image including graphic object

Abstract

This electronic device comprises: a first display of an LFD type; a communication circuit; a camera; a memory that stores instructions; and at least one processor including processing circuitry. When the instructions are individually or collectively executed, the at least one processor: displays, on the first display, a visual object shared with an external device with which communication is performed through the communication circuit; when the visual object is moved in a Z-axis direction on a second display provided in the external device, receives, from the external device, first expected protrusion distance information according to the Z-axis movement of the visual object; identifies second expected protrusion distance information of the visual object on the basis of the first expected protrusion distance information and a viewing distance of a first user obtained through the camera; identifies a disparity between a left-eye image and a right-eye image including the visual object, on the basis of the second expected protrusion distance information; and displays, on the first display, an image including the visual object on the basis of the left-eye image and the right-eye image corresponding to the identified disparity.

Description

Method for providing 3D images including electronic devices and graphic objects

[0001] One or more embodiments of the present disclosure relate to an electronic device, and for example, to an electronic device for rendering and providing a 3D image including a graphic object, and a method for providing a 3D image.

[0002] Portable electronic devices (hereinafter, "electronic devices"), such as smartphones, can provide users with diverse user experiences through various applications and multimedia content. Electronic devices may include a display for providing applications and multimedia content. Electronic devices may provide graphical objects, such as application icons and widgets, through the display, and may provide various information and / or user interaction through these graphical objects.

[0003] To provide users with a more immersive experience, 3D (three-dimensional) displays are being offered. 3D displays can be categorized into glasses-based and glasses-free types, and electronic devices such as smartphones are being developed with glasses-free 3D displays due to their inherent characteristics.

[0004] An electronic device according to one or more embodiments comprises: a first display of a Light Field Display (LFD) type; a memory storing instructions; a camera; a communication circuit; And at least one processor including processing circuitry; wherein the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to display a visual object shared with an external device performing communication through the communication circuit on the first display, and when the visual object is moved in the z-axis direction on a second display provided in the external device, receive first expected saliency distance information according to the z-axis movement of the visual object from the external device, identify second expected saliency distance information of the visual object based on the first expected saliency distance information and a viewing distance of the first user acquired through the camera, identify parallax of a left-eye image and a right-eye image including the visual object based on the second expected saliency distance information, and display a 3D image including the visual object on the first display based on the left-eye image and the right-eye image corresponding to the identified parallax.

[0005] A method for providing a three-dimensional image of an electronic device including a first display and a camera of an LFD type according to one or more embodiments includes: displaying a visual object shared with an external device with which communication is being made on the first display; receiving first expected saliency distance information according to a z-axis movement of the visual object from an external device when the visual object is moved in the z-axis direction on a second display provided on the external device; identifying second expected saliency distance information of the visual object based on the first expected saliency distance information and a viewing distance of a first user acquired through the camera; identifying parallax of a left-eye image and a right-eye image including the visual object based on the second expected saliency distance information; and displaying a 3D image including the visual object on the first display based on the left-eye image and the right-eye image corresponding to the identified parallax.

[0006] A non-transitory computer-readable medium storing computer instructions that, when executed by a processor of an electronic device including a first display and a camera of an LFD type according to one or more embodiments, cause the electronic device to perform an operation, the operation comprising: displaying a visual object shared with an external device with which communication is being made on the first display; receiving, from the external device, first expected saliency distance information according to a z-axis movement of the visual object when the visual object is moved in the z-axis direction on a second display provided in the external device; identifying second expected saliency distance information of the visual object based on the first expected saliency distance information and a viewing distance of a first user acquired through the camera; identifying parallax of a left-eye image and a right-eye image including the visual object based on the second expected saliency distance information; and displaying a 3D image including the visual object on the first display based on the left-eye image and the right-eye image corresponding to the identified parallax.

[0007] The above and other aspects, features and advantages of specific embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings.

[0008] FIG. 1 is a block diagram of an electronic device within a network environment according to one or more embodiments.

[0009] FIGS. 2A and 2B illustrate the structure of a 3D display according to one embodiment.

[0010] FIG. 3 illustrates a method for an electronic device to provide a three-dimensional effect using two images according to one embodiment.

[0011] FIG. 4A is a block diagram of an electronic device according to one or more embodiments.

[0012] FIG. 4b is a block diagram illustrating an implementation example of an electronic device according to one or more embodiments.

[0013] FIG. 5 is a drawing for explaining a method for sharing visual objects between an electronic device (400) and an external device (500) according to one or more embodiments.

[0014] FIG. 6 is a drawing for explaining a visual object display method of an electronic device (400) according to one or more embodiments.

[0015] FIG. 7 is a drawing for explaining a method for displaying an expected protrusion range of an electronic device (400) and an external device (500) according to one or more embodiments.

[0016] FIGS. 8A and 8B are drawings for explaining a display method according to a z-axis movement distance of a visual object according to one or more embodiments.

[0017] FIGS. 9A and 9B are drawings for explaining a method for providing a visual object according to one or more embodiments.

[0018] FIG. 10 is a diagram illustrating a method for providing user images and visual objects according to one or more embodiments.

[0019] FIGS. 11A and 11B are drawings for explaining a method of providing a screen according to a viewing distance of a first user and a second user according to one or more embodiments.

[0020] FIGS. 12A to 12C are drawings for explaining a method of displaying additional visual information according to one or more embodiments.

[0021] FIGS. 13A to 13D are drawings for explaining a method of displaying additional visual information according to one or more embodiments.

[0022] FIG. 14 is a drawing for explaining a method of providing a screen in a plurality of devices according to one or more embodiments.

[0023] FIGS. 15A and 15B are drawings for explaining a method for setting a viewpoint of a visual object according to one or more embodiments.

[0024] FIGS. 16A to 16C are drawings for explaining a screen providing method according to an implementation example of a plurality of devices according to one or more embodiments.

[0025] FIGS. 17A and 17B are drawings for explaining a method for providing additional visual information according to one or more embodiments.

[0026] FIGS. 18A and 18B are drawings for explaining a method for providing additional visual information according to one or more embodiments.

[0027] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

[0028] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to one or more embodiments. Referring to FIG. 1 , in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) or the server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0029] The processor (120) may control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing, for example, software (e.g., a program (140)), and may perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculation, the processor (120) may store a command or data received from another component (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the command or data stored in the volatile memory (132), and store the resulting data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0030] The auxiliary processor (123) may control at least a part of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0031] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0032] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0033] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0034] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0035] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0036] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0037] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0038] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0039] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0040] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0041] The camera (180) can capture still images and videos. In one embodiment, the camera (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0042] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0043] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0044] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0045] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) may support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0046] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. According to some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0047] According to one or more embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first surface (e.g., a bottom surface) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second surface (e.g., a top surface or a side surface) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

[0048] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0049] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service by itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0050] Hereinafter, the direction perpendicular to the display (or display panel) of the electronic device may be defined as the z-axis direction, the horizontal direction of the display perpendicular to the z-axis may be referred to as the x-axis direction, and the vertical direction may be referred to as the y-axis direction.

[0051] FIGS. 2A and 2B illustrate the structure of a 3D display according to one embodiment.

[0052] According to one embodiment, an electronic device (e.g., electronic device (101) of FIG. 1) may include a display (e.g., display module (160) of FIG. 1) that outputs various images. The images output from the display of the electronic device may be 3D (3 dimensional) images that provide a three-dimensional effect to the user.

[0053] 3D displays include a glasses-based method that allows a user to feel a three-dimensional effect when wearing glasses, and a glasses-free method that allows a user to feel a three-dimensional effect based on a structure arranged on the front of the display panel (210) without wearing glasses.

[0054] According to one embodiment, an electronic device can implement a glasses-free 3D display. The electronic device utilizes binocular parallax to generate a left-eye image (or first image) that can be recognized by the user's left eye (10) and a right-eye image (or second image) that can be recognized by the user's right eye (20), and can simultaneously display the left-eye image and the right-eye image using designated pixels of a display panel (210). For example, the glasses-free 3D display can be implemented as an LFD (Light Field Display) type.

[0055] According to one embodiment, the electronic device may include an optical structure disposed in front of the display panel (210) (or in the +z direction from the display panel (210)) such that a left-eye image output from the display panel (210) is substantially recognized by the user's left eye (10) and not recognized by the user's right eye (20), and a right-eye image is substantially recognized by the user's right eye (20) and not recognized by the user's left eye (10). For example, the electronic device may include a lenticular lens structure (260) of FIG. 2A or a parallax barrier structure (270) of FIG. 2B such that the left-eye image and the right-eye image are recognizable by the left eye (10) and the right eye (20), respectively.

[0056] Referring to FIG. 2A, a film (or glass) including a plurality of lenticular lenses (260) may be disposed on the front of a display panel (210) of an electronic device. According to one embodiment, the plurality of lenticular lenses (260) included in the film may cover each pixel column or each group including a plurality of pixel columns. A left-eye image may be displayed on odd-numbered pixel columns (or groups) (220) on the display panel (210), and a right-eye image may be displayed on even-numbered pixel columns (or groups) (230).

[0057] According to one embodiment, the lenticular lenses covering the pixel column (220) on which the left-eye image is displayed may refract light corresponding to the output image in a direction corresponding to the position of the user's left eye (10), and the lenticular lenses covering the pixel column (230) on which the right-eye image is displayed may refract light corresponding to the output image in a direction corresponding to the position of the user's right eye (20). Here, the direction corresponding to the left-eye (10) position and the direction corresponding to the right-eye (20) position may be directions toward the positions of the user's left eye (10) and right eye (20) in each pixel column when the user looks straight ahead at a predetermined distance from the display. According to this structure, even if the left-eye image and the right-eye image are simultaneously output on the display panel (210), only the left-eye image can be substantially recognized through the user's left eye (10), and only the right-eye image can be substantially recognized through the user's right eye (20).

[0058] Referring to FIG. 2B, a parallax barrier (270) may be arranged on the front of the display panel (210) of the electronic device. According to one embodiment, the parallax barrier (270) may include a blocking bar (or black bar) (274) that blocks light output from the display panel (210) and a slit (272) that allows light to pass therethrough, which may be arranged to intersect. The horizontal widths of the blocking bars (274) and the slits (272) may correspond to the horizontal widths of a pixel row or a group including multiple pixel rows that intersect and output a left-eye image and a right-eye image. As the blocking bar (274) and the slit (272) are arranged to intersect in the parallax barrier (270), the left-eye image output from the odd-numbered pixel column (or group) (220) passes through the slit (272) and is recognized by the user's left eye (10), and the right-eye image output from the even-numbered pixel column (or group) (230) can be recognized by the user's right eye (20). In addition, the left-eye image is blocked by the blocking bar (274) in the direction toward the user's right eye (20) and is thus substantially not recognized by the user's right eye (20), and the right-eye image is blocked by the blocking bar (274) in the direction toward the user's left eye (10) and is thus substantially not recognized by the user's left eye (10).

[0059] In the present disclosure, an example in which an electronic device implements a 3D screen using a lenticular lens structure (260) of FIG. 2A or a parallax barrier structure (270) of FIG. 2B is described, but the present invention is not limited thereto, and the electronic device may include a glasses-free 3D display of another type.

[0060] FIG. 3 illustrates a method for an electronic device to provide a three-dimensional effect using two images according to one embodiment.

[0061] According to one embodiment, the electronic device may provide a 3D image that provides a three-dimensional effect to the user by using a 3D display such as the lenticular lens structure (260) of FIG. 2A or the parallax barrier structure (270) of FIG. 2B. For example, the electronic device may generate a left-eye image and a right-eye image, respectively, and display the left-eye image and the right-eye image crosswise through predetermined pixel columns (e.g., odd-numbered columns and even-numbered columns) of the display panel (210), so that the left-eye image may be recognized by the user's left eye (10) and the right-eye image may be recognized by the user's right eye (20).

[0062] Referring to (a) of FIG. 3, when the graphic object of the left-eye image and the graphic object of the right-eye image are output through adjacent pixels on the display panel (210), the graphic object (300) of the 3D image recognized by the user as the graphic object of the left-eye image and the graphic object of the right-eye image are output can be recognized at the position of the display panel (210) based on the z-axis direction from the user's line of sight.

[0063] Referring to (b) of FIG. 3, when the graphic object (311) of the left-eye image is positioned on the left (or -x direction) and the graphic object (321) of the right-eye image is positioned on the right (or +x direction) based on the x-axis direction, the graphic object (301) of the 3D image can be recognized at a position further from the display panel (210) based on the z-axis direction from the user's line of sight. When the distance between the graphic object (311) of the left-eye image and the graphic object (321) of the right-eye image is formed to be further than in the example of (b) of FIG. 3, the graphic object (301) of the 3D image can be recognized at a position further from the user's line of sight based on the z-axis direction.

[0064] Referring to (c) of FIG. 3, when the graphic object (312) of the left eye image is positioned on the right (or +x direction) based on the x-axis direction and the graphic object (322) of the right eye image is positioned on the left, the graphic object (302) of the 3D image can be recognized at a position closer to the display panel (210) based on the z-axis direction from the user's line of sight. When the distance between the graphic object (312) of the left eye image and the graphic object (322) of the right eye image is formed to be greater than in the example of (c) of FIG. 3, the graphic object (302) of the 3D image can be recognized at a position closer to the z-axis direction from the user's line of sight.

[0065] According to one embodiment, the electronic device can control the depth perception of a 3D image perceived by a user by adjusting the relative positions and / or distances of the left-eye image and the right-eye image. For example, the electronic device can render a left-eye image and a right-eye image including a graphic object corresponding to a graphic object of a 3D image to provide a 3D effect for a specific graphic object, and can determine the position of each graphic object on the left-eye image and the right-eye image based on the z-axis position of the determined graphic object.

[0066] FIG. 4A is a block diagram of an electronic device according to one or more embodiments.

[0067] Referring to FIG. 4A, an electronic device (400) according to an embodiment may include a display (200), a processor (410), a memory (420), a camera (430), and a communication circuit (440). Even if some of the components illustrated in FIG. 4A are omitted or replaced, one or more embodiments of the present disclosure may be implemented. At least some of the illustrated components may be operatively, electrically, and / or functionally connected to each other. The electronic device (400) may include at least some of the components and / or functions of the electronic device (101) of FIG. 1.

[0068] According to one embodiment, some of the components of the electronic device (400) (e.g., processor (410), memory (420)) may be disposed inside the housing of the electronic device (400), and other components (e.g., display (200), camera (430)) may have at least some of them exposed outside the housing.

[0069] According to one embodiment, the display (200) can output various images provided from the processor (410). For example, the display (200) can be implemented as a liquid crystal display (LCD), a light-emitting diode (LED) display, or an organic light-emitting diode (OLED) display, but is not limited thereto. The display (200) can be configured as a touch screen that detects a touch and / or proximity touch (or hovering) input using a part of a user's body (e.g., a finger) or a stylus. The display (200) can include at least some of the configurations and / or functions of the display module (160) of FIG. 1.

[0070] According to one embodiment, the display (200) may be implemented as a 3D display that provides a 3D image to a user. For example, the display (200) may include a structure such as a lenticular lens (e.g., a lenticular lens (260) of FIG. 2A) or a parallax barrier (e.g., a parallax barrier (270) of FIG. 2B) arranged in front of a display panel including a plurality of pixels. The structure of the display (200) for implementing a 3D effect has been described with reference to FIGS. 2A and 2B.

[0071] According to one embodiment, the processor (410) may render a left-eye image that can be recognized as the user's left eye and a right-eye image that can be recognized as the user's right eye and display them alternately in odd-numbered pixel columns (or a group including pixel columns) and even-numbered columns (or a group including pixel columns) of the display (200), respectively. The processor (410) may control the z-axis direction position and depth perception of the 3D image perceived by the user by adjusting the relative position and / or distance of the left-eye image and the right-eye image. An image rendering method of the processor (410) for implementing a 3D effect has been described with reference to FIG. 3. Hereinafter, the left-eye image may be referred to as a first image, and the right-eye image may be referred to as a second image.

[0072] According to one embodiment, the camera (430) can capture a surrounding subject, convert the image information into digital data, and provide it to the processor (410). The electronic device (400) can include at least one camera (430) on the front side where the display (200) is included in the housing and / or the rear side opposite thereto. According to one embodiment, the camera (430) can include a lens assembly including at least one lens that collects light emitted from an external environment (or a subject), an image sensor (e.g., a CCD (charged coupled device) sensor, a CMOS (complementary metal oxide semiconductor) sensor) that converts the light collected through the lens assembly into an electrical signal to generate image data, and an image signal processor that performs various processing on the image data acquired from the image sensor. At least some of the above-described components of the camera (430) can be omitted or replaced with other components. The camera (430) can provide an image captured from the external environment to the processor (410) in real time through an interface (e.g., a mobile industry processor interface). For example, the camera (430) may include at least one of a regular (or basic) camera, an ultra-wide-angle camera, a depth camera, or an IR camera. The camera (430) may include at least some of the configurations and / or functions of the camera (180) of FIG. 1.

[0073] According to one embodiment, the processor (410) can track the user's gaze position (or head position) based on an image acquired from a camera (430) (e.g., a front camera). For example, the processor (410) can analyze an image acquired in real time from the camera (430) to extract the user's eye area and track the user's gaze position by monitoring the movement of the pupil. For example, the processor (410) can input the image acquired from the camera (430) into a trained artificial intelligence model to acquire the user's gaze position.

[0074] According to one embodiment, the processor (410) can track the user's location, for example, the user's gaze location, based on sensing data acquired through various types of sensors, as well as the camera (430). For example, the sensors may include various types of sensors, such as an optical sensor, an IR sensor, a depth sensor, a thermal imaging sensor, and a laser sensor.

[0075] According to one embodiment, the electronic device (400) may include a communication circuit (440). The electronic device (400) may include at least some of the configurations and / or functions of the communication module (190) of FIG. 1. According to one example, the electronic device (400) may communicate with an external device using the communication circuit (440).

[0076] According to one embodiment, the memory (420) may temporarily or permanently store various data, including volatile memory and non-volatile memory. The memory (420) may include at least some of the configuration and / or functions of the memory (130) of FIG. 1, and may store the program (140) of FIG. 1. The memory (420) may store various instructions that may be executed by the processor (410). Such instructions may include control commands such as arithmetic and logical operations, data movement, and input / output that may be recognized by the processor (410).

[0077] According to one embodiment, the processor (410) may be configured as a component capable of performing calculations or data processing related to control and / or communication of each component of the electronic device (400), and may be configured as at least one processor. The processor (410) may include at least some of the configurations and / or functions of the processor (120) of FIG. 1. The processor (410) may be operatively, functionally, and / or electrically connected to each component of the electronic device (400), such as the display (200), the camera (430), the sensor (450), and the memory (420). The calculation and data processing functions that the processor (410) may implement on the electronic device (400) are not limited, but in the present disclosure, one or more embodiments for determining the z-axis direction position of a graphic object based on an attribute of the graphic object and rendering a 3D image so that the image can be recognized at the determined z-axis direction position will be described. The operations of the processor (410) to be described later can be performed by loading instructions stored in the memory (420).

[0078] According to one embodiment, the electronic device (400) can render a 2D image or a 3D image and output it through the display (200). For example, the processor (410) can generate a first image (or a left-eye image) and a second image (or a right-eye image) constituting a 3D image, respectively, and place pixel data of the first image in odd-numbered pixel columns (or pixel column groups including pixels) and place pixel data of the second image in even-numbered pixel columns (or pixel column groups including pixels), thereby generating a 3D image. When a 3D image is output by the display (200), only the first image may be substantially recognized by the user's left eye, and only the second image may be substantially recognized by the user's right eye, depending on an optical structure such as a lenticular lens or a parallax barrier.

[0079] According to one embodiment, an image output through the display (200) may include at least one graphic object. Here, the graphic object may include at least one of a widget, an icon, an application execution screen, or a pop-up window, but is not limited thereto. Hereinafter, a widget will be described as an example of a graphic object, but the embodiments described below may also be applied to other types of graphic objects other than widgets.

[0080] According to one embodiment, the processor (410) can generate a graphic object as a 2D graphic object or a 3D graphic object, and can compose a 3D image using the 2D graphic object and / or the 3D graphic object. For example, even if a first graphic object and a second graphic object displayed in an image are both 2D graphic objects, if the z-axis direction positions of the first graphic object and the second graphic object are different, the user can feel a three-dimensional effect. The 3D image can include at least one 2D graphic object and at least one 3D graphic object, and at least one 2D or 3D graphic object can be recognized at a different position in the z-axis direction from other graphic objects.

[0081] According to one embodiment, the processor (410) may track a real-time viewing distance of a user (hereinafter, a first user) of the electronic device (400) based on an image acquired from the camera (430), and determine a position of a graphic object displayed on a 3D image based on the real-time viewing distance. For example, the real-time viewing distance of the user may include a gaze position of the first user. For example, the processor (410) may acquire a pupil image of the first user in real time using the front camera (430), and track the gaze position according to a change in the position of the pupil.

[0082] According to one embodiment, when movement information about a visual object shared with an external device (500) performing communication is received from the external device (500), the processor (410) can adjust and display the degree of protrusion of the visual object based on the received information.

[0083] For example, the information received from the external device (500) may be first expected protrusion distance information according to the movement of the visual object in the z-axis direction on the second display provided in the external device (500). The first expected protrusion distance may be information predicting how much the visual object is displayed to be protruded toward the user (hereinafter, the second user) of the external device (500) according to the z-axis movement of the visual object on the second display provided in the external device (500). For example, the first expected protrusion distance may be identified based on the z-axis movement distance of the visual object and the real-time viewing distance of the second user. For example, the first expected protrusion distance may be identified based on the z-axis movement distance of the visual object, the real-time viewing distance of the second user, and the three-dimensional maximum protrusion distance of the second display.

[0084] As another example, the information received from the external device (500) may be movement position information (e.g., 3D movement coordinate information) of a visual object as it moves in the z-axis direction on a second display provided in the external device (500) and viewing distance information of the second user. For example, the electronic device (400) may identify first expected protrusion distance information of the visual object on the second display based on the received movement position information of the visual object and the viewing distance information of the second user. For example, the electronic device (400) may identify first expected protrusion distance information of the visual object on the second display based on the received movement position information of the visual object, the viewing distance information of the second user, and the three-dimensional maximum protrusion distance of the second display. The electronic device (400) may pre-store the three-dimensional maximum protrusion distance of the second display or may receive it from the external device (500).

[0085] According to one embodiment, the processor (410) can identify second expected saliency distance information of a visual object based on the first expected saliency distance information and the real-time viewing distance of the first user.

[0086] In one example, if the processor (410) identifies that the visual object is provided based on negative parallax from the external device (500) based on the first expected saliency distance, the processor (410) may identify second expected saliency distance information of the visual object based on positive parallax. Alternatively, if the processor (410) identifies that the visual object is provided based on positive parallax from the external device (500) based on the first expected saliency distance, the processor (410) may identify second expected saliency distance information of the visual object based on negative parallax.

[0087] In stereoscopic images, parallax is the difference between two images. When this value is positive (+), the visual object appears to be behind the screen, and when it is negative (-), the visual object appears to be in front of the screen. Accordingly, a parallax value that makes a visual object appear to protrude from the screen can be negative parallax, and a parallax value that makes a visual object appear to be sunken from the screen can be positive parallax.

[0088] According to one embodiment, the processor (410) may identify a parallax between a left-eye image and a right-eye image including a visual object based on the second expected protrusion distance information, and display an image including a visual object on the first display (200) based on the left-eye image and the right-eye image corresponding to the identified parallax.

[0089] FIG. 4b is a block diagram illustrating an implementation example of an electronic device according to one or more embodiments.

[0090] According to one embodiment, the electronic device (400) may include at least one sensor (450). For example, the electronic device (400) may further include various types of sensors, such as an acceleration sensor, a gesture sensor, a gyro sensor, a pressure sensor, a magnetic sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor. The electronic device (400) may include at least some of the configurations and / or functions of the sensor module (176) of FIG. 1.

[0091] According to one embodiment, the electronic device (400) can detect the inclination (or angle formed with respect to the ground) of the electronic device (400) using at least one sensor (450) (e.g., an acceleration sensor, a gyro sensor). For example, the acceleration sensor can sense acceleration in the direction of three axes (x-axis, y-axis, z-axis), the gyro sensor can sense angular velocity, and the processor (410) can determine the inclination of the electronic device (400) based on acceleration data obtained from the acceleration sensor and angular velocity data obtained from the gyro sensor.

[0092] According to one embodiment, the electronic device (400) may include a stylus that a user can use to input on the display (200). The stylus may be configured to be inserted and removed into the housing of the electronic device (400). For example, the user may use the stylus to make a touch input on the display (200) and may make a hovering input (or proximity touch input) while spaced apart from the display (200). The stylus may have an overall thin and long shape, and the electronic device (400) may include a groove into which the stylus can be inserted. The stylus may include at least one button, and when a user input on the button is detected, a signal corresponding to the user input may be transmitted to the electronic device (400) via short-range wireless communication.

[0093] FIG. 5 is a drawing for explaining a method for sharing visual objects between an electronic device (400) and an external device (500) according to one or more embodiments.

[0094] According to FIG. 5, the electronic device (400) can perform remote communication with an external device (500) located in a different space. For example, the electronic device (400) and the external device (500) can perform remote communication for various purposes, such as video chatting, online lectures, real-time counseling, online shopping, etc.

[0095] For example, the electronic device (400) can execute an application (or software or program) capable of video calling (hereinafter, video calling application) (hereinafter, video calling application) and transmit a captured image acquired through the camera (430) to an external device (500). The video calling application can be provided in the form of an icon interface on the screen of the electronic device (400). In one example, when the video calling application is executed, the camera (430) can be automatically turned on, but in some cases, it can be turned on according to a user command.

[0096] The external device (500) may be a user terminal of a second user who is a video call participant connected via a video call application. For example, the captured video may be transmitted to the external device (500) via an external server (not shown) that communicates with the video call application. However, for convenience of explanation, the following description will describe the transmission from the electronic device (400) to the external device (500).

[0097] According to one example, a captured image obtained through a camera (430) provided in an electronic device (400) may be an image (hereinafter, “first user image”) of a user (hereinafter, “first user”) of the electronic device (400). In this case, the first user image (520) may be displayed on the display of the external device (500). On the other hand, the second user image (510) obtained through the camera of the external device (500) may be displayed on the display (200) of the electronic device (400) (hereinafter, “first display (200)”).

[0098] According to one embodiment, the display (200) of the electronic device (400) may be implemented as an LFD (Light Field Display) type. According to one example, the display of the external device (500) may be implemented as an LFD or another type of display, but for the convenience of explanation, it is assumed that it is implemented as an LFD. Hereinafter, the case where the displays of the electronic device (400) and the external device (500) that perform mutual communication are both implemented as LFDs is referred to as an LFD environment.

[0099] According to one embodiment, in an LFD environment, a first user of an electronic device (400) and a second user of an external device (500) may perform video communication and share a visual object (531). The visual object may be implemented as various objects depending on the communication environment, communication purpose, etc. between the electronic device (400) and the external device (500). For example, the visual object may include various types of UI objects, such as game-related objects, education-related objects, shopping-related objects, and sponsorship-related objects.

[0100] FIG. 6 is a drawing for explaining a visual object display method of an electronic device (400) according to one or more embodiments.

[0101] According to FIG. 6, in operation 605, the electronic device (400) may activate the UFD function according to a preset event. For example, when the electronic device (400) and the external device (500) are switched to a 3D sharing mode according to a preset event in a video chat situation, the LFD function in an inactive state may be activated and the camera (430) may be turned on. The 3D sharing mode may be a mode in which the electronic device (400) and the external device (500) share a visual object (531). The LFD function being activated may mean that the first display (200) is operating in a 3D mode. For example, the electronic device (400) may control the plurality of lenticular lenses (260) to operate in the 3D mode by applying a preset voltage corresponding to the 3D mode to the plurality of lenticular lenses (260). The LFD function being inactive may mean that the first display (200) is operating in 2D mode. For example, the electronic device (400) may control the plurality of lenticular lenses (260) to operate in 2D mode by applying a preset voltage corresponding to the 2D mode to the plurality of lenticular lenses (260).

[0102] In operation 610, the electronic device (400) may overlay a virtual plane on the 2D screen when the LFD function is activated. The virtual plane is a plane where z=0 in the coordinate plane, an xy-plane in a three-dimensional space, and may be defined by the (x, y, 0) coordinates.

[0103] For example, when the LFD function is activated, the electronic device (400) may overlay a second image layer (or graphic layer) including a virtual plane on a first image layer (or background layer) including a second user image received from an external device (500). An image layer is a concept for independently processing individual elements that constitute an image by separating them, and a layer stack in which each image layer is stacked generates a final image.

[0104] In operation 615, the electronic device (400) can place a visual object on a virtual plane.

[0105] For example, each image layer may have transparency, and the first image layer and the second image layer may be mixed by adjusting the transparency of each layer. For example, the electronic device (400) may mix the first image layer and the second image layer using an alpha blending image (or alpha map). The alpha blending image may be a map in which an alpha value corresponding to a pixel position of a first area (a background image including a second user image) is smaller than a threshold value, and an alpha value corresponding to a pixel position of a second area (a visual object) is larger than the threshold value. For example, when an alpha value corresponding to the first area is 0 and an alpha value corresponding to a background area is 255, the first area has high transparency and the second area has low transparency through alpha blending. Accordingly, the first area includes the second user image, and the second area includes a visual object, so that a final image can be obtained.

[0106] In operation 620, the electronic device (400) can identify the z-axis movement distance of the visual object. For example, the electronic device (400) can identify the z-axis movement distance of the visual object relative to the virtual plane.

[0107] In operation 625, the electronic device (400) may identify an expected protrusion distance based on the z-axis movement distance of the visual object. The expected protrusion distance may be information predicting how much the visual object will be protruded toward the user based on the z-axis movement of the visual object. For example, the expected protrusion distance may be identified based on the z-axis movement distance of the visual object, the real-time viewing distance of the first user, and the maximum three-dimensional protrusion distance of the first display (200).

[0108] In operation 630, the electronic device (400) can adjust the binocular parallax distance of the left-eye image and the right-eye image including the visual object based on the expected protrusion distance identified in operation 625. For example, the electronic device (400) can calculate the binocular parallax of the left-eye image and the right-eye image so that the visual object appears to be protruded to the user by the expected protrusion distance identified in operation 625, and can acquire the left-eye image and the right-eye image based on the calculated binocular parallax.

[0109] In operation 635, the electronic device (400) may display a screen including a visual object based on the left-eye image and the right-eye image with adjusted binocular parallax intervals. For example, as illustrated in FIG. 5 , the electronic device (400) may provide a stereoscopic effect to a first user, making the visual object (531) appear to protrude forward, i.e., making the object appear to be in front of the screen. On the other hand, the external device (500) may provide a stereoscopic effect to a second user, making the visual object (531) appear to be recessed backward, i.e., making the object appear to be behind the screen.

[0110] Meanwhile, in Fig. 6, the order is mapped for all steps for convenience of explanation, but it is of course not necessarily limited to the order of steps that are not related to the order or can be performed in parallel.

[0111] FIG. 7 is a drawing for explaining a method for displaying an expected protrusion range of an electronic device (400) and an external device (500) according to one or more embodiments.

[0112] According to FIG. 7, in operations 705 and 710, the electronic device (400) and the external device (500) may each display a visual object according to a preset event. For example, a visual object may be displayed on the electronic device (400) and the external device (500) according to a user command input through at least one of the electronic device (400) and the external device (500).

[0113] In operation 715, the electronic device (400) may identify the first viewing distance information of the first user. For example, the electronic device (400) may identify the first viewing distance and viewing angle of the first user based on the captured image of the first user obtained through the camera (430). For example, the viewing angle may be identified based on the shooting direction of the camera (430) (or the front of the first display (200).

[0114] In operation 720, the external device (500) can identify the second viewing distance information of the second user. For example, the external device (500) can identify the first viewing distance and viewing angle of the second user based on a captured image of the second user obtained through the equipped camera.

[0115] In operation 725, the electronic device (400) can identify a first expected protrusion range of a visual object based on the first viewing distance information of the first user and the maximum three-dimensionality protrusion distance of the first display (200). The first expected protrusion range may be a range corresponding to a Z-axis movement distance of a virtual space in which the first user can perceive the three-dimensionality. The first expected protrusion range may be determined based on the maximum three-dimensionality protrusion distance of the first display (200), but may be changed based on the first viewing distance information of the first user.

[0116] In operation 730, the external device (500) can identify a second expected range of the visual object based on the second viewing distance information of the second user and the maximum stereoscopic projection distance of the second display in the same / similar manner as operation 725. The second expected protrusion range may be a range corresponding to the Z-axis movement distance of the virtual space in which the second user can perceive stereoscopic effects. The second expected protrusion range may be determined based on the maximum stereoscopic projection distance of the second display, but may be changed based on the second viewing distance information of the second user.

[0117] In operation 735, the electronic device (400) may transmit information about a first expected protrusion range of a visual object to an external device (500). In one example, the first expected protrusion range may be in a predefined format promised to the external device (500).

[0118] In operation 740, the external device (500) may transmit information about the second expected protrusion range to the electronic device (400) in a manner identical / similar to operation 735.

[0119] In operation 735, the electronic device (400) is identified as calculating a first expected protrusion range and transmitting it to the external device (500), but this is not limited thereto. In another example, the electronic device (400) may transmit information on the first viewing distance of the first user and information on the maximum three-dimensional protrusion distance of the first display (200) to the external device (500), and the external device (500) may calculate the first expected protrusion range based on the received information. The same / similar operation may also be applied in operation 740.

[0120] In operation 745, the electronic device (400) may display a guide UI for a range within which the second user can perceive three-dimensionality based on the second expected protrusion range received from the external device (500). For example, the first user may limit the movement range of the visual object based on the guide UI.

[0121] In operation 750, the external device (500) may display a guide UI for a range within which the first user can perceive three-dimensionality based on the first expected protrusion range received from the electronic device (400). For example, the second user may limit the range of movement of the visual object based on the guide UI.

[0122] In operation 755, the electronic device (400) can identify whether the first viewing distance information of the first user has changed.

[0123] In operation 760, the electronic device (400) can identify whether the second viewing distance information of the second user has changed.

[0124] In operation 755, if the first viewing distance of the first user is changed (S755:Y), the changed first viewing distance information can be identified in operation 760.

[0125] In operation 760, when the second viewing distance of the second user changes (S760:Y), the changed second viewing distance information can be identified in operation 770.

[0126] In operation 775, the electronic device (400) can transmit the changed first viewing distance information to the external device (500).

[0127] In operation 780, the external device (500) can transmit the changed second viewing distance information to the electronic device (400).

[0128] In operation 785, the electronic device (400) can change and display the 3D expected protrusion range of the second user based on the changed second viewing distance information received from the external device (500).

[0129] In operation 790, the external device (500) can change and display the 3D expected protrusion range of the first user based on the changed first viewing distance information received from the electronic device (400).

[0130] Meanwhile, in Fig. 7, the order is mapped for all steps for convenience of explanation, but it is of course not necessarily limited to the order of steps that are not related to the order or can be performed in parallel.

[0131] FIGS. 8A and 8B are drawings for explaining a display method according to a z-axis movement distance of a visual object according to one or more embodiments.

[0132] According to one embodiment, the electronic device (400) and the external device (500) may perform remote communication in different spaces. In this case, the first virtual plane (or zero plane) of the first display (200) of the electronic device (400) and the second virtual plane of the second display of the external device (500) may be the same virtual plane. For example, the virtual plane is a plane where z=0 in a coordinate plane, which is an xy-plane in a three-dimensional space and may be defined by the coordinates (x, y, 0). That the first virtual plane and the second virtual plane are the same may mean that the first virtual plane and the second virtual plane provide the same three-dimensional effect standard. For example, the first virtual plane providing 0 depth to a visual object in the electronic device (400) and the second virtual plane providing 0 depth to a visual object in the external device (500) may be mapped to one virtual plane.

[0133] Hereinafter, for convenience of explanation, the first virtual plane and the second virtual plane mapped to provide the same three-dimensional effect standard in the electronic device (400) and the external device (500) are referred to as “virtual planes.” For example, when a 3D object is positioned on the virtual plane, neither the first user of the electronic device (400) nor the second user of the external device (500) experiences a three-dimensional effect.

[0134] According to FIG. 8a, a visual object (720) can have relative parallax information in the electronic device (400) and the external device (500) with respect to a virtual plane (810).

[0135] For example, as illustrated in FIG. 8A, a visual object (820) may be displayed so as to appear protruding to a first user of the electronic device (400) and so as to appear sunken to a second user of the external device (500). For example, parallax information (or depth information) may be expressed as a bit value. For example, the expression range of 8-bit parallax information may have a parallax value of -128 to +128. However, this is only an example, and the parallax information may be expressed in various bits (e.g., 9-bit, 10-bit, etc.) according to various criteria.

[0136] For example, based on a preset event, a visual object (820) as illustrated in FIG. 8B may be displayed to appear sunken to a first user of the electronic device (400) and to appear protruding to a second user of the external device (500). In this case, the visual object may have a positive parallax in the direction of the first user (the direction of the electronic device (400)) and a negative parallax in the direction of the second user (the direction of the external device (500)) with respect to the virtual plane. For example, the parallax size for displaying the visual object on the electronic device (400) and the parallax size for displaying the visual object on the external device (500) may be the same. The same parallax size may mean that the three-dimensional effect felt by the first user and the second user is the same. In this case, the parallax size processed by the processor may be different depending on the display characteristics of each of the electronic device (400) and the external device (500). For example, display characteristics may include information such as size (or resolution), panel type, and hardware configuration.

[0137] For example, when a visual object (820) is displayed with a negative parallax so that it appears to protrude to a first user as illustrated in FIG. 8A, and the visual object (820) is moved backward in the z-axis direction according to a first user input (e.g., a backward swiping input), the electronic device (400) may display the visual object (820) by reducing the size of the negative parallax. In this case, the visual object (820) may be displayed to the first user so that the degree of protrusion is gradually reduced. Meanwhile, when a visual object (720) is displayed with a positive parallax so that it appears to protrude to the first user as illustrated in FIG. 8A, it may be displayed so that it appears to be sunken to the second user. When the visual object (820) is moved backward in the z-axis direction according to the first user input in the electronic device (400), the external device (500) may display the visual object (820) by reducing the size of the positive parallax. In this case, the degree of sagging of the visual object (820) may be displayed to the second user on the external device (500) so that it appears to be gradually reduced.

[0138] Next, when the visual object (820) reaches the virtual plane (810) according to the first user input that continues in the electronic device (400), the visual object (720) may be displayed to the first user in a non-stereoscopic manner. In this case, the visual object (820) may be displayed to the second user in a non-stereoscopic manner in the external device (500).

[0139] Thereafter, if the visual object (820) is moved in the opposite direction of the virtual plane (810) according to the continued first user input, the visual object (820) may be displayed as sunken to the first user, and the degree of sunken may increase as the movement distance increases. In this case, the visual object (820) may be displayed as protruding to the second user in the external device (500), and the degree of protrusion may increase as the movement distance increases.

[0140] For convenience of explanation, in the following, the state in which a visual object (820) is displayed so as to protrude to the user is called a “protruding state,” and the state in which it is displayed so as to be sunken to the user is called a “sunken state.”

[0141] For example, the electronic device (400) may display the visual object (820) by gradually decreasing its size as the visual object (820) changes from a protruding state to a sunken state, and may display the visual object (820) by gradually increasing its size as the visual object (820) changes from a sunken state to a protruding state. However, this is merely an example, and the size of the visual object (820) may also be displayed to maintain the same size regardless of whether it is protruding or sunken.

[0142] As described above, according to one embodiment, a visual object (820) can be displayed according to contrasting parallax values ​​in an electronic device (400) and an external device (500) so that a first user and a second user can experience contrasting stereoscopic effects.

[0143] FIGS. 9A and 9B are drawings for explaining a method for providing a visual object according to one or more embodiments.

[0144] According to one embodiment, the maximum three-dimensional projection distance of an LFD may vary depending on the display characteristics, and thus the optimal viewing distance for a user may also vary. For example, the display characteristics may include information such as size (or resolution), panel type, and hardware configuration.

[0145] The maximum protrusion distance for stereoscopic effects can indicate the extent to which an object on the screen can be projected toward the user. For example, the maximum protrusion distance for stereoscopic effects for LFDs can be determined based on negative parallax.

[0146] Referring to Fig. 9a, the protrusion distance of a visual object according to the user's real-time viewing distance in LFD can be calculated as in the following mathematical expression 1.

[0147]

[0148] IPD (interpupillary distance) is the distance between the centers of the user's two eyes, and disparity can mean the difference between the R image and the L image.

[0149] However, depending on the characteristics of the first display (200), the expected protrusion distance actually perceived by the first user may differ. Accordingly, the expected protrusion distance of a visual object may be determined based on the maximum three-dimensional protrusion distance of the first display (200) and the real-time viewing distance of the first user.

[0150] For example, the electronic device (400) can identify the expected protrusion distance of the visual object (820) based on the three-dimensional maximum protrusion distance A of the first display (200) and the real-time viewing distance B of the first user, as illustrated in FIG. 9B. As an example, the electronic device (400) can transmit information on at least one of the three-dimensional maximum protrusion distance A or the real-time viewing distance B of the first user to the external device (500). As another example, the electronic device (400) can transmit the expected protrusion distance information of a predefined format identified based on the three-dimensional maximum protrusion distance A and the real-time viewing distance B of the first user to the external device (500). In this case, the electronic device (400) and the external device (500) can store a lookup table of a predefined format for the protrusion distance. For example, information in a predefined format for a plurality of protrusion distances (or protrusion levels) that are mapped to different protrusion degrees that can be perceived by a typical viewer may be stored in the form of a lookup table in the electronic device (400) and the external device (500).

[0151] In this case, the external device (500) can identify the parallax value of the visual object displayed on the external device (500) based on the information received from the electronic device (400). In the same manner, the electronic device (400) can identify the parallax value of the visual object to be displayed on the display (200) based on at least one of the three-dimensional maximum protrusion distance of the external device (500) received from the external device (500) or the real-time viewing distance of the second user, or can identify the parallax value of the visual object based on second expected protrusion distance information of a predefined format received from the external device (500).

[0152] According to one embodiment, the electronic device (400) may display a guide UI to guide a virtual space (or a manipulation range of a visual object) in which a visual object is provided, as illustrated in FIG. 9B. According to one example, the guide UI may be displayed based on information received from an external device (500). For example, the guide UI may be displayed to correspond to a virtual space calculated based on the three-dimensional maximum protrusion distance of the external device (500) received from the external device (500) and the real-time viewing distance of the second user. Accordingly, the visual object may be guided so as not to move to an area (C) that deviates from the virtual space. For example, the guide UI may be displayed in a form in which the first user can visually recognize the depth stage of the visual object. For example, the guide UI may be displayed in an area where there is no interference with screen image information or visual objects.

[0153] FIG. 10 is a diagram illustrating a method for providing user images and visual objects according to one or more embodiments.

[0154] According to one embodiment, the electronic device (400) can display an image including a second user image and a visual object of an external device (500). According to one example, the electronic device (400) can change the display status of the second user image, the guide UI, and the visual object in real time based on the gaze information (eye position information or head position information) of the first user. For example, the display status of the second user image, the guide UI, and the visual object can include a state in which at least one of the size, position, or shape of the second user image, the guide UI, and the visual object is changed.

[0155] For example, the electronic device (400) may acquire a captured image of a first user using the camera (130), and track the first user's pupils in the captured image to acquire the user's gaze information. For example, the electronic device (400) may acquire the first user's gaze information using a depth camera, or may acquire the first user's gaze information using multiple cameras. The multiple cameras may be spaced apart at a preset interval to capture different viewpoints. For example, the preset interval may be the same / similar distance as the distance between a person's two eyes, but is not limited thereto. In general, a person receives a two-dimensional image with left / right differences from both eyes, and can perceive a three-dimensional distance through a process in which the human brain fuses the input images. When multiple cameras are used, the electronic device (400) may acquire the first user's gaze information in three-dimensional space using the same mechanism. For example, the electronic device (400) may identify at least one of the user's head or eyes in the captured image. For example, the electronic device (400) can identify at least one of the user's head or eyes included in the captured image through at least one of object recognition, object detection, object tracking, or image segmentation.

[0156] For example, referring to the left drawing of the first line, if the electronic device (400) identifies that the first user's gaze position is in a downward direction based on a preset horizontal direction, the electronic device (400) may display a second user image (1011) and a guide UI (1012) corresponding to a viewing angle from below to above of the first user. The preset horizontal direction may be a horizontal line direction corresponding to the center of the first display (200) of the electronic device (400). For example, a visual object (1013) may be displayed at a location that does not block the second user area included in the second user image (1011).

[0157] For example, referring to the right drawing of the first line, if the electronic device (400) identifies that the first user's gaze position is in an upward direction based on a preset horizontal direction, the electronic device (400) may display a second user image (1021) and a guide UI (1022) corresponding to a viewing angle from above to below of the first user. The preset horizontal direction may be a horizontal line direction corresponding to the center of the first display (200) of the electronic device (400). For example, a visual object (1023) may be displayed at a location that does not block the second user area included in the second user image (1021).

[0158] For example, referring to the left drawing of the second line, if the electronic device (400) identifies that the first user's gaze position is in the right direction based on a preset vertical direction, the electronic device (400) may display a second user image (1031) and a guide UI (1032) corresponding to the viewing angle at which the first user views the screen from the right direction. The preset horizontal direction may be a vertical line direction corresponding to the center of the first display (200) of the electronic device (400). For example, the visual object (1033) may be displayed at a location that does not block the second user area included in the second user image (1031).

[0159] For example, referring to the right drawing of the second line, if the electronic device (400) identifies that the first user's gaze position is in the left direction based on a preset vertical direction, the electronic device (400) may display a second user image (1041) and a guide UI (1042) corresponding to the viewing angle at which the first user views the screen from the left direction. The preset horizontal direction may be a vertical line direction corresponding to the center of the first display (200) of the electronic device (400). For example, a visual object (1043) may be displayed at a location that does not block the second user area included in the second user image (1041).

[0160] For example, referring to the left drawing of the third line, if the electronic device (400) identifies that the first user's gaze position is in a forward position based on a preset distance, it can display a second user image (1051) and guide UI (1052) corresponding to the case where the first user is viewing the screen close to the screen. The preset distance may be a value set based on the virtual plane described above. For example, a visual object (1053) may be displayed at a location that does not block the second user area included in the second user image (1051).

[0161] For example, referring to the right drawing of the third line, if the electronic device (400) identifies that the first user's gaze position is at a rearward position based on a preset distance, the electronic device (400) may display a second user image (1061) and a guide UI (1062) corresponding to the case where the first user is viewing the screen close to the screen. The preset distance may be a value set based on the virtual plane described above. For example, a visual object (1063) may be displayed at a location that does not block the second user area included in the second user image (1061).

[0162] FIGS. 11A and 11B are drawings for explaining a method of providing a screen according to a viewing distance of a first user and a second user according to one or more embodiments.

[0163] According to one embodiment, the maximum protrusion distance of the three-dimensional effect in the first display (200) provided in the electronic device (400) and the second display provided in the external device (500) and the range of the virtual space and / or the additional visual information (or additional information) displayed on the screen may be changed according to the positions of the first and second users.

[0164] In Figures 11a and 11b, communication between a car salesperson and a customer is assumed and explained. For convenience of explanation, the car salesperson is assumed to be the first user, the customer is assumed to be the second user, the electronic device (400) is assumed to be the car salesperson's device, and the external device (500) is assumed to be the customer's device.

[0165] According to FIG. 11A, the left drawing may be a second user screen (1110), i.e., a screen provided by a customer's device, and the center drawing may be a first user screen (1120), i.e., a screen provided by a car salesperson's device. As an example, as shown in the right drawing, the first user screen (1120) may include a second user's image (1121), a guide UI (1122), a visual object (car object) (1123), and a black area (1124). For example, the black area (1124) may be provided in a remaining screen area other than the area where the second user's image (1121) or the guide UI (1122) is provided. For example, a shadow area of ​​the visual object (car object) (1123) may also be provided. However, this is merely an example, and an image of a background area other than the black area (1124) may also be provided.

[0166] For example, as shown in the right drawing, if the distance (1125) between the second user (customer) and the display included in the customer's device is greater than or equal to a threshold value, the size of the virtual space on the first user screen (car salesperson screen) (1120), for example, the z-axis movement range for a visual object (car object), may be greater than the threshold distance.

[0167] According to FIG. 11b, the left drawing may be a second user screen (1140), that is, a screen provided by a customer's device, and the center drawing may be a first user screen (1150), that is, a screen provided by a car salesperson's device. For example, as shown in the right drawing, the first user screen (1150) may include a second user's image (1151), a guide UI (1152), a visual object (car object) (1153), a black area (1154), and a menu UI (1155). For example, the menu UI (1155) may include various menus related to the visual object (1153). For example, the menu UI (1155) may include various menus related to the type, three-dimensionality, and / or interaction of the visual object (1153), and may be provided as a three-dimensional menu, but is not limited thereto. For example, a shadow area of ​​the guide UI (1152) and the visual object (car object) (1153) may also be provided.

[0168] For example, as shown in the right drawing, if the distance (1125) between the second user (customer) and the display included in the customer's device is less than a threshold value, the size of the virtual space on the first user screen (car salesperson screen) (1150), for example, the z-axis movement range for a visual object (car object), may be less than the threshold distance.

[0169] FIGS. 12A to 12C are drawings for explaining a method of displaying additional visual information according to one or more embodiments.

[0170] According to one embodiment, the electronic device (400) may provide various additional visual information related to a visual object. For example, the additional visual information may include various information related to the three-dimensionality and / or interaction of the visual object.

[0171] According to an example, the electronic device (400) may display a screen (1210) including a menu UI (1211) and a guide UI (1212), as illustrated on the left side of FIG. 12A. For example, the menu UI (1211) may include various menus related to visual objects. For example, the menu UI (1211) may include various menus related to the type, three-dimensionality, and / or interaction of the visual object. For example, the menu UI (1211) may be provided as a three-dimensional menu. For example, the guide UI (1212) may be provided in a form that allows for recognizing a sense of distance in a virtual space corresponding to a 3D protrusion distance.

[0172] For example, when a visual object is selected through a menu UI (1211) and a location (1212-1) of the visual object is selected through a guide UI (1212) as shown on the right side of FIG. 12B, a visual object (1213) can be provided at the selected location. For example, when a specific location (1212-1) of guide coordinates divided in a checkerboard pattern is selected, a visual object (1213) can be provided at a location spaced a certain distance from the selected guide coordinates.

[0173] For example, a screen (1220) including a specific icon (1221) and a visual object (1222) may be displayed, as shown on the left side of FIG. 12B. For example, the specific icon (1221) may be a toggle-type menu icon for turning the guide UI ON / OFF. However, this is not limited thereto, and the ON / OFF menu may also be provided as a separate icon.

[0174] For example, when the guide UI is turned ON by toggling a specific icon (1221) as shown on the right side of FIG. 12b, the electronic device (400) can display the guide UI (1223) as shown on the left side of FIG. 12b and remove the specific icon (1221) from the screen.

[0175] According to an example, the electronic device (400) may include a screen (1230) including a visual object (1231) and a guide UI (1232), as shown on the left side of FIG. 12c.

[0176] For example, when a first user of an electronic device (400) moves a visual object (1231) in the z-axis direction, if the distance exceeds a specific z-axis distance range (1233) (1234), a crosstalk phenomenon (or 3D breakage phenomenon) may occur to a second user of an external device (500). The specific z-axis distance range (1233) may correspond to a safe viewing distance at which the second user can view the visual object without a crosstalk phenomenon from an external device (500) that displays the visual object by receiving z-axis movement distance information of the visual object (1231) from the electronic device (200).

[0177] For example, the electronic device (400) can reproduce and display a crosstalk screen that can be shown to a second user from an external device (500), as illustrated on the right side of FIG. 12C. For example, the electronic device (400) can reproduce and display a blur effect that becomes increasingly severe for the second user as the user moves away from a specific z-axis distance range (1233). Accordingly, the first user can intuitively recognize how much crosstalk occurs to the other party, i.e., the second user, depending on the z-axis movement distance of the visual object (1231).

[0178] FIGS. 13A to 13D are drawings for explaining a method of displaying additional visual information according to one or more embodiments.

[0179] According to one embodiment, the electronic device (400) may display a screen (1310) including a second user image (1311), a guide UI (1312), and a visual object (1313) of an external device (500), as illustrated in FIG. 13A. According to one example, the guide UI (1312) may be provided in a form that allows for recognizing a sense of distance in a virtual space corresponding to a 3D protrusion distance. For example, the guide UI (1312) may be displayed in different colors so as to allow for stepwise recognizing of a distance from the closest distance to the farthest distance on a virtual plane, as illustrated in FIG. 13A.

[0180] According to one embodiment, the electronic device (400) may display a screen (1320) including a second user image (1321), a guide UI (1322), and a visual object (1323) of an external device (500), as illustrated in FIG. 13b. For example, the guide UI (1321) may be provided in the form of a three-dimensional guide on four sides of the screen. For example, the guide UI (1321) may be provided in the form of a cage that can recognize the entire space in which the visual object can move, as illustrated in FIG. 13b.

[0181] According to one embodiment, the electronic device (400) may display a screen (1330) including a second user image (1331), a guide UI (1332), and a visual object (1333) of an external device (500), as illustrated in FIG. 13C. For example, the guide UI (1331) may be provided in the form of a navigation bar on one side of the screen. For example, the guide UI (1331) may be in the form of a navigation bar in which z-axis values ​​for 3D protrusion distances are displayed in different gray levels, as illustrated in FIG. 13C. For example, the guide UI (1331) may include the z-axis position (1332-1) of the current visual object and the z-axis position (1332-2) at which crosstalk occurs. In this case, unlike the crosstalk reproduction method illustrated in FIG. 12C, only simplified information about the stereoscopic effect that the other party can feel can be provided.

[0182] According to one embodiment, the electronic device (400) may display a screen (1340) including a second user image (1341), a guide UI (1342), and visual objects (1343, 1344) of an external device (500), as illustrated in FIG. 13d. For example, the electronic device (400) may provide the guide UI (1342) in a form designed based on the communication purpose, communication situation, type of visual objects (1343, 1344), etc. Accordingly, it is possible to provide a virtual space as a space of various environments.

[0183] FIG. 14 is a drawing for explaining a method of providing a screen in a plurality of devices according to one or more embodiments.

[0184] According to one embodiment, when the device characteristics of the electronic device (400) and the external device (500) that are communicating with each other are different, a guide UI may be provided based on the characteristics of the other device. For example, the device characteristics may include information such as the display size (or resolution), panel type, and hardware configuration.

[0185] According to one example, the electronic device (400) may receive information about a 3D protrusion distance of an external device (500) and provide a screen (1410) including a guide UI (1411), and the external device (500) may receive information about a 3D protrusion distance of the electronic device (400) and provide a screen (1420) including a guide UI (1421).

[0186] According to an example, as illustrated in FIG. 14, the electronic device (400) may be implemented as a PC monitor, and the external device (500) may be implemented as a mobile terminal. In this case, the resolution of the display of the electronic device (400) and the resolution of the display of the external device (500) may be different. For example, let us assume that the display of the electronic device (400) is implemented as 32 inches, and the display of the external device (500) is implemented as 7 inches. In this case, the 32-inch display may have a 3D protrusion distance of 15 cm, and the 7-inch display may have a 3D protrusion distance of 7 cm. Accordingly, the z-axis distance range of the guide UI (1411) provided by the electronic device (400) may be shorter than the z-axis distance range of the guide UI (1421) provided by the external device (500).

[0187] FIGS. 15A and 15B are drawings for explaining a method for setting a viewpoint of a visual object according to one or more embodiments.

[0188] According to one embodiment, the electronic device (400) can adjust and display a viewpoint for a visual object. For example, the electronic device (400) can adjust the viewpoint for a visual object based on at least one of a user command, the type of the visual object, and the type of the currently running application screen. The viewpoint for a visual object can indicate whether the first user of the electronic device (400) and the second user of the external device (500) view the visual object from the same or opposite viewpoints.

[0189] For example, as illustrated in FIG. 15A, the viewpoints of a visual object on a screen (1510) provided to a first user and the viewpoints of a visual object on a screen (1520) provided to a second user may be opposite viewpoints. For example, the front view (1511) of the visual object may be provided to the first user, and the back view (1521) of the visual object may be provided to the second user. For example, users may view the visual object from opposite viewpoints, providing the effect of viewing an object together in reality. In one example, the opposite viewpoint may be set as the default viewpoint, and the viewpoint settings may be selectively changed.

[0190] For example, as illustrated in FIG. 15b, the viewpoint of a visual object on a screen (1510) provided to a first user and the viewpoint of a visual object on a screen (1520) provided to a second user may be the same viewpoint. For example, the front (1511) of a visual object may be provided to both the first user and the second user.

[0191] FIGS. 16A to 16C are drawings for explaining a screen providing method according to an implementation example of a plurality of devices according to one or more embodiments.

[0192] According to one embodiment, depending on the implementation example of the plurality of devices in communication, each of the user images corresponding to the other user and the object overlaid on the user images may be provided in a two-dimensional or three-dimensional form.

[0193] As an example, according to FIG. 16A, it is assumed that both the displays provided in the electronic device (400) and the external device (500) are implemented as LFDs as shown on the upper side. For example, each other's user image may be provided as a 2D image on the LFDs provided in each of the electronic device (400) and the external device (500), and an object overlaid on the user image may be provided in 3D. For example, as shown on the lower side, an object (1612) may be placed on a virtual plane (1613), and a user image (1611) may be placed at a position that is recessed and spaced apart from the virtual plane (1613) by a preset distance, thereby forming a sense of space between the user image (611) and the object (1612). For example, a user image (1611) can be placed on a virtual plane (1613) and an object (1612) can be placed at a position protruding from the virtual plane (1613) by a preset distance to create a sense of space between the user image (611) and the object (1612).

[0194] As an example, according to FIG. 16b, it is assumed that the display provided in the electronic device (400) as illustrated in the upper part is implemented as an LFD, and the display provided in the external device (500) is implemented as a general display. For example, as illustrated in the lower part, in the LFD provided in the electronic device (400), a visual object (1622) may be provided by overlaying a 2D user image (1621). On the other hand, in the general display provided in the external device (400), the user image (1621) and the object (1622) may be provided in 2D.

[0195] As an example, according to FIG. 16c, it is assumed that both the displays provided in the electronic device (400) and the external device (500) are implemented as LFDs as illustrated on the upper side, but the external device (500) is equipped with a depth sensor. For example, as illustrated on the lower side, the external device (500) can sense the second user and the surrounding environment of the second user using the depth sensor, and generate a 3D user image (1631) based on the sensing data and transmit it to the electronic device (400). In this case, the electronic device (400) can provide the received 3D user image (1631) (second user image) and object (1633) in 3D based on the virtual plane (1633).

[0196] FIGS. 17A and 17B are drawings for explaining a method for providing additional visual information according to one or more embodiments.

[0197] According to one embodiment, various functions such as a visual display function or an authority control function may be provided based on the relative positional relationship between a virtual plane and a visual object based on the same virtual plane in the electronic device (400) and the external device (500).

[0198] For example, as illustrated in FIG. 17A, when a visual object (1720) is first displayed on an electronic device (400) according to a preset event, for example, a user command, the visual object (1720) may be displayed only on the electronic device (400) so that it is visible only to the first user. For example, the electronic device (400) may create a virtual area (1731) based on a maximum protrusion range with respect to a virtual plane and provide the visual object (1720). Thereafter, when the visual object (1720) moves to the opposite side of the virtual plane, i.e., a second virtual space (1732), according to an operation of the first user, the visual object (1720) may be displayed on an external device (500) so that it is visible to the second user as well. In this way, only the first user who created the visual object (1720) can see the visual object (1720) until the visual object (1720) passes through the virtual plane, and after passing through the virtual plane, not only the first user but also the second user can see the visual object (1720), thereby providing various 3D sharing experiences.

[0199] For example, as illustrated in FIG. 17b, the electronic device (400) and the external device (500) can be implemented so that the first user and the second user can have control authority over the visual object (1720) within virtual spaces (1731, 7132) where they can each feel a three-dimensional effect based on the virtual plane (1710). For example, when the visual object (1720) is located within the first virtual space (1731), only the first user can move the visual object (1720), and the second user cannot move the visual object (1720). In addition, when the visual object (1720) is located within the second virtual space (1732), only the second user can move the visual object (1720), and the first user cannot move the visual object (1720).

[0200] FIGS. 18A and 18B are drawings for explaining a method for providing additional visual information according to one or more embodiments.

[0201] According to one embodiment, when the electronic device (400) is implemented as an LFD and the external device (500) is implemented as an HMD (Head-mounted Display) and virtual reality (VR) content or augmented reality (AR) content is provided, a guide UI for a 3D protrusion range that allows a first user of the electronic device (400) to feel a three-dimensional effect can be provided. This allows for expanding a 3D sharing experience between users using different types of devices.

[0202] As an example, as shown on the left side of FIG. 18A, assume that a first user views 3D content through an LFD device (1811) and a second user views VR content by wearing an HMD (1812). In this case, as shown on the right side of FIG. 18A, a guide UI (1814) indicating a 3D protrusion range of the LFD device (1811), i.e., the first user, may be displayed in the virtual content provided to the second user. Accordingly, the second user can move a visual object (1814) based on the guide UI (1814), and the first user can experience effective virtual reality.

[0203] As an example, as shown on the left side of FIG. 18b, assume that a first user does not wear an HMD (1823) and a second user wears an HMD (1823) and views an LFD device (1821) displaying a visual object (1822) together. In this case, as shown on the right side of FIG. 18b, a guide UI (1824) indicating a 3D protrusion range of the LFD device (1821) from the viewpoint of the second user can be displayed as AR (Augmented Reality) content. Accordingly, the first user and the second user can share a 3D experience.

[0204] As described above, an electronic device according to one embodiment includes a first display (200) implemented as a Light Field Display (LFD), a memory (420) storing instructions, a camera (430), a communication circuit (440), and at least one processor (410) including processing circuitry. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display a visual object shared with an external device performing communication through the communication circuit on the first display, and, when the visual object is moved in the z-axis direction on a second display provided in the external device, receive first expected saliency distance information according to the z-axis movement of the visual object from the external device, identify second expected saliency distance information of the visual object based on the first expected saliency distance information and a viewing distance of the first user acquired through the camera, identify parallax of a left-eye image and a right-eye image including the visual object based on the second expected saliency distance information, and display a 3D image including the visual object on the first display based on the left-eye image and the right-eye image corresponding to the identified parallax.

[0205] According to one example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify second expected saliency distance information of the visual object based on positive parallax when the visual object is identified as being provided from the external device based on the first expected saliency distance based on negative parallax, and to identify second expected saliency distance information of the visual object based on negative parallax when the visual object is identified as being provided from the external device based on the first expected saliency distance based on positive parallax.

[0206] According to one example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to transmit a first user image captured by the first user through the camera to the external device, receive a second user image captured by the second user of the external device from the external device, and display a 3D image captured based on a layer stack in which a first image layer including the second user image and a second image layer including the visual object are stacked on the first display.

[0207] According to one example, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to identify a virtual space in which the second user can perceive three-dimensionality for the visual object based on characteristic information of the second display received from the external device and viewing distance information of the second user, and to display a guide UI indicating a Z-axis movement distance of the virtual space in which the second user can perceive three-dimensionality on the first display, wherein the characteristic information of the second display may include at least one of size information of the second display or three-dimensional maximum protrusion distance information of the second display.

[0208] According to one example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to adjust a relative size and a relative position between the second user image, the visual object, and the guide UI based on a viewing distance and a viewing angle of the first user acquired through the camera, and to display an image including the second user image, the visual object, and the guide UI with the adjusted relative size and relative position on the first display.

[0209] According to one example, the guide UI includes a plurality of grid areas arranged in the Z-axis direction, and the instructions, when individually or collectively executed by the at least one processor, can cause the electronic device to display the plurality of grid areas in a stepwise manner according to the Z-axis distance.

[0210] According to one example, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to, when it is determined that a crosstalk phenomenon has occurred in the second display based on the first expected protrusion distance information received from the external device, display guide information indicating that a crosstalk phenomenon has occurred in the second display on the first display, wherein the guide information may include an imitation image of the crosstalk phenomenon occurring in the second display of the external device.

[0211] According to one example, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to control movement of the visual object based on a user command of the first user within a first virtual space corresponding to a first direction relative to a first virtual plane of the first display, and to transmit control authority movement information for the visual object to the external device based on a user command of the first user within a second virtual space corresponding to a second direction opposite to the first direction relative to the first virtual plane without controlling control of the visual object, wherein the first virtual plane may correspond to a second virtual plane of the second display.

[0212] According to one example, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to display a UI on the first display that includes a menu for selecting a display surface of the visual object, the menu including a first menu for displaying the same surface of the visual object on the first display and the second display, and a second menu for displaying a different surface of the visual object.

[0213] According to an embodiment, a method for providing a three-dimensional image of an electronic device including a first display and a camera implemented as a Light Field Display (LFD) may include: displaying a visual object shared with an external device with which communication is being made on the first display; receiving first expected saliency distance information according to a z-axis movement of the visual object from an external device when the visual object is moved in the z-axis direction on a second display provided on the external device; identifying second expected saliency distance information of the visual object based on the first expected saliency distance information and a viewing distance of a first user acquired through the camera; identifying parallax of a left-eye image and a right-eye image including the visual object based on the second expected saliency distance information; and displaying a 3D image including the visual object on the first display based on the left-eye image and the right-eye image corresponding to the identified parallax.

[0214] According to one example, the operation of identifying the second expected saliency distance information of the visual object may include: if the visual object is identified as being provided from the external device based on negative parallax based on the first expected saliency distance, identifying the second expected saliency distance information of the visual object based on positive parallax; and if the visual object is identified as being provided from the external device based on positive parallax based on the first expected saliency distance, identifying the second expected saliency distance information of the visual object based on negative parallax.

[0215] According to one example, the method for providing a 3D image further includes an operation of transmitting a first user image captured by the first user through the camera to the external device, and an operation of receiving a second user image captured by the second user of the external device from the external device, and the operation of displaying an image including the visual object on the first display may include an operation of displaying a 3D image acquired based on a layer stack in which a first image layer including the second user image and a second image layer including the visual object are stacked on the first display.

[0216] According to one example, the method for providing a three-dimensional image further includes an operation of identifying a virtual space in which the second user can perceive a three-dimensional effect for the visual object based on characteristic information of the second display received from the external device and viewing distance information of the second user, and an operation of displaying a guide UI indicating a Z-axis movement distance of the virtual space in which the second user can perceive a three-dimensional effect on the first display, wherein the characteristic information of the second display may include at least one of size information of the second display or maximum three-dimensional effect protrusion distance information of the second display.

[0217] According to one example, the method for providing a three-dimensional image may further include an operation of adjusting a relative size and a relative position between the second user image, the visual object, and the guide UI based on a viewing distance and a viewing angle of the first user acquired through the camera, and an operation of displaying an image including the second user image, the visual object, and the guide UI, with the relative size and relative position adjusted, on the first display.

[0218] According to one example, the guide UI includes a plurality of grid areas listed in the Z-axis direction, and an operation of displaying the guide UI on the first display may include an operation of displaying the plurality of grid areas in a stepwise manner according to the Z-axis distance.

[0219] According to one example, the method for providing a three-dimensional image further includes an operation of displaying guide information indicating that crosstalk occurs in the second display on the first display when a crosstalk phenomenon is identified to occur in the second display based on the first expected protrusion distance information received from the external device, wherein the guide information may include an imitation image of the crosstalk phenomenon occurring in the second display of the external device.

[0220] According to one example, the method for providing a three-dimensional image further includes an operation of controlling movement of the visual object based on a user command of the first user in a first virtual space corresponding to a first direction with respect to a first virtual plane of the first display, and an operation of transmitting control authority movement information for the visual object to the external device based on a user command of the first user without controlling control of the visual object in a second virtual space corresponding to a second direction opposite to the first direction with respect to the first virtual plane, wherein the first virtual plane may correspond to a second virtual plane of the second display.

[0221] According to one example, the method for providing a three-dimensional image further includes an operation of displaying a UI including a menu for selecting a display surface of the visual object on the first display, wherein the menu may include a first menu for displaying the same surface of the visual object on the first display and the second display, and a second menu for displaying different surfaces of the visual object.

[0222] In one embodiment, a non-transitory computer-readable medium storing computer instructions that, when executed by a processor of an electronic device, cause the electronic device to perform an operation, the operation may include: displaying a visual object shared with an external device with which communication is being made on the first display; receiving, from the external device, first expected saliency distance information according to a z-axis movement of the visual object when the visual object is moved in the z-axis direction on a second display provided in the external device; identifying second expected saliency distance information of the visual object based on the first expected saliency distance information and a viewing distance of a first user acquired through the camera; identifying parallax of a left-eye image and a right-eye image including the visual object based on the second expected saliency distance information; and displaying a 3D image including the visual object on the first display based on the left-eye image and the right-eye image corresponding to the identified parallax.

[0223] According to the various embodiments described above, in a two-way communication situation using LFD, a rich sense of space can be created by providing each user with a different three-dimensional effect (e.g., a contrasting three-dimensional effect) for a shared visual object.

[0224] Electronic devices according to one or more embodiments disclosed herein may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to embodiments disclosed herein are not limited to the aforementioned devices.

[0225] It should be understood that one or more embodiments of the present disclosure and terminology used therein are not intended to limit the technical features described in the present disclosure to specific embodiments, but rather to encompass various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly dictates otherwise. In the present disclosure, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B, or C," "at least one of A, B, and C," and "at least one of A, B, or C" can each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0226] The term "module" as used in one or more embodiments of the present disclosure may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0227] One or more embodiments of the present disclosure may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0228] According to one embodiment, a method according to one or more embodiments disclosed in the present disclosure may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read only memory (CD-ROM)) or may be downloaded from an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0229] According to one or more embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and arranged in other components. According to one or more embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component of the plurality of components prior to the integration. According to one or more embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In electronic devices, A first display (200) of LFD (Light Field Display) type; Memory (420) for storing instructions; Camera (430); Communication circuit (440); At least one processor (410) comprising processing circuitry; The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Displaying a visual object shared with an external device that performs communication through the above communication circuit on the first display, When the visual object is moved in the z-axis direction on the second display provided in the external device, first expected protrusion distance information according to the z-axis movement of the visual object is received from the external device, Identifying second expected protrusion distance information of the visual object based on the first expected protrusion distance information and the viewing distance of the first user obtained through the camera, Identifying the parallax of the left-eye image and the right-eye image including the visual object based on the second expected protrusion distance information, An electronic device that displays a 3D image including the visual object on the first display based on the left-eye image and the right-eye image corresponding to the identified parallax.

2. In paragraph 1, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: If the visual object is identified as being provided from the external device based on the first expected saliency distance based on negative parallax, then the second expected saliency distance information of the visual object is identified based on positive parallax, An electronic device that identifies second expected protrusion distance information of the visual object based on negative parallax when the visual object is identified as being provided from the external device based on the first expected protrusion distance.

3. In paragraph 1, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Transmitting the first user image captured by the first user through the camera to the external device, Receive a second user image captured by the second user of the external device from the external device, An electronic device that displays a 3D image obtained based on a layer stack in which a first image layer including the second user image and a second image layer including the visual object are stacked on the first display.

4. In paragraph 1, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Identifying a virtual space in which the second user can perceive three-dimensionality of the visual object based on the characteristic information of the second display received from the external device and the viewing distance information of the second user, To display a guide UI indicating the Z-axis movement distance of the virtual space in which the three-dimensional perception of the second user is possible on the first display. The characteristic information of the above second display is: An electronic device comprising at least one of size information of the second display or maximum three-dimensional projection distance information of the second display.

5. In paragraph 4, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Adjusting the relative size and relative position between the second user image, the visual object, and the guide UI based on the viewing distance and viewing angle of the first user acquired through the camera; An electronic device that displays a 3D image including the second user image, the visual object, and the guide UI, the relative size and relative position of which are adjusted, on the first display.

6. In paragraph 4, The above guide UI is, Contains multiple grid areas listed in the Z-axis direction, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: An electronic device that displays the plurality of grid areas in a stepwise manner according to the Z-axis distance.

7. In paragraph 1, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: When it is determined that a crosstalk phenomenon occurs in the second display based on the first expected protrusion distance information received from the external device, guide information indicating that crosstalk occurs in the second display is displayed on the first display. An electronic device, wherein the guide information includes an imitation image of a crosstalk phenomenon occurring on the second display of the external device.

8. In paragraph 1, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Controlling the movement of the visual object based on a user command of the first user within a first virtual space corresponding to a first direction based on a first virtual plane of the first display, In a second virtual space corresponding to a second direction opposite to the first direction based on the first virtual plane, control authority for the visual object is not controlled based on a user command of the first user, and information on movement of control authority for the visual object is transmitted to the external device. The above first virtual plane is, An electronic device corresponding to a second virtual plane of the second display.

9. In paragraph 1, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: A UI including a menu for selecting a display surface of the visual object is displayed on the first display, The above menu is, An electronic device comprising a first menu for displaying the same side of the visual object on the first display and the second display, and a second menu for displaying the other side of the visual object.

10. A method for providing a three-dimensional image of an electronic device including a first display of the LFD (Light Field Display) type and a camera, An action of displaying a visual object shared with an external device with which communication is being made on said first display; An operation of receiving first expected protrusion distance information according to the z-axis movement of the visual object from the external device when the visual object is moved in the z-axis direction on the second display provided in the external device; An operation of identifying second expected protrusion distance information of the visual object based on the first expected protrusion distance information and the viewing distance of the first user acquired through the camera; An operation of identifying a parallax between a left-eye image and a right-eye image including the visual object based on the second expected protrusion distance information; and A method for providing a 3D image, comprising: an operation of displaying a 3D image including the visual object on the first display based on a left-eye image and a right-eye image corresponding to the identified parallax; 11. In paragraph 10, The operation of identifying the second expected saliency distance information of the above visual object is: An operation of identifying second expected saliency distance information of the visual object based on positive parallax when the visual object is identified as being provided from the external device based on the first expected saliency distance; and A method for providing a three-dimensional image, comprising: an operation of identifying second expected protrusion distance information of the visual object based on negative parallax when the visual object is identified as being provided from the external device based on positive parallax based on the first expected protrusion distance; 12. In paragraph 10, The above 3D image providing method is, An operation of transmitting a first user image captured by the first user through the camera to the external device; and Further comprising an action of receiving a second user image captured by the second user of the external device from the external device; The operation of displaying an image including the above visual object on the first display is: A method for providing a 3D image, comprising: an operation of displaying a 3D image acquired based on a layer stack in which a first image layer including the second user image and a second image layer including the visual object are stacked on the first display; 13. In paragraph 10, The above 3D image providing method is, An operation of identifying a virtual space in which the second user can perceive three-dimensionality of the visual object based on characteristic information of the second display received from the external device and viewing distance information of the second user; and It further includes an operation of displaying a guide UI indicating the Z-axis movement distance of a virtual space in which the three-dimensional perception of the second user is possible on the first display; The characteristic information of the above second display is: A method for providing a three-dimensional image, comprising at least one of size information of the second display or maximum three-dimensional projection distance information of the second display.

14. In paragraph 13, The above 3D image providing method is, An operation of adjusting the relative size and relative position between the second user image, the visual object, and the guide UI based on the viewing distance and viewing angle of the first user acquired through the camera; and A method for providing a three-dimensional image, further comprising: an operation of displaying an image including the second user image, the visual object, and the guide UI, the relative size and relative position of which are adjusted, on the first display.

15. A non-transitory computer-readable medium storing computer instructions that cause an electronic device to perform an operation when executed by a processor of an electronic device including a first display and a camera of the LFD (Light Field Display) type, The above action is, An action of displaying a visual object shared with an external device with which communication is being made on said first display; An operation of receiving first expected protrusion distance information according to the z-axis movement of the visual object from the external device when the visual object is moved in the z-axis direction on the second display provided in the external device; An operation of identifying second expected protrusion distance information of the visual object based on the first expected protrusion distance information and the viewing distance of the first user acquired through the camera; An operation of identifying a parallax between a left-eye image and a right-eye image including the visual object based on the second expected protrusion distance information; and A non-transitory computer-readable medium comprising: an operation of displaying a 3D image including the visual object on the first display based on a left-eye image and a right-eye image corresponding to the identified parallax;

Images