Electronic devices including multiple cameras and their control methods
By designing a rotatable housing and flexible display structure in the electronic device, and combining multiple cameras, the shortcomings of existing technologies in stereo image capture and depth information acquisition are addressed, thereby improving user experience and functional versatility.
Patent Information
- Application Number
- CN202580026524.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2025-03-28
- Filing Date
- 2025-03-31
- Publication Date
- 2026-11-10
AI Technical Summary
Existing electronic devices struggle to effectively utilize multiple cameras to capture stereo images and acquire depth information, resulting in a subpar user experience.
Design an electronic device comprising a rotatable first housing and a second housing, housing a flexible display and multiple cameras connected by a hinge structure, capable of capturing stereo images upon user input and generating left-eye and right-eye images based on multiple camera images, thereby obtaining depth information of the stereo image.
It enables the efficient capture of stereo images and depth information through multiple cameras, improving the user experience and enhancing the functionality and practical value of electronic devices.
Smart Images

Figure CN123077079A_ABST
Abstract
Description
Technical Field
[0001] Various embodiments of this disclosure relate to an electronic device including multiple cameras and a method for controlling the same. Background Technology
[0002] The variety of services and additional functions offered through electronic devices (such as portable devices like smartphones) is steadily increasing. To enhance the practical value of these devices and meet the diverse needs of users, communication service providers and electronic device manufacturers are competitively developing devices to offer a wide range of functions and differentiate themselves from other businesses. Consequently, the functions offered through electronic devices are becoming increasingly sophisticated.
[0003] Recently, technologies have been developed that allow electronic devices including multiple cameras to capture images in various ways. These devices can use multiple cameras to capture stereoscopic images. Stereoscopic images are obtained by capturing a pair of 2D images, which can utilize the visual differences between the two eyes to perceive three-dimensional depth. Based on the principle that stereoscopic images are essentially represented by a pair of images (left-eye image and right-eye image), the image projected onto each eye differs due to parallax based on the distance between the two eyes (e.g., binocular parallax), and this difference is perceived to produce a sense of depth.
[0004] The information described above may be provided as related technology for the purpose of aiding understanding of this disclosure. No statement or determination has been made as to whether any description in the foregoing can be used as prior art in relation to this disclosure. Summary of the Invention
[0005] Technical solution According to an embodiment, an electronic device may include: a first housing including a first surface and a second surface opposite to the first surface; a second housing including a third surface and a fourth surface opposite to the third surface; a hinge structure rotatably connecting the first housing and the second housing; a flexible display disposed on the first surface and the third surface; a first rear camera and a second rear camera disposed on the second surface; a front camera disposed on the fourth surface; at least one processor including processing circuitry; and a memory configured to store instructions. According to an embodiment, the instructions, when executed individually or jointly by the at least one processor, may cause the electronic device to perform the following operations: based on received user input for capturing a stereoscopic image, acquire a first image via the first rear camera, acquire a second image via the second rear camera, and acquire a third image via the front camera. According to an embodiment, the instructions, when executed individually or jointly by the at least one processor, may cause the electronic device to perform the following operations: based on the first image and the second image, acquire a left-eye image and a right-eye image of the stereoscopic image. According to an embodiment, the instructions, when executed individually or jointly by the at least one processor, may cause the electronic device to perform the following operations: based on the first image and the third image, acquire first depth information of the stereoscopic image.
[0006] According to an embodiment, an operation method for an electronic device is provided, and the electronic device may include: a first housing including a first surface and a second surface opposite to the first surface; a second housing including a third surface and a fourth surface opposite to the third surface; a hinge structure rotatably connecting the first housing and the second housing; a flexible display disposed on the first surface and the third surface; a first rear camera and a second rear camera disposed on the second surface; and a front camera disposed on the fourth surface. According to an embodiment, the operation method of the electronic device may include: obtaining a first image via the first rear camera, obtaining a second image via the second rear camera, and obtaining a third image via the front camera based on received user input for capturing a stereoscopic image. According to an embodiment, the operation method of the electronic device may include: obtaining a left-eye image and a right-eye image of the stereoscopic image based on the first image and the second image. According to an embodiment, the operation method of the electronic device may include: obtaining first depth information of the stereoscopic image based on the first image and the third image.
[0007] According to an embodiment, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium is configured to store instructions, and the instructions, when executed individually or jointly by at least one processor, can cause an electronic device to operate, wherein the electronic device includes: a first housing including a first surface and a second surface opposite to the first surface; a second housing including a third surface and a fourth surface opposite to the third surface; a hinge structure rotatably connecting the first housing and the second housing; a flexible display disposed on the first surface and the third surface; a first rear camera and a second rear camera disposed on the second surface; and a front camera disposed on the fourth surface, wherein the operation includes: obtaining a first image through the first rear camera, obtaining a second image through the second rear camera, and obtaining a third image through the front camera based on received user input for capturing a stereoscopic image; obtaining a left-eye image and a right-eye image of the stereoscopic image based on the first image and the second image; and obtaining first depth information of the stereoscopic image based on the first image and the third image. Attached Figure Description
[0008] Figure 1 This is a block diagram of an electronic device in a network environment according to an embodiment.
[0009] Figure 2a This is a diagram showing the flat state of an electronic device according to an embodiment of the present disclosure.
[0010] Figure 2b This is a diagram showing the folded state of an electronic device according to an embodiment of the present disclosure.
[0011] Figure 3 This is a diagram illustrating the state in which multiple cameras are arranged in an electronic device according to embodiments of the present disclosure.
[0012] Figure 4 This is a block diagram of an electronic device including multiple cameras according to embodiments of the present disclosure.
[0013] Figure 5 This is a diagram illustrating the concept of a depth information generation method according to an embodiment of the present disclosure.
[0014] Figure 6 This is an illustration of a concept for a depth information estimation method using stereo images according to an embodiment of the present disclosure.
[0015] Figure 7 This is a diagram illustrating a method for generating depth information according to an embodiment of the present disclosure.
[0016] Figure 8 This is an illustration of an example of a data format for stereoscopic video including depth information according to an embodiment of the present disclosure.
[0017] Figure 9a and Figure 9b This is a diagram illustrating a method for displaying stereoscopic video using depth information in an electronic device or wearable electronic device according to an embodiment of the present disclosure.
[0018] Figure 10a and Figure 10b This is a diagram illustrating a method for displaying stereoscopic video using depth information in an electronic device or wearable electronic device according to an embodiment of the present disclosure.
[0019] Figure 11 This is a diagram illustrating an example of depth information estimation based on baseline length difference according to an embodiment of the present disclosure.
[0020] Figure 12 This is a diagram illustrating a method for displaying stereoscopic video using depth information in a wearable electronic device according to an embodiment of the present disclosure.
[0021] Figure 13 This is a diagram illustrating a method for displaying stereoscopic video using depth information in a wearable electronic device according to an embodiment of the present disclosure.
[0022] Figure 14 This is a flowchart illustrating a method for an electronic device to obtain a stereoscopic image according to an embodiment of the present disclosure. Detailed Implementation
[0023] Figure 1 This is a block diagram illustrating an electronic device in a network environment according to an embodiment.
[0024] Reference Figure 1In network environment 100, electronic device 101 can communicate with electronic device 102 via a first network 198 (e.g., a short-range wireless communication network), or with at least one of electronic device 104 or server 108 via a second network 199 (e.g., a long-range wireless communication network). According to an embodiment, electronic device 101 can communicate with electronic device 104 via server 108. According to an embodiment, electronic device 101 may include a processor 120, memory 130, input module 150, sound output module 155, display module 160, audio module 170, sensor module 176, interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, user identification module (SIM) 196, or antenna module 197. In some embodiments, at least one component (e.g., connection terminal 178) may be omitted from electronic device 101, or one or more other components may be added to electronic device 101. In some embodiments, some components of the assembly (e.g., sensor module 176, camera module 180, or antenna module 197) may be implemented as a single component (e.g., display module 160).
[0025] Processor 120 may run software (e.g., program 140) to control at least one other component (e.g., hardware or software component) of electronic device 101 in conjunction with processor 120, and may perform various data processing or calculations. According to one embodiment, as at least part of the data processing or calculation, processor 120 may store commands or data received from another component (e.g., sensor module 176 or communication module 190) in volatile memory 132, process the commands or data stored in volatile memory 132, and store the resulting data in non-volatile memory 134. According to embodiments, processor 120 may include a main processor 121 (e.g., central processing unit (CPU) or application processor (AP)) or an auxiliary processor 123 (e.g., graphics processing unit (GPU), neural processing unit (NPU), image signal processor (ISP), sensor central processor, or communication processor (CP)) that is operationally independent of or combined with the main processor 121. For example, when electronic device 101 includes a main processor 121 and an auxiliary processor 123, the auxiliary processor 123 may be adapted to consume less power than the main processor 121, or to be dedicated to a specific function. The auxiliary processor 123 may be implemented separately from the main processor 121, or may be implemented as part of the main processor 121.
[0026] When the main processor 121 is inactive (e.g., in sleep mode), the auxiliary processor 123 (rather than the main processor 121) can control at least some of the functions or states associated with at least one component of the electronic device 101 (e.g., display module 160, sensor module 176, or communication module 190), or when the main processor 121 is active (e.g., running an application), the auxiliary processor 123 can work with the main processor 121 to control at least some of the functions or states associated with at least one component of the electronic device 101 (e.g., display module 160, sensor module 176, or communication module 190). According to embodiments, the auxiliary processor 123 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., camera module 180 or communication module 190) functionally associated with the auxiliary processor 123. According to embodiments, the auxiliary processor 123 (e.g., a neural processing unit) may include hardware architecture dedicated to artificial intelligence model processing. Artificial intelligence models can be generated through machine learning. For example, such learning can be performed via electronic device 101 where artificial intelligence is performed or via a separate server (e.g., server 108). The learning algorithm may include, but is not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. The artificial intelligence model may include multiple layers of artificial neural networks. The artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more thereof, but is not limited thereto. Additionally or optionally, the artificial intelligence model may include software structures in addition to hardware structures.
[0027] Memory 130 may store various data used by at least one component of electronic device 101 (e.g., processor 120 or sensor module 176). The various data may include, for example, software (e.g., program 140) and input or output data for commands associated with it. Memory 130 may include volatile memory 132 or non-volatile memory 134.
[0028] The program 140 may be stored as software in the memory 130, and the program 140 may include, for example, an operating system (OS) 142, middleware 144, or application 146.
[0029] Input module 150 can receive commands or data from outside electronic device 101 (e.g., a user) that will be used by another component of electronic device 101 (e.g., processor 120). Input module 150 may include, for example, a microphone, mouse, keyboard, keys (e.g., buttons), or digital pen (e.g., stylus).
[0030] The sound output module 155 can output sound signals to the outside of the electronic device 101. The sound output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as playing multimedia or playing records. The receiver can be used to receive incoming calls. According to an embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.
[0031] Display module 160 can visually provide information to the outside of electronic device 101 (e.g., to a user). Display module 160 may include, for example, a display, a holographic device, or a projector, and control circuitry for controlling a respective one of the display, holographic device, and projector. According to an embodiment, display module 160 may include a touch sensor adapted to detect touch or a pressure sensor adapted to measure the intensity of the force caused by touch.
[0032] The audio module 170 can convert sound into electrical signals and vice versa. According to an embodiment, the audio module 170 can obtain sound via the input module 150, or output sound via the sound output module 155 or via headphones of an external electronic device (e.g., electronic device 102) that is directly (e.g., wired) or wirelessly connected to the electronic device 101.
[0033] Sensor module 176 can detect the operating state of electronic device 101 (e.g., power or temperature) or the environmental state outside electronic device 101 (e.g., user state), and then generate an electrical signal or data value corresponding to the detected state. According to embodiments, sensor module 176 may include, for example, a gesture sensor, gyroscope sensor, atmospheric pressure sensor, magnetic sensor, accelerometer, grip sensor, proximity sensor, color sensor, infrared (IR) sensor, biometric sensor, temperature sensor, humidity sensor, or illuminance sensor.
[0034] Interface 177 may support one or more specific protocols used to enable electronic device 101 to be directly (e.g., wired) or wirelessly coupled to external electronic device (e.g., electronic device 102). According to embodiments, interface 177 may include, for example, a High Definition Multimedia Interface (HDMI), a Universal Serial Bus (USB) interface, a Secure Digital Card (SD) interface, or an audio interface.
[0035] Connection 178 may include a connector, through which electronic device 101 may be physically connected to an external electronic device (e.g., electronic device 102). According to embodiments, connection 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).
[0036] The haptic module 179 can convert electrical signals into mechanical stimuli (e.g., vibration or motion) or electrical stimuli that can be recognized by a user through his touch or kinesthesia. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.
[0037] Camera module 180 can capture still or moving images. According to an embodiment, camera module 180 may include one or more lenses, an image sensor, an image signal processor, or a flash.
[0038] The power management module 188 manages the power supply to the electronic device 101. According to one embodiment, the power management module 188 may be implemented as at least part of, for example, a power management integrated circuit (PMIC).
[0039] Battery 189 can power at least one component of electronic device 101. According to an embodiment, battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable rechargeable battery, or a fuel cell.
[0040] Communication module 190 can support the establishment of a direct (e.g., wired) or wireless communication channel between electronic device 101 and external electronic devices (e.g., electronic device 102, electronic device 104, or server 108), and perform communication via the established communication channel. Communication module 190 may include one or more communication processors capable of operating independently of processor 120 (e.g., application processor (AP)) and supporting direct (e.g., wired) or wireless communication. According to embodiments, communication module 190 may include wireless communication module 192 (e.g., cellular communication module, short-range wireless communication module, or Global Navigation Satellite System (GNSS) communication module) or wired communication module 194 (e.g., local area network (LAN) communication module or power line communication (PLC) module). A corresponding one of these communication modules can communicate via a first network 198 (e.g., a short-range communication network, such as Bluetooth). TM The wireless communication module 192 can communicate with external electronic devices via a second network 199 (e.g., a long-range communication network, such as a traditional cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))). These various types of communication modules can be implemented as a single component (e.g., a single chip) or as multiple components separate from each other (e.g., multiple chips). The wireless communication module 192 can identify and verify the electronic device 101 in the communication network (such as a first network 198 or a second network 199) using user information (e.g., an International Mobile Subscriber Identity (IMSI)) stored in the user identification module 196.
[0041] Wireless communication module 192 can support 5G networks following 4G networks and next-generation communication technologies (such as new radio (NR) access technologies). NR access technologies can support enhanced mobile broadband (eMBB), massive machine-type communication (mMTC), or ultra-reliable low-latency communication (URLLC). Wireless communication module 192 can support high-frequency bands (e.g., millimeter-wave bands) to achieve, for example, high data transmission rates. Wireless communication module 192 can support various technologies used to ensure performance in high-frequency bands, such as, for example, beamforming, massive MIMO, full-dimensional MIMO (FD-MIMO), array antennas, analog beamforming, or massive antennas. Wireless communication module 192 can support various requirements specified in electronic device 101, external electronic devices (e.g., electronic device 104), or network systems (e.g., second network 199). According to an embodiment, the wireless communication module 192 may support peak data rates (e.g., 20 Gbps or greater) for implementing eMBB, lost coverage (e.g., 164 dB or less) for implementing mMTC, or U-plane latency (e.g., 0.5 ms or less for each of the downlink (DL) and uplink (UL), or 1 ms or less round trip) for implementing URLLC.
[0042] Antenna module 197 can transmit or receive signals or power to or from the exterior of electronic device 101 (e.g., external electronic device). According to an embodiment, antenna module 197 may include an antenna comprising a radiating element formed of a conductive material or conductive pattern formed in or on a substrate (e.g., a printed circuit board (PCB)). According to an embodiment, antenna module 197 may include multiple antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication scheme used in a communication network (such as a first network 198 or a second network 199) can be selected from the multiple antennas by, for example, communication module 190 (e.g., wireless communication module 192). Signals or power can then be transmitted or received between communication module 190 and the external electronic device via the selected at least one antenna. According to an embodiment, additional components besides the radiating element (e.g., a radio frequency integrated circuit (RFIC)) may be additionally incorporated into antenna module 197.
[0043] According to various embodiments, antenna module 197 can form a millimeter-wave antenna module. According to an embodiment, the millimeter-wave antenna module may include a printed circuit board, an RFIC, and multiple antennas (e.g., an array antenna), wherein the RFIC is disposed on or adjacent to a first surface (e.g., a bottom surface) of the printed circuit board and is capable of supporting a specified high-frequency band (e.g., a millimeter-wave band), and the multiple antennas are disposed on or adjacent to a second surface (e.g., a top surface or a side surface) of the printed circuit board and are capable of transmitting or receiving signals of the specified high-frequency band.
[0044] At least some of the aforementioned components can be combined with each other and communicate signals (e.g., commands or data) between them via an inter-peripheral communication scheme (e.g., bus, general purpose input / output (GPIO), serial peripheral interface (SPI), or mobile industrial processor interface (MIPI)).
[0045] According to an embodiment, commands or data can be sent or received between electronic device 101 and external electronic device 104 via server 108 connected to a second network 199. Each of electronic device 102 or electronic device 104 can be a device of the same type as electronic device 101, or a device of a different type. According to an embodiment, all or some operations that would be performed on electronic device 101 can be performed on one or more of external electronic devices 102, 104, or 108. For example, if electronic device 101 is required to automatically perform a function or service, or is required to perform a function or service in response to a request from a user or another device, electronic device 101 may request one or more external electronic devices to perform at least a portion of the function or service, instead of running the function or service, or electronic device 101 may request the one or more external electronic devices to perform at least a portion of the function or service in addition to running the function or service. Upon receiving the request, the one or more external electronic devices may perform at least a portion of the requested function or service, or perform additional functions or services related to the request, and transmit the result of the execution to electronic device 101. Electronic device 101 may provide the result as at least a partial response to the request, with or without further processing of the result. For this purpose, technologies such as cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing may be used. Electronic device 101 may use, for example, distributed computing or mobile edge computing to provide ultra-low latency services. In another embodiment, external electronic device 104 may include an Internet of Things (IoT) device. Server 108 may be an intelligent server using machine learning and / or neural networks. According to an embodiment, external electronic device 104 or server 108 may be included in a second network 199. Electronic device 101 may be applied to intelligent services based on 5G communication technology or IoT-related technologies (e.g., smart homes, smart cities, smart cars, or healthcare).
[0046] The AI-related functions according to this disclosure operate via a processor and memory. The processor may include one or more processors. These processors may include the functions of a general-purpose processor (such as a central processing unit (CPU), application processor (AP), or digital signal processor (DSP)), a dedicated graphics processor (such as a graphics processing unit (GPU) or vision processing unit (VPU)), or a dedicated AI processor (such as a neural network processing unit (NPU)). The processors control the processing of input data according to predefined operating rules or AI models stored in memory. Optionally, if one or more processors are dedicated AI processors, the dedicated AI processors may be designed with a hardware architecture tailored for processing a specific AI model.
[0047] The characteristics of a predefined operating rule or AI model can be constructed through learning. As used herein, "made through learning" means that the base AI model learns from multiple pieces of learning data through a learning algorithm, thereby creating a redefined operating rule or AI model configured to perform desired features (or purposes). This learning can be implemented by the device for performing AI according to this disclosure itself, or by a separate server and / or system. The learning algorithm may include, but is not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.
[0048] An AI model can be configured with multiple neural network layers. Each of the multiple neural network layers has multiple weight values, and neural network computation is performed via computation between these weight values and the computation results of the previous layer. The weight values of the multiple neural network layers can be optimized based on the learning results of the AI model. For example, multiple weight values can be updated to reduce or minimize the loss or cost values acquired by the AI model during learning processing. Artificial neural networks can include, but are not limited to, deep neural networks (DNNs), such as convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), or deep Q-networks.
[0049] Figure 2a This is a diagram showing the flat state of an electronic device according to an embodiment of the present disclosure. Figure 2b This is a diagram showing the folded state of an electronic device according to an embodiment of the present disclosure.
[0050] Reference Figure 2a and Figure 2b Electronic device 101 (e.g., Figure 1The electronic device 101 may include a housing 201, a hinged cover 240 covering a foldable portion of the housing 201, and a display 230 disposed within the space formed by the housing 201 (e.g., Figure 1 The display module 160 in the middle.
[0051] According to an embodiment, the surface exposed on the screen output from the display 230 can be defined as the front surface of the electronic device 101 (e.g., a first front surface 210a and a second front surface 220a). The opposing surfaces of the front surfaces can be defined as the rear surfaces of the electronic device 101 (e.g., a first rear surface 210b and a second rear surface 220b). In an embodiment, the surface surrounding the space between the front and rear surfaces can be defined as the side surfaces of the electronic device 101 (e.g., a first side surface 210c and a second side surface 220c). The side surfaces of the electronic device 101 can be at least one side surface of the first housing 210 or the second housing 220. Figure 2a and Figure 2b The electronic device 101 may be referred to as a foldable electronic device, a portable electronic device, or a portable foldable electronic device. According to an embodiment, the housing 201 may be referred to as a foldable housing. The display 230 may be referred to as a "flexible display".
[0052] According to an embodiment, the housing 201 may form at least a portion of the exterior of the electronic device 101. The housing 201 may include a first housing 210, a second housing 220 rotatable relative to the first housing 210, a first rear cover 280, and a second rear cover 290. The housing 201 of the electronic device 101 is not limited to... Figure 2a and Figure 2b The forms and combinations shown are possible and can be achieved through combinations and / or combinations of other shapes or components. For example, in an embodiment, the first housing 210 and the first rear cover 280 may be integrally formed, and the second housing 220 and the second rear cover 290 may be integrally formed. According to an embodiment, the first rear cover 280 may be defined and / or referred to as the first rear plate 280. The second rear cover 290 may be defined and / or referred to as the second rear plate 290.
[0053] According to an embodiment, the first housing 210 may be connected to a hinge structure (e.g., Figure 4The electronic device 101 is connected to the hinge structure 202 and may include a first front surface 210a facing a first direction and a first rear surface 210b facing a second direction opposite to the first direction. A second housing 220 may be connected to the hinge structure, may include a second front surface 220a facing a third direction and a second rear surface 220b facing a fourth direction opposite to the third direction, and is rotatable about the hinge structure 202 relative to the first housing 210. Therefore, the electronic device 101 can change between a folded state and a flat state. Folding or unfolding the electronic device 101 can be understood as rotation of the first housing 210 relative to the hinge structure or rotation of the second housing 220 relative to the hinge structure. In the folded state of the electronic device 101, the first front surface 210a may face the second front surface 220a. In the flat state of the electronic device 101, the third direction may be the same as the first direction. Hereinafter, unless otherwise stated, directions will be described based on the flat state of the electronic device 101.
[0054] According to an embodiment, the first housing 210 and the second housing 220 may be located on opposite sides of the folding axis Ax and may have a shape that is symmetrical about the entire folding axis Ax. As described below, the angle or distance between the first housing 210 and the second housing 220 may vary depending on whether the electronic device 101 is in a flat state, a folded state, or an intermediate state. According to an embodiment, the second housing 220 may also include a sensor region 224 where a sensor (e.g., a camera) is disposed, but may have a symmetrical shape in other regions.
[0055] According to an embodiment, the folding axis Ax can be a plurality of (e.g., two) parallel folding axes. In this disclosure, the folding axis Ax is arranged along the longitudinal direction (Y-axis direction) of the electronic device 101, but the direction of the folding axis Ax is not limited thereto. For example (not shown), embodiments may be implemented in which the electronic device 101 includes folding axes extending along the width direction (e.g., X-axis direction).
[0056] According to an embodiment, the electronic device 101 may include a structure for attaching a digital pen (not shown). For example, the electronic device 101 may include a magnet configured to attach the digital pen to a side surface of a first housing 210 or a side surface of a second housing 220. According to an embodiment, the electronic device 101 may include a structure for inserting a digital pen. For example, a hole (not shown) for inserting a digital pen may be formed on a side surface of the first housing 210 or the side surface of the second housing 220 of the electronic device 101.
[0057] According to an embodiment, at least a portion of the first housing 210 and the second housing 220 may be formed of a metallic or non-metallic material having a selected level of stiffness to support the display 230. At least a portion formed of a metallic material may provide a ground plane for the electronic device 101 and may be electrically connected to a printed circuit board (e.g., Figure 4 The grounding wire on the board portion 260).
[0058] According to an embodiment, sensor region 224 may be formed having a predetermined area adjacent to a corner or edge of the second housing 220. However, the arrangement, shape, and size of sensor region 224 are not limited to the illustrated example. According to an embodiment, sensor region 224 may be disposed on another corner of the second housing 220, in any area between the top and bottom corners, or on the first housing 210. In an embodiment, components embedded in the electronic device 101 for performing various functions may be exposed on the front surface of the electronic device 101 through sensor region 224 or through one or more openings provided in sensor region 224. In an embodiment, components may include various types of sensors. Sensors may include at least one of, for example, a front-facing camera, a receiver, or a proximity sensor.
[0059] According to an embodiment, a first rear cover 280 may be disposed on one side of the folding axis Ax on the rear surface of the electronic device 101, and may have a periphery that is, for example, substantially rectangular, and this periphery may be surrounded by another structure of the first housing 210. Similarly, a second rear cover 290 may be disposed on the other side of the folding axis Ax on the rear surface of the electronic device 101, and its periphery may be surrounded by another structure of the second housing 220.
[0060] According to an embodiment, the first rear cover 280 and / or the second rear cover 290 may have a shape that is substantially symmetrical with respect to the folding axis Ax. However, the first rear cover 280 and the second rear cover 290 do not necessarily have symmetrical shapes with each other, and in an embodiment, the electronic device 101 may include a first rear cover 280 and a second rear cover 290 with asymmetrical shapes.
[0061] According to embodiments, the first rear cover 280, the second rear cover 290, the first housing 210, and the second housing 220 provide space for various components of the electronic device 101 (e.g., printed circuit boards or batteries). According to embodiments, one or more components may be disposed on or visually exposed on the rear surface of the electronic device 101. For example, at least a portion of the sub-display 234 may be visually exposed through at least a portion of the first rear cover 280. According to embodiments, one or more components or sensors may be visually exposed through at least a portion of the second rear cover 290. In various embodiments, the sensor may include a proximity sensor and / or a camera module 206 (e.g., a rear camera).
[0062] According to embodiments, a front-facing camera exposed to the front surface of the electronic device 101 through one or more openings in the sensor region 224, or a camera module 206 (e.g., a rear-facing camera) exposed through at least a portion of the second rear cover 290, may include one or more lenses, an image sensor, and / or an image signal processor. In some embodiments, two or more lenses (infrared cameras, wide-angle and telephoto lenses) and an image sensor may be disposed on one surface of the electronic device 101.
[0063] According to an embodiment, the camera module 206, exposed through at least a portion of the second rear cover 290, may face the opposite direction of the second display area 232 of the display 230.
[0064] According to an embodiment, the hinge cover 240 may be disposed between the first housing 210 and the second housing 220 to cover internal components (e.g., Figure 4 (Hinge structure 202 in the middle). According to an embodiment, depending on the state of the electronic device 101 (flat state (expanded state) or folded state), the hinge cover 240 may be covered by a portion of the first housing 210 and the second housing 220 or exposed to the outside.
[0065] According to an embodiment, as shown in FIG2, when the electronic device 101 is in a flat state, the hinge cover 240 can be covered by the first housing 210 and the second housing 220 without being exposed. According to an embodiment, as... Figure 2b As shown, when the electronic device 101 is in a folded state (e.g., a fully folded state), the hinge cover 240 may be exposed to the outside between the first housing 210 and the second housing 220. According to an embodiment, when the first housing 210 and the second housing 220 are in an intermediate state where they are folded at an angle, the hinge cover 240 may be partially exposed to the outside between the first housing 210 and the second housing 220. However, in this case, the exposed area may be smaller than the exposed area in the fully folded state. In an embodiment, the hinge cover 240 may include a curved surface.
[0066] According to an embodiment, the display 230 may be disposed in a space formed (or defined) by the housing 201. For example, the display 230 may be disposed in a recess provided by the housing 201 and may form a large portion of the front surface of the electronic device 101. Therefore, the front surface of the electronic device 101 may include the display 230, and a portion of the first housing 210 and a portion of the second housing 220 adjacent to the display 230. The rear surface of the electronic device 101 may include a first rear cover 280, a portion of the first housing 210 adjacent to the first rear cover 280, a second rear cover 290, and a portion of the second housing 220 adjacent to the second rear cover 290.
[0067] According to an embodiment, the display 230 may include a plurality of display areas spaced apart from each other. For example, the display 230 may include a first display area 231 disposed on a first housing 210, a second display area 232 disposed on a second housing 220, and a folding area 233. According to an embodiment, the first display area 231 and the second display area 232 may rotate about a folding axis Ax.
[0068] According to an embodiment, display 230 may indicate a display whose at least partial area is deformable into a flat or curved surface. For example, display 230 may be a foldable or flexible display. According to an embodiment, display 230 may include a folded region 233, a first display area 231 disposed on one side relative to the folded region 233 (e.g., the left side of the folded region 233 shown in FIG. 2), and a second display area 232 disposed on the other side (e.g., the right side of the folded region 233 shown in FIG. 2). However, this division of display 230 into regions is exemplary, and display 230 may be divided into multiple (e.g., four or more, or two) regions depending on its structure or function. For example, in the embodiment shown in FIG. 2, the regions of display 230 may be divided by a folded region 233 extending parallel to the Y-axis or a folding axis Ax. According to an embodiment, display 230 may be divided into multiple regions based on another folded region (e.g., a folded region parallel to the X-axis) or another folding axis (e.g., a folding axis parallel to the X-axis). According to an embodiment, the display 230 may be combined with or adjacent to a touch detection circuit, a pressure sensor capable of measuring the intensity (pressure) of a touch, and / or a digital converter (not shown) configured to detect a magnetic field type stylus.
[0069] According to an embodiment, the first display area 231 and the second display area 232 may have a shape that is generally symmetrical with respect to the folded area 233. According to an embodiment (not shown), unlike the first display area 231, the second display area 232 may include a notch cut due to the presence of the sensor area 224, but the remaining area may have a shape that is substantially symmetrical with the first display area 231. For example, the first display area 231 and the second display area 232 may include portions with shapes that are symmetrical to each other and portions with shapes that are asymmetrical to each other.
[0070] In the following description, the operation of the first housing 210 and the second housing 220 and each area of the display 230 will be described according to the state of the electronic device 101 (e.g., flat state (or unfolded state) and folded state).
[0071] According to an embodiment, when the electronic device 101 is in a flat state (e.g., FIG. 2), the first housing 210 and the second housing 220 may be configured to face the same direction and form an angle of substantially 180 degrees. The surfaces of the first display area 231 and the second display area 232 of the display 230 may form an angle of 180 degrees with each other and face the same direction (e.g., the front surface direction of the electronic device). The folding region 233 may form a coplanar plane with the first display area 231 and the second display area 232.
[0072] According to an embodiment, when the electronic device 101 is in a folded state (e.g., Figure 2b When the electronic device 101 is in a folded state, the first housing 210 and the second housing 220 may be configured to face each other. The surfaces of the first display area 231 and the second display area 232 of the display 230 may face each other, thereby forming a narrow angle (e.g., between 0 and 10 degrees). When the electronic device 101 is in a folded state, at least a portion of the folded area 233 may be formed by a curved surface having a predetermined curvature.
[0073] According to an embodiment, when the electronic device 101 is in an intermediate state (not shown), the first housing 210 and the second housing 220 may be arranged at an angle to each other. The surfaces of the first display area 231 and the second display area 232 of the display 230 may form an angle larger than that in the folded state and smaller than that in the flat state. At least a portion of the folded area 233 may be formed by a curved surface having a predetermined curvature, and the curvature may be smaller than that in the folded state.
[0074] Figure 3 This is a diagram illustrating the state in which multiple cameras are arranged in an electronic device according to embodiments of the present disclosure.
[0075] Reference Figure 3 Electronic devices according to embodiments (e.g., Figure 1 The electronic device 101 in the middle may include multiple cameras (e.g., Figure 1 (Camera module 190 in the middle). This disclosure is not limited thereto; the electronic device 101 according to the embodiment may also include a camera module 190. Figure 1 Other components described in the document.
[0076] like Figure 3 As shown, when housings 210 and 220 are in a flat state, the electronic device according to the embodiment can use multiple rear cameras (e.g., first rear camera 310 and second rear camera 320) included in camera module 206 to capture stereoscopic images. For example, the electronic device can capture stereoscopic images using multiple rear cameras (e.g., first rear camera 310 and second rear camera 320) included in camera module 206 while being positioned in a horizontal direction (e.g., the electronic device is rotated 90 degrees).
[0077] According to an embodiment, a "stereoscopic image" refers to an image obtained by two or more cameras and can be used to represent a pair of 2D images capable of perceiving three-dimensional depth by utilizing the visual differences between the two eyes. For example, a stereoscopic image may include a left-eye image corresponding to the left eye and a right-eye image corresponding to the right eye. According to an embodiment, a stereoscopic image can be reproduced as a 3D image to the user by an electronic device developed in a form that can be worn on a user (e.g., wearable electronic devices such as augmented reality (AR) glasses, video see-through (VST) devices, or head-mounted display (HMD) devices). Stereoscopic images may also be referred to by the term "spatial video," but are not limited to that specific term.
[0078] Additionally, the second housing 210, as described below, can be coupled with... Figure 2a and Figure 2b The first housing 210 is the same as that in the previous one. Additionally, the first housing 220 may be... Figure 2a and Figure 2b The second housing 220 is the same.
[0079] According to an embodiment, the camera may include multiple cameras, and among the multiple cameras, a front-facing camera 330 may be disposed on a fourth surface of the second housing 210 (e.g., the front surface when the electronic device 101 is in a folded state), and among the multiple cameras, multiple rear-facing cameras 310 and 320 may be disposed on a second surface of the first housing 220 (e.g., the rear surface when the electronic device 101 is in a folded state). For example, in the horizontal direction of the electronic device (from... Figure 2a While the direction shown is rotated 90 degrees vertically, at least two of a plurality of rear cameras 392 arranged in a line along a first direction (e.g., the Y direction) can be used to capture stereoscopic images. According to an embodiment, the plurality of rear cameras 310 and 320 may include a first rear camera 310 and a second rear camera 320 of the same or different types for capturing stereoscopic images. The first rear camera 310 and the second rear camera 320 of the same or different types may be rear cameras disposed on the rear surface of the electronic device. For example, when the first rear camera 310 and the second rear camera 320 are cameras of different types, the first rear camera 310 may be a wide-angle camera, and the second rear camera 320 may be an ultra-wide-angle camera. The first rear camera 310 configured as a wide-angle camera may be referred to as the main camera, but is not limited to that term.
[0080] According to an embodiment, when the electronic device is a foldable device, such as Figure 2a and Figure 3As shown, when the electronic device is fully extended, the front camera 330 and the rear cameras 310 and / or 320 can face the same direction. Therefore, depth information can be obtained using the front camera 330 disposed on the front surface and any one of the plurality of rear cameras 310 and 320 disposed on the rear surface (e.g., the first rear camera 310 or the second rear camera 320).
[0081] According to embodiments, for ease of explanation in the embodiments described below, among the plurality of cameras used to capture stereoscopic images, a first rear camera 310 may be configured as a camera corresponding to the left eye, and a second rear camera 320 may be configured as a camera corresponding to the right eye. According to various embodiments, the first rear camera 310 may be configured as a camera corresponding to the right eye, and the second rear camera 320 may be configured as a camera corresponding to the left eye.
[0082] According to embodiments, based on received user input for capturing stereoscopic images, the electronic device can acquire a first image via a first rear camera 310, a second image via a second rear camera 320, and a third image via a front camera 330. According to embodiments, the electronic device can acquire left-eye and right-eye images of the stereoscopic image based on the first and second images. The electronic device can acquire first depth information of the stereoscopic image based on the first and third images. According to various embodiments, the accuracy of the first depth information can be higher than the accuracy of the second depth information obtained based on the first and second images. More will be discussed later. Figure 11 A detailed description is provided in the description.
[0083] According to various embodiments, the electronic device may store data of a stereoscopic image obtained based on a first image and a second image in a memory. The electronic device may also store first depth information obtained based on the first image and a third image in the memory. For example, the electronic device may store the first depth information together with a left-eye image (e.g., the first image) and a right-eye image (e.g., the second image) of the stereoscopic image. Based on received user input for reproducing the stereoscopic image, the electronic device may display the stereoscopic image using the first depth information via a display (e.g., a flexible display). Alternatively, based on received user input for reproducing the stereoscopic image in a wearable electronic device (e.g., an HMD device, a VST device, a VR device, or an XR device), the first depth information may be used to display the stereoscopic image via a display of the wearable electronic device (e.g., an HMD device, a VST device, a VR device, or an XR device). Reference will be made later. Figure 9a , Figure 9b , Figure 10a and Figure 10b Various embodiments for displaying stereoscopic images using first depth information are described in detail.
[0084] According to various embodiments, when capturing stereoscopic images, the user can, as follows: Figure 3 The device is shown capturing stereoscopic images while being held horizontally. At this time, because the user typically perceives the image as being captured by the rear cameras 310 and / or 320, the front camera 330 may be obscured by the hand. Even if the front camera 330 is obscured, stereoscopic images can still be obtained through the multiple rear cameras 310 and 320. However, since the front camera 330 can be used to obtain high-precision first depth information, the device can display a guide screen to allow the user to change their grip position when the front camera 330 is obscured. The user can recognize the displayed guide screen and adjust their grip position to obtain depth information through the front camera 330. According to various embodiments, the guide screen may include a screen displaying the position of the front camera 330 or a screen displaying guidance for the hand's position.
[0085] Figure 4 This is a block diagram of an electronic device including multiple cameras according to embodiments of the present disclosure.
[0086] Reference Figure 4 The electronic device 101, which includes multiple cameras, may include a processor 420 (e.g., Figure 1 The processor 120 and memory 430 (e.g., Figure 1 The memory 130 in the flexible display 200 (e.g., memory 130) Figure 1 The display module 160 in the middle) and multiple cameras (e.g., Figure 1 (Camera module 180 in the middle). Multiple cameras may include a first rear camera 310 and a second rear camera 320. Although Figure 4 Two rear cameras are shown, but this is for illustrative purposes only, and three or more rear cameras may be included. Multiple cameras may include at least one front-facing camera 330.
[0087] According to an embodiment, the processor 420 can control the overall operation of the electronic device 101. For example, the processor 420 can be coupled with... Figure 1 The processor 420 in the embodiment is implemented in the same or similar manner. The processor 420 according to the embodiment can execute software (e.g., Figure 1 The program 140 in the processor 420 controls at least one other component (e.g., a hardware or software component) of the electronic device 101 connected to the processor 420, and can perform data processing or operations based on instructions. Instructions according to embodiments may include commands configurable in a machine language capable of being processed by the electronic device 101 or the processor 420. For example, instructions may include commands corresponding to operational instructions used in the program.
[0088] Furthermore, although electronic device 101 is shown in FIG. 2 as including a processor 420, this is exemplary, and the technical concept of this disclosure is not limited thereto. For example, electronic device 101 may include at least one processor. For example, processor 420 may be implemented as at least one processor.
[0089] According to an embodiment, memory 430 (e.g., Figure 1 The memory 130 in the processor 420 may store at least one instruction that causes at least one operation of the electronic device 101. The at least one instruction, when executed alone or together by the processor 420, causes the electronic device 101 to perform the corresponding operation.
[0090] According to an embodiment, memory 430 may store data of electronic device 101. For example, the data may include information about content, and the content may include images and / or videos.
[0091] According to an embodiment, the generative AI model may be stored in memory 430. Alternatively, the generative AI model may be stored in an external electronic device (e.g., a server). For example, processor 220 may use the generative AI model to obtain a three-dimensional spatial image (or three-dimensional spatial video) using images obtained from at least one of the first rear camera 310, the second rear camera 320, or the front camera 330.
[0092] According to an embodiment, when a user inputs the execution of an application related to taking photos or videos on the electronic device 101, the processor 420 can execute the corresponding program (e.g., ...) stored in the memory 430 for the application related to taking photos or videos. Figure 1 (Program 140 in the middle).
[0093] According to an embodiment, the processor 420 may display a preview image via the flexible display 200 as the application associated with taking photos or videos is executed. The preview image displayed via the flexible display 200 may correspond to a first image (e.g., a left-eye image) obtained (or captured) by the first rear camera 310 or a second image (e.g., a right-eye image) obtained (or captured) by the second rear camera 320.
[0094] According to an embodiment, when the application receives user input to start capturing stereoscopic images, and then receives user input to end capturing images after a specific time period, the processor 420 can store a first image obtained by the first rear camera 310 and a second image obtained by the second rear camera 320 as a stereoscopic image in the memory 430 from the start time to the end time of the capture. According to an embodiment, the stereoscopic image stored in the memory 430 can be stored as a file compressed based on a specified encoding method (e.g., Multi-View High Efficiency Video Coding (HV-HEVC)). According to an embodiment, when the capture mode in the application is configured as a stereoscopic image capture mode, the user input to start stereoscopic image capture can be configured or recognized as input to a video capture start button, or can be configured or recognized as input to a separately configured stereoscopic image capture start button. According to various embodiments, the processor 420 can obtain a third image via the front camera 330 while capturing stereoscopic images. The processor 420 can obtain depth information based on the first image obtained by the first rear camera 310 and the third image obtained by the front camera 330. According to another embodiment, processor 420 can obtain depth information based on a second image obtained by a second rear camera 320 and a third image obtained by a front camera 330. Processor 420 can store the obtained depth information in memory 430. Processor 420 can synchronize the depth information with a stereoscopic image and store the synchronized depth information in memory 430. See below for further details. Figure 6 A detailed description is provided of a specific embodiment in which depth information is generated in processor 420 based on a first image (or a second image) and a third image.
[0095] Figure 5 This is a diagram illustrating the concept of a depth information generation method according to an embodiment of the present disclosure.
[0096] Reference Figure 5 According to an embodiment, electronic devices (e.g., Figure 4 The electronic device 101 may use a stereoscopic image capture module 510, a depth information generation module 520, an additional information storage module 530, and a codec 540 to capture and obtain stereoscopic images. For example, the stereoscopic image capture module 510, the depth information generation module 520, the additional data storage module 530, and the codec 540 may be included in the electronic device 101 and may be implemented as at least part of software or hardware.
[0097] According to the embodiments, such as Figure 5 As shown, when the electronic device 101 is positioned in a horizontal direction (e.g., the electronic device is rotated 90 degrees), the electronic device 101 can capture stereoscopic images using multiple rear cameras (e.g., a first rear camera 310 and a second rear camera 320). According to various embodiments, such as Figure 5 As shown, electronic devices (e.g., Figure 4 The electronic device 101 can capture stereoscopic images using multiple rear cameras (e.g., a first rear camera 310 and a second rear camera 320) while the electronic device 101 is positioned in a vertical direction (e.g., the electronic device is rotated 0 degrees or 180 degrees). However, for ease of explanation, the operation of capturing stereoscopic images will be mainly described as being performed when the electronic device 101 is positioned in a horizontal direction.
[0098] According to an embodiment, the stereoscopic image capture module 510 may use multiple rear cameras 310 and 320 to acquire a first image (e.g., a left-eye image) and a second image (e.g., a right-eye image). For example, the stereoscopic image capture module 510 may include an application related to taking photos or videos. For example, when an input requesting the execution of the stereoscopic image capture module 510 (e.g., an application related to taking photos or videos) is recognized, the electronic device 101 may execute the stereoscopic image capture module 510 (e.g., an application related to taking photos or videos). When user input for initiating stereoscopic image capture via the stereoscopic image capture module 510 (e.g., an application related to taking photos or videos) is recognized, the first image (e.g., a left-eye image) and the second image (e.g., a right-eye image) may be acquired from the capture start time to the capture end time via the first rear camera 310 and the second rear camera 320.
[0099] According to an embodiment, the depth information generation module 520 may acquire a first image and a third image using a first rear camera 310 and a front camera 330 when capturing a stereoscopic image. The depth information generation module 520 may use the acquired first and third images to obtain depth information related to the stereoscopic image. For example, the depth information may include depth information of the space in which the stereoscopic image is captured. The depth information generation module 520 may obtain camera parameters related to the first rear camera 310 and the front camera 330. The depth information generation module 520 may use camera parameters to obtain depth information. For example, camera parameters may include intrinsic parameters and / or extrinsic parameters. For example, intrinsic parameters may include information about the placement distance (e.g., baseline distance) between the first rear camera 310 and the front camera 330, the focal length of the first rear camera 310, and the focal length of the front camera 330. Extrinsic parameters may include information about the external brightness of the electronic device 101, the presence or absence of lighting, and whether it is indoors. According to another embodiment, the depth information generation module 520 may also use a second image obtained by the second rear camera 320 and a third image obtained by the front camera 330 to obtain depth information.
[0100] According to an embodiment, the additional information storage module 530 can obtain additional information (hereinafter referred to as additional information) related to the stereoscopic image capture. For example, the additional information may include time information (e.g., information about the time of capturing the stereoscopic image), location information (e.g., information about the location of capturing the stereoscopic image), and / or person information (e.g., information about the person included in the stereoscopic image).
[0101] According to an embodiment, codec 540 may generate data 550 for stereoscopic images and store the generated data 550 in a memory (e.g., Figure 4 The data 550 may include a left-eye view (or left-eye image), a right-eye view (or right-eye image), depth information, and / or additional information based on a specified format. For example, the codec 540 may store a first image and a second image obtained from the stereo image capture module 510 as a stereo image. For example, the stereo image may be stored as a file compressed based on a specified encoding method (e.g., Multi-View High-Efficiency Video Coding (HV-HEVC)). The codec 540 may store depth information obtained from the depth information generation module 520 in the data 550. For example, the codec 540 may synchronize the depth information with the stereo image and store the synchronized depth information in the data 550. Additionally, the codec 540 may store additional information obtained from the additional information storage module 530 in the data 550.
[0102] According to an embodiment, electronic device 101 can display (e.g., based on stored data 550) via a screen. Figure 4 The display 200 in the device reproduces or displays a corresponding stereoscopic image. According to another embodiment, after obtaining data about the stereoscopic image stored by the electronic device 101, the wearable electronic device (e.g., an HMD device, VST device, VR device, or XR device) can display the stereoscopic image using first depth information via the display of the wearable electronic device. For example, the wearable electronic device can display the stereoscopic image using the first depth information based on user input identified for reproducing the stereoscopic image. For example, the electronic device 101 can send data about the stereoscopic image to the wearable electronic device upon request.
[0103] Figure 6 This is an illustration of a concept for a depth information estimation method using stereo images according to an embodiment of the present disclosure.
[0104] Reference Figure 6 According to an embodiment, electronic devices (e.g., Figure 4 The electronic device 101 in the image can obtain depth information based on the first image 611 (e.g., the first rear camera 310) and the third image 512 (e.g., the front camera 330).
[0105] According to an embodiment, electronic device 101 can identify and compare similar pixels included in a first image 611 and a third image 612. For example, electronic device 101 can identify a first pixel 621 included in the first image 611 and a first pixel 622 included in the third image 612, and compare the horizontal distance DIS1 between the identified pixels 621 and 622. Electronic device 101 can identify a second pixel 626 included in the first image 611 and a second pixel 627 included in the third image 612, and compare the horizontal distance DIS2 between the identified pixels 626 and 627. For example, the horizontal distance DIS1 or DIS2 can be determined in pixels.
[0106] According to an embodiment, electronic device 101 can use a stereo disparity map estimation model to obtain a displacement map 640 for a first image 611 and a third image 612. Electronic device 101 can obtain the displacement map 640 based on displacement distances DIS1 and DIS2. For example, stereo disparity map estimation model 630 can be configured or trained to obtain the displacement map based on displacement distances DIS1 and DIS2.
[0107] According to an embodiment, electronic device 101 can use depth information conversion model 650 to convert displacement map 640 into depth map 660. For example, depth information conversion model 650 can be configured or trained to convert displacement map into depth map. For example, electronic device 101 can obtain information about depth D based on the focal length (f) of at least one of the first rear camera 310 or front camera 320, the distance (b) between the first rear camera 310 and the front camera 320 (e.g., baseline), and displacement distance (d). For example, depth D can be determined as shown in Equation 1 below. For example, depth D can be determined in mm. For example, displacement distance (d) can be determined based on the horizontal distance DIS1 between first pixels 621 and 622 and / or the horizontal distance DIS2 between second pixels 626 and 627.
[0108] [Equation 1]
[0109] According to an embodiment, the electronic device 101 can obtain a depth map 660 as depth information for the first image 611 and the third image 612.
[0110] Based on the above method, electronic device 101 can obtain depth information using at least two images obtained from different cameras 310 and 330. This will be explained below. Figure 7 The method for obtaining depth information for electronic device 101 is described in detail.
[0111] Figure 7 This is a diagram illustrating a method for generating depth information according to an embodiment of the present disclosure.
[0112] Reference Figure 7 According to an embodiment, the depth information generation module 520 may include a first image correction module 710, an image rectification module 720, a depth information estimation module 730, and a second image correction module 740.
[0113] According to an embodiment, the first image correction module 710 can receive a first image obtained from the first rear camera 310 and a third image obtained from the front camera 330. The first image correction module 710 can correct, alter, or rotate the captured images based on the vertical arrangement of the first rear camera 310 and the front camera 330. For example, the first image correction module 710 can rotate the vertically positioned image by 90 degrees (or 270 degrees). Additionally, the first image correction module 710 can reduce the rotated image to a specified size (e.g., 0.25 times the original size). For example, the image size reduction operation can be omitted to reduce computational load.
[0114] According to an embodiment, the image correction module 720 can acquire or output a horizontally set image based on information about pre-calibrated camera parameters (e.g., intrinsic and / or extrinsic parameters) and the reception of a first image and a third image. For example, the first image and the third image may be images modified (e.g., rotated and / or resized) by the first image correction module. For example, the image correction module 720 can acquire and output multiple binocular images (e.g., two images corresponding to the first image and the third image) based on horizontally aligning the first image and the third image side by side in a virtual planar space.
[0115] According to an embodiment, the depth information estimation module 730 can estimate and output depth information based on received binocular images aligned in the horizontal direction. For example, the depth information estimation module 730 can estimate the depth information based on a deep learning model or a non-deep learning model (e.g., a convolutional method). For example, the depth information estimation module 730 can include the above-mentioned... Figure 6 The stereo disparity map estimation module 630 and depth map transformation model 650 are described herein. For example, when the depth information estimation module 730 uses a deep learning model, Figure 6 At least one of the stereo disparity map estimation module 630 and depth map transformation model 650 described herein may include a deep learning model.
[0116] According to an embodiment, the second image correction module 740 can correct and output the depth information received from the depth information estimation module 730 based on changes made to the first image correction module 710 (e.g., rotation and size reduction). For example, the second image correction module 740 can reverse the depth information by 90 degrees (or 270 degrees) and adjust the size of the depth information (e.g., increase the size) to match the target size. The second image correction module 740 can output the corrected depth information as depth information for both the first and third images.
[0117] Based on the above method, electronic device 101 can use at least two images obtained from different cameras 310 and 330 to obtain depth information.
[0118] Figure 8 This is an illustration of an example of a data format for stereoscopic video including depth information according to an embodiment of the present disclosure.
[0119] Reference Figure 8 According to an embodiment, the data used for stereoscopic images (e.g., Figure 5 The data 550 may include a left-eye view 810, a right-eye view 820, and a depth image 830 corresponding to depth information, captured or acquired in chronological order. For example, the left-eye view 810 may include a first image (e.g., a left-eye image) acquired by the first rear camera 310, and the right-eye view 820 may include a second image (e.g., a left-eye image) acquired by the second rear camera 320.
[0120] According to an embodiment, electronic devices (e.g., Figure 4 The electronic device 101 can match the storage time of the left-eye view 810 and the right-eye view 820 with the storage time of the depth image 830. For example, the electronic device 101 can determine the storage time of the depth image 830 based on the capture time of the left-eye image corresponding to the left-eye view 810. For example, since the left-eye image corresponding to the left-eye view 810 is necessary for generating both the stereoscopic image and the depth image, the storage time of the left-eye view 810 and the right-eye view 820, as well as the storage time of the depth image 830, can be matched or synchronized based on the capture time of the left-eye image corresponding to the left-eye view 810.
[0121] According to an embodiment, electronic devices (e.g., Figure 4 The electronic device 101 can compress the left-eye view 810 and the right-eye view 820 based on a base view. For example, when the image corresponding to the base view is the left-eye view 810, the electronic device 101 can store data 550 based on the compression of the right-eye image 820 according to the left-eye view 810.
[0122] According to an embodiment, in addition to the left-eye view 810 and right-eye view 820 of the stereoscopic image, the electronic device 101 may also store the depth image 830 in the data 550. For example, the electronic device 101 may synchronize the depth image with the left-eye view 810 and right-eye view 820, and may add and store the depth image synchronized with the left-eye view 810 and right-eye view 820 in the MV-HEVC video track. For example, since the depth image 830 is configured with single-channel depth values that are not similar to the stereoscopic binocular images 810 and 820, compression based on the reference view may not be possible. Therefore, the electronic device 101 may store the depth image 830 together with the compressed binocular images 810 and 820 of the stereoscopic image in the MV-HEVC video track, such as... Figure 8 As shown.
[0123] Figure 9a and Figure 9b This is a diagram illustrating a method for displaying stereoscopic video using depth information in an electronic device or wearable electronic device according to an embodiment of the present disclosure.
[0124] Reference Figure 9a and Figure 9b According to an embodiment, electronic device 101 or wearable electronic device (e.g., already from...) Figure 4 The electronic device 101 in the middle (a wearable electronic device that obtains data about stereoscopic video) can use depth information to display stereoscopic video. For example, the wearable electronic device may include an HMD device, a VST device, a VR device, or an XR device.
[0125] Reference Figure 9aAccording to an embodiment, electronic device 101 or wearable electronic device can generate parallax-adjusted binocular video content (e.g., binocular image) 930 based on a first image (e.g., an image obtained by a first rear camera 310) 910 for a left-eye image and a second image (e.g., an image obtained by a second rear camera 320) for a right-eye image. At this time, electronic device 101 or wearable electronic device can use depth information (e.g., depth information generated based on the first image 910 and a third image (e.g., an image obtained by a front camera 330)) to generate a depth-sensing binocular image 930. For example, electronic device 101 or wearable electronic device can obtain a warped image 920 based on the parallax (or depth) between the first image 910 and the second image. The warped image may include at least a portion of the first image 910 and the second image 920 based on the parallax. For example, at least a portion of the warped image may include uncolored portions. Electronic device 101 or wearable electronic device may use the first image 910, the second image, and depth information (or the third image) to generate or obtain an inpainted image 930. For example, the inpainted image 930 may include an image in which unpainted portions of a distorted image 920 have been additionally colored. Electronic device 101 or wearable electronic device may obtain the inpainted image 930 as parallax-adjusted binocular video content (e.g., a binocular image).
[0126] Based on the above method, electronic device 101 or wearable electronic device can use depth information to obtain images for binocular viewing with enhanced depth perception and spatial impression.
[0127] Reference Figure 9b According to an embodiment, the electronic device 101 or wearable electronic device can be as described above. Figure 9a As described, a stereoscopic image is displayed using a first image 940 for a parallax-adjusted left-eye image and a second image 950 for a right-eye image (e.g., an image obtained via a second rear camera 320). The electronic device 101 or wearable electronic device can adjust the depth perception of the stereoscopic image based on user input. The electronic device 101 or wearable electronic device can recognize the depth perception level corresponding to the user input and generate a stereoscopic image corresponding to the recognized depth perception level.
[0128] According to an embodiment, electronic device 101 or wearable electronic device may display a control bar 960 for adjusting the sense of depth. For example, wearable electronic device may display the control bar 960 on a screen displaying a stereoscopic image. Based on user input to the control bar 960, electronic device 101 or wearable electronic device may display a screen with adjusted depth. For example, when user input moving to the right on the control bar 960 is recognized (e.g., user input to increase the sense of depth), electronic device 101 may display a stereoscopic image on the screen based on the length of the rightward movement (e.g., a stereoscopic image with enhanced depth based on that length).
[0129] Based on the above method, electronic device 101 or wearable electronic device can use depth information to obtain images for binocular viewing with enhanced depth perception and spatial impression. Furthermore, electronic device 101 or wearable electronic device can display stereoscopic images with depth perception adjusted based on user input.
[0130] Figure 10a and Figure 10b This is a diagram illustrating a method for displaying stereoscopic video using depth information in an electronic device or wearable electronic device according to an embodiment of the present disclosure.
[0131] Reference Figure 10a According to an embodiment, electronic device 101 or wearable electronic device (e.g., already from...) Figure 4 The electronic device 101 in the middle (a wearable electronic device that obtains data about stereoscopic video) can use depth information to display stereoscopic video. For example, the wearable electronic device may include an HMD device, a VST device, a VR device, or an XR device.
[0132] Reference Figure 10aAccording to an embodiment, electronic device 101 or wearable electronic device can generate a stereoscopic image representing three-dimensional space based on multiple images captured by electronic device 101 (e.g., images obtained by the first rear camera 310, the second rear camera 320, and the front camera 330) 1011, 1012, 1013, and 1014. At this time, electronic device 101 or wearable electronic device can use depth information to generate an image 1020 representing a three-dimensional space with a sense of depth. For example, electronic device 101 or wearable electronic device can obtain depth information using images obtained by the first rear camera 310 and the front camera 330 included in electronic device 101. Electronic device 101 or wearable electronic device can generate a stereoscopic image representing three-dimensional space by utilizing multiple images 1011, 1012, 1013, and 1014 and depth information. For example, electronic device 101 or wearable electronic device can generate an image of portions not represented in the multiple images 1011, 1012, 1013, and 1014 by performing repair based on multiple images 1011, 1012, 1013, and 1014 and depth information. Thus, electronic device 101 or wearable electronic device can generate a stereoscopic image representing three-dimensional space. Furthermore, electronic device 101 can support free-view functionality for three-dimensional space. For example, electronic device 101 or wearable electronic device can display a screen representing the three-dimensional space viewed from a user-requested location via free-view functionality.
[0133] Reference Figure 10b According to an embodiment, the electronic device 101 or wearable electronic device may be based on the above... Figure 10a The method for generating three-dimensional space described herein generates and displays an image 1030 representing three-dimensional space. Electronic device 101 can use depth information to generate a three-dimensional space with a more enhanced sense of depth and spatial impression. Furthermore, electronic device 101 or wearable electronic device can use depth information to provide free-view functionality for three-dimensional space.
[0134] According to an embodiment, electronic device 101 or wearable electronic device may display a control window 1040 on an image 1030 representing three-dimensional space to support free-view functionality. For example, based on user input to the control window 1040, electronic device 101 or wearable electronic device may display a screen representing the three-dimensional space viewed from a position indicated by the user input.
[0135] Based on the above method, electronic device 101 or wearable electronic device can use depth information display to represent a three-dimensional space with enhanced depth and spatial impression.
[0136] Figure 11 This is a diagram illustrating an example of depth information estimation based on baseline length difference according to an embodiment of the present disclosure.
[0137] Reference Figure 11 According to an embodiment, the first distance (or first baseline) (e.g., 85 mm) between the first rear camera 310 and the front camera 330 may be longer than the second distance (or second baseline) (e.g., 19 mm) between the first rear camera 310 and the second rear camera 320.
[0138] Reference Figure 11 According to a comparative embodiment, electronic device 101 may not be able to estimate depth information based on the first image 1110 and the second image 1120 obtained using rear cameras 310 and 320. For example, the second distance (or second baseline distance) between the first rear camera 310 and the second rear camera 320 may be 19 mm, and the parallax between the first image 1110 and the second image 1120 may be identified as 3 (e.g., 13 pixels). For example, electronic device 101 may reduce the size of the first image 1110 and the second image 1120 to a specified size (e.g., a reduction of 0.25 times) for high-speed calculation of depth information. For example, depth information estimation model 1150 may identify the parallax between the first image 1115 and the second image 1125 reduced to a specified size as approximately 0, and the corresponding length as approximately 0 mm. Therefore, electronic device 101 may not be able to obtain depth information based on the first image 1110 and the second image 1120 obtained using rear cameras 310 and 320 using depth information estimation model 1150.
[0139] According to an embodiment, electronic device 101 can estimate depth information based on a first image 1110 and a third image 1130 obtained using a first rear camera 310 and a front camera 330. For example, a first distance (or a first baseline distance) between the first rear camera 310 and the front camera 330 can be 85 mm, and the parallax between the first image 1110 and the third image 1130 can be identified as 13 (e.g., 13 pixels), and the corresponding length can be identified as approximately 6800 mm. For example, electronic device 101 can reduce the size of the first image 1110 and the third image 1130 to a specified size (e.g., a reduction of 0.25 times) for high-speed calculation of depth information. For example, depth information estimation model 1150 can identify the parallax between the first image 1115 and the third image 1135 reduced to a specified size as approximately 3, and the corresponding length can be identified as approximately 1700 mm. Therefore, the electronic device 101 can use the depth information estimation model 1150 to obtain depth information based on the first image 1110 and the third image 1130 obtained using the first rear camera 310 and the front camera 330.
[0140] Figure 12This is a diagram illustrating a method for displaying stereoscopic video using depth information in a wearable electronic device according to an embodiment of the present disclosure.
[0141] Reference Figure 12 A, According to an embodiment, electronic devices (e.g., Figure 4 The electronic device 101 can estimate first depth information based on a first image 1210 and a third image 1220 obtained using a first rear camera 310 and a front camera 330. For example, the electronic device 101 can generate a first depth image 1240 corresponding to the estimated first depth information. Subsequently, a wearable electronic device (e.g., an HMD device, a VST device, a VR device, or an XR device) that has obtained the first depth image 1240 and an image for stereoscopic imaging (e.g., an image captured by the first rear camera 310 and the second rear camera 320) from the electronic device 101 can obtain and display a first stereoscopic image 1260 based on the first depth image 1240.
[0142] Reference Figure 12 B. According to a comparative embodiment, electronic device 101 can estimate second depth information based on a first image 1210 and a second image 1230 obtained using rear cameras 310 and 320. For example, electronic device 101 can generate a second depth image 1250 corresponding to the estimated second depth information. Electronic device 101 can obtain a second stereoscopic image 1270 based on the second depth image 1250. For example, the accuracy of the second depth information may be lower than the accuracy of the first depth information. Therefore, the accuracy of the second depth image 1250 may be lower than the accuracy of the first depth image 1240. In addition, the accuracy of the second stereoscopic image 1270 may be lower than the accuracy of the first stereoscopic image 1260. For example, the accuracy of a portion 1275 of the second stereoscopic image 1270 may be lower than the accuracy of a portion 1265 of the first stereoscopic image 1260.
[0143] Based on the above method, electronic device 101 can use depth information from images obtained by the first rear camera 310 and the front camera 330 to generate stereoscopic images with enhanced depth and spatial impression. Wearable electronic devices (e.g., already from...) Figure 4 The electronic device 101 in the wearable electronic device receives data for stereoscopic images and can receive stereoscopic images generated simply and efficiently by the electronic device 101, and can reproduce the received stereoscopic images.
[0144] Figure 13 This is a diagram illustrating a method for displaying stereoscopic video using depth information in a wearable electronic device according to an embodiment of the present disclosure.
[0145] Reference Figure 13 According to an embodiment, electronic devices (e.g., Figure 4The electronic device 101 can estimate first depth information based on a first image 1310 and a third image 1320 obtained using a first rear camera 310 and a front camera 330. For example, the electronic device 101 can generate a first depth image 1340 corresponding to the estimated first depth information. Subsequently, a wearable electronic device (e.g., an HMD device, VST device, VR device, or XR device) that has obtained the first depth image 1240 and images for stereoscopic images (e.g., images captured by the first rear camera 310 and the second rear camera 320) from the electronic device 101 can obtain and display a first spatial image (e.g., a spatial cloud image) 1360 for reconstructing (or generating) a three-dimensional space (or a three-dimensional spatial model) based on the first depth image 1340.
[0146] Reference Figure 12 B. According to a comparative embodiment, electronic device 101 can estimate second depth information based on a first image 1315 and a second image 1330 obtained using rear cameras 310 and 320. For example, electronic device 101 can generate a second depth image 1350 corresponding to the estimated second depth information. Electronic device 101 can obtain a second spatial image (e.g., a spatial cloud image) 1370 based on the second depth image 1350 for reconstructing (or generating) a three-dimensional space (or a three-dimensional spatial model). For example, the accuracy of the second depth information may be lower than the accuracy of the first depth information. Therefore, the accuracy of the second depth image 1350 may be lower than the accuracy of the first depth image 1340. In addition, the accuracy of the second spatial image 1370 may be lower than the accuracy of the first spatial image 1360.
[0147] Based on the above method, electronic device 101 can generate (or obtain) a more accurate spatial image using depth information from images obtained by the first rear camera 310 and the front camera 330. Electronic device 101 can use the more accurate spatial image to generate (or reconstruct) an image representing a three-dimensional space with a more enhanced sense of depth and spatial impression. Wearable electronic devices (e.g., already from...) Figure 4 The electronic device 101 in the wearable electronic device receives data for stereoscopic images and can receive stereoscopic images generated simply and efficiently by the electronic device 101 and reproduce the received stereoscopic images.
[0148] Figure 14 This is a flowchart illustrating a method for an electronic device to obtain a stereoscopic image according to an embodiment of the present disclosure.
[0149] Reference Figure 14 According to an embodiment, in operation 1401, the electronic device (e.g., Figure 4 The electronic device 101 in the middle can, based on the user input received for capturing stereo images, transmit the data via a first rear camera (e.g., ...). Figure 4The first image is obtained by the first rear camera 310, and the second rear camera (e.g., ...) is used to obtain the first image. Figure 4 The second image is obtained by the second rear camera 320, and by the front camera (e.g., ...). Figure 4 The front-facing camera 330 in the device 101 acquires a third image. For example, when the housing of the electronic device 101 is in a fully extended state (e.g., the extended state of the electronic device), the electronic device 101 may acquire or capture a first image, a second image, and a third image. For example, when the electronic device 101 is positioned in a designated direction where the first rear camera 310 and the second rear camera 320 are arranged vertically (or when the electronic device 101 is arranged horizontally), the electronic device 101 may acquire or capture a first image, a second image, and a third image. For example, the electronic device 101 may acquire the first image, the second image, and the third image based on the recognition that the electronic device 101 is arranged in a designated direction and the electronic device is fully extended. For example, based on the recognition that the electronic device 101 is not arranged in a designated direction or the electronic device is not fully extended, the electronic device 101 may display guidance information to guide the electronic device 101 to be arranged in the designated direction and fully extended.
[0150] According to an embodiment, when the housing of electronic device 101 is not fully unfolded, electronic device 101 can obtain a first image via the first rear camera 310 and obtain or capture a second image via the second rear camera 320. For example, based on the recognition that electronic device 101 is not fully unfolded, electronic device 101 may obtain only the first and second images. For example, electronic device 101 may obtain and store a general image based on the first and second images. Optionally, based on the recognition that electronic device 101 is not fully unfolded, electronic device 101 may display guidance information to guide electronic device 101 to fully unfold to obtain a stereoscopic image. According to an embodiment, electronic device 101 may obtain and store left-eye and right-eye images of a stereoscopic image based on the first and second images without depth information. According to an embodiment, in operation 1403, electronic device 101 may obtain and store left-eye and right-eye images of a stereoscopic image based on the first and second images. For example, the first image may be a left-eye image for a left-eye view, and the second image may be a right-eye image for a right-eye view.
[0151] According to an embodiment, in operation 1405, electronic device 101 may obtain and store first depth information related to the stereoscopic image based on the first image and the third image. For example, operation 1405 may be executed in parallel or simultaneously with operation 1403.
[0152] According to an embodiment, in operation 1407, electronic device 101 may store data of a stereoscopic image including first depth information. For example, electronic device 101 may store the first depth information together with left-eye and right-eye images of the stereoscopic image. For example, electronic device 101 may store data of a stereoscopic image based on a specified format (e.g., a format based on the Multi-View High Efficiency Video Coding (HV-HEVC) method).
[0153] According to an embodiment, in operation 1409, after receiving or acquiring stereoscopic image data from electronic device 101, the wearable electronic device (e.g., HMD device, VST device, VR device, or XR device) can display the stereoscopic image using first depth information on the display of the wearable electronic device. For example, the wearable electronic device can display the stereoscopic image using the first depth information based on user input identified for reproducing the stereoscopic image. For example, based on user input identified for reproducing the stereoscopic image, the wearable electronic device can request electronic device 101 to send data about the stereoscopic image. For example, electronic device 101 can send the stereoscopic image data to the wearable electronic device according to the request of the wearable electronic device. For example, the wearable electronic device can use the first depth information included in the data of the stereoscopic image to display the stereoscopic image. The wearable electronic device can use the first depth information to generate (or reconstruct) a stereoscopic image with higher precision and depth or an image representing a three-dimensional space with higher precision and depth, and can display the stereoscopic image or the image representing a three-dimensional space.
[0154] According to another embodiment, the electronic device 101 may, based on received user input for reproducing a stereoscopic image, use first depth information to display (e.g., ...) Figure 4 The flexible display 200 in the device displays a stereoscopic image. For example, the electronic device 101 can use the first depth information to display a stereoscopic image with higher precision and spatial impression. In addition, the electronic device 101 can use the first depth information to generate (or reconstruct) an image representing a three-dimensional space with higher precision and depth, and can display an image representing the three-dimensional space.
[0155] Based on the above method, the electronic device 101 can use depth information from images obtained by the first rear camera 310 and the front camera 330 to capture and generate stereoscopic images with higher precision and spatial impression. Furthermore, the electronic device 101 can simply and effectively capture and generate stereoscopic images using depth information from images obtained by the first rear camera 310 and the front camera 330.
[0156] According to an embodiment, an electronic device 101 may include: a first housing including a first surface and a second surface opposite to the first surface; a second housing including a third surface and a fourth surface opposite to the third surface; a hinge structure rotatably connecting the first housing and the second housing; a flexible display 200 disposed on the first surface and the third surface; a first rear camera 310 and a second rear camera 320 disposed on the second surface; a front camera 330 disposed on the fourth surface; at least one processor 420 including processing circuitry; and a memory 430 configured to store instructions. According to an embodiment, the instructions, when executed individually or jointly by at least one processor, may cause the electronic device to: acquire a first image via the first rear camera, acquire a second image via the second rear camera, and acquire a third image via the front camera based on received user input for capturing a stereoscopic image. According to an embodiment, the instructions, when executed individually or jointly by at least one processor, may cause the electronic device to: acquire left-eye and right-eye images of a stereoscopic image based on the first and second images. According to an embodiment, the instructions, when executed individually or jointly by at least one processor, can cause an electronic device to perform the following operation: obtain first depth information of a stereoscopic image based on a first image and a third image.
[0157] According to an embodiment, the accuracy of the first depth information may be higher than the accuracy of the second depth information obtained based on the first image and the second image.
[0158] According to an embodiment, the instructions, when executed individually or jointly by at least one processor, can cause the electronic device to perform the following operation: store stereoscopic image data in a memory. According to an embodiment, the instructions, when executed individually or jointly by at least one processor, can cause the electronic device to perform the following operation: display the stereoscopic image via a flexible display using first depth information based on received user input for reproducing the stereoscopic image.
[0159] According to an embodiment, the instructions, when executed individually or jointly by at least one processor, can cause an electronic device to perform the following operation: based on received user input for reproducing a stereoscopic image, generate information about the three-dimensional space of the stereoscopic image using a first image, a second image, and first depth information. According to an embodiment, the instructions, when executed individually or jointly by at least one processor, can cause an electronic device to perform the following operation: display the stereoscopic image using the information about the three-dimensional space.
[0160] According to an embodiment, when executed by at least one processor individually or jointly, the instructions can cause the electronic device to perform the following operation: storing data of a stereoscopic image including the first depth information based on synchronizing the first depth information with a first image and a second image.
[0161] According to an embodiment, the instructions, when executed individually or jointly by at least one processor, may cause the electronic device to perform the following operation: estimate a disparity map for the first and third images based on the horizontal distance between pixels corresponding to the same object included in each of the first and third images. According to an embodiment, the instructions, when executed individually or jointly by at least one processor, may cause the electronic device to perform the following operation: obtain first depth information based on the disparity map.
[0162] According to an embodiment, the instructions, when executed individually or jointly by at least one processor, can cause the electronic device to perform the following operation: obtain first depth information based on providing a first image and a third image to a pre-trained model.
[0163] According to an embodiment, the instructions, when executed individually or jointly by at least one processor, can cause the electronic device to perform the following operations: identify whether the electronic device is positioned in a specified direction and whether the electronic device is fully deployed. According to an embodiment, the instructions, when executed individually or jointly by at least one processor, can cause the electronic device to perform the following operations: based on identifying that the electronic device is positioned in a specified direction and that the electronic device is fully deployed, obtain a first image, a second image, and a third image.
[0164] According to an embodiment, when executed by at least one processor individually or jointly, the instructions may cause the electronic device to perform the following operations: based on the identification that the electronic device is not positioned in a specified direction or is not fully deployed, display guidance information to guide the electronic device to be positioned in the specified direction and the electronic device to be fully deployed.
[0165] According to an embodiment, the first distance between the first rear camera and the front camera may be longer than the second distance between the first rear camera and the second rear camera.
[0166] According to an embodiment, an operation method for an electronic device 101 is provided, and the electronic device may include: a first housing including a first surface and a second surface opposite to the first surface; a second housing including a third surface and a fourth surface opposite to the third surface; a hinge structure rotatably connecting the first housing and the second housing; a flexible display 200 disposed on the first surface and the third surface; a first rear camera 310 and a second rear camera 320 disposed on the second surface; and a front camera 330 disposed on the fourth surface. According to an embodiment, the operation method of the electronic device may include: obtaining a first image via the first rear camera, obtaining a second image via the second rear camera, and obtaining a third image via the front camera based on received user input for capturing a stereoscopic image. According to an embodiment, the operation method of the electronic device may include: obtaining a left-eye image and a right-eye image of the stereoscopic image based on the first image and the second image. According to an embodiment, the operation method of the electronic device may include: obtaining first depth information of the stereoscopic image based on the first image and the third image.
[0167] According to an embodiment, the accuracy of the first depth information may be higher than the accuracy of the second depth information obtained based on the first image and the second image.
[0168] According to an embodiment, the operation method of the electronic device may further include: storing data of a stereoscopic image in a memory. According to an embodiment, the operation method of the electronic device may further include: displaying the stereoscopic image via a flexible display using first depth information based on received user input for reproducing the stereoscopic image.
[0169] According to an embodiment, the step of displaying a stereoscopic image may include: generating information about the three-dimensional space of the stereoscopic image using a first image, a second image, and first depth information, based on received user input for reproducing the stereoscopic image. According to an embodiment, the step of displaying a stereoscopic image may include displaying the stereoscopic image using the information about the three-dimensional space.
[0170] According to an embodiment, the step of storing data may include: storing data of a stereoscopic image including the first depth information based on synchronizing the first depth information with the first image and the second image.
[0171] According to an embodiment, the step of obtaining the first depth information may include: estimating a disparity map for the first image and the third image based on the horizontal distance between pixels corresponding to the same object included in each of the first image and the third image. According to an embodiment, the step of obtaining the first depth information may include: obtaining the first depth information based on the disparity map.
[0172] According to an embodiment, the step of obtaining the first depth information may include: obtaining the first depth information based on providing the first image and the third image to a pre-trained model.
[0173] According to an embodiment, the method may further include: identifying whether the electronic device is positioned along a specified direction and whether the electronic device is fully deployed. According to an embodiment, the step of obtaining the first image, the second image, and the third image may include: obtaining the first image, the second image, and the third image based on identifying that the electronic device is positioned along a specified direction and that the electronic device is fully deployed.
[0174] According to an embodiment, the operation method of the electronic device may further include: based on the identification that the electronic device is not set in a specified direction or the electronic device is not fully unfolded, displaying guidance information to guide the electronic device to be set in a specified direction and the electronic device to be fully unfolded.
[0175] According to an embodiment, a non-transitory computer-readable storage medium 130 or 430 is provided, wherein the non-transitory computer-readable storage medium 130 or 430 is configured to store instructions, and the instructions, when executed individually or jointly by at least one processor 420, can cause an electronic device 101 to operate, wherein the electronic device 101 includes: a first housing including a first surface and a second surface opposite to the first surface; a second housing including a third surface and a fourth surface opposite to the third surface; a hinge structure rotatably connecting the first housing and the second housing; a flexible display 200 disposed on the first surface and the third surface; a first rear camera 310 and a second rear camera 320 disposed on the second surface; and a front camera 330 disposed on the fourth surface, wherein the operation includes: obtaining a first image through the first rear camera, obtaining a second image through the second rear camera and obtaining a third image through the front camera based on received user input for capturing a stereoscopic image, obtaining a left-eye image and a right-eye image of the stereoscopic image based on the first image and the second image, and obtaining first depth information of the stereoscopic image based on the first image and the third image.
[0176] The electronic device according to various embodiments can be one of a variety of types of electronic devices. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. According to embodiments of this disclosure, the electronic device is not limited to those described above.
[0177] It should be understood that the various embodiments of this disclosure and the terminology used therein are not intended to limit the technical features set forth herein to the specific embodiments, but rather to include various changes, equivalents, or substitutions to the respective embodiments. In the description of the drawings, similar reference numerals may be used to refer to similar or related elements. It will be understood that, unless the relevant context clearly indicates otherwise, the singular form of a noun corresponding to a term may include one or more things. As used herein, each of the phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B, or C,” “at least one of A, B, and C,” and “at least one of A, B, or C” may include any one or all possible combinations of the items enumerated together with the corresponding phrase among the plurality of phrases. As used herein, terms such as “first” and “second” or “first” and “second” may be used to simply distinguish one component from another and do not limit the components in other respects (e.g., importance or order). It will be understood that, whether the terms “operably” or “communically” are used or not, if an element (e.g., a first element) is referred to as “combined with another element (e.g., a second element),” “combined to another element (e.g., a second element),” “connected to another element (e.g., a second element),” or “attached to another element (e.g., a second element)”, it means that the first element can be directly (e.g., wiredly) combined with the second element, wirelessly combined with the second element, or combined with the second element via a third element.
[0178] As used in connection with various embodiments of this disclosure, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with other terms such as "logic," "logic block," "part," or "circuit." A module may be a single integrated component adapted to perform one or more functions, or the smallest unit or part of such a single integrated component. For example, according to embodiments, a module may be implemented in the form of an application-specific integrated circuit (ASIC).
[0179] The various embodiments set forth herein can be implemented as software (e.g., program 140) containing one or more instructions readable by a machine (e.g., electronic device 101) stored in a storage medium (e.g., internal memory 136 or external memory 138). For example, under the control of a processor, the processor (e.g., processor 120) of the machine (e.g., electronic device 101) can invoke and execute at least one of the one or more instructions stored in the storage medium, with or without the use of one or more other components. This enables the machine to operate to perform at least one function according to the invoked at least one instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. Machine-readable storage media can be provided in the form of non-transitory storage media. The term "non-transitory" simply means that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but this term does not distinguish between data being stored semi-permanently in the storage medium and data being temporarily stored in the storage medium.
[0180] According to embodiments, methods according to various embodiments of this disclosure may be included and provided in a computer program product. The computer program product can be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)) or via an app store (e.g., the Play Store). TM The computer program product may be published online (e.g., downloaded or uploaded), or may be distributed directly between two user devices (e.g., smartphones) (e.g., downloaded or uploaded). If published online, at least a portion of the computer program product may be temporarily generated, or at least a portion of the computer program product may be temporarily stored in a machine-readable storage medium (such as the memory of a manufacturer's server, an app store's server, or a forwarding server).
[0181] According to various embodiments, each of the above components (e.g., a module or program) may include a single entity or multiple entities, and some of the multiple entities may be separately disposed in different components. According to various embodiments, one or more of the above components may be omitted, or one or more other components may be added. Optionally or additionally, multiple components (e.g., modules or programs) may be integrated into a single component. In this case, according to various embodiments, the integrated component may still perform the one or more functions of each of the multiple components in the same or similar manner as the corresponding component of the multiple components performed one or more functions before integration. According to various embodiments, the operations performed by a module, program, or other component may be performed sequentially, in parallel, repeatedly, or heuristically, or one or more operations may be run in a different order or omitted, or one or more other operations may be added.
Claims
1. An electronic device (101), comprising: The first housing includes a first surface and a second surface opposite to the first surface; The second housing includes a third surface and a fourth surface opposite to the third surface; A hinge structure rotatably connects the first housing and the second housing; A flexible display (200) is disposed on a first surface and a third surface; A first rear camera (310) and a second rear camera (320) are disposed on the second surface; The front-facing camera (330) is located on the fourth surface; At least one processor (420), including processing circuitry; and Memory (430) stores instructions, wherein the instructions are configured to cause the electronic device to perform the following operations when executed individually or jointly by the at least one processor: Based on the received user input for capturing stereo images, a first image is obtained through a first rear camera, a second image is obtained through a second rear camera, and a third image is obtained through the front camera; Based on the first image and the second image, obtain the left-eye image and the right-eye image of the stereoscopic image; as well as Based on the first image and the third image, the first depth information of the stereo image is obtained.
2. The electronic device as claimed in claim 1, wherein, The accuracy of the first depth information is higher than that of the second depth information obtained based on the first and second images.
3. The electronic device according to any one of claims 1 to 2, wherein, The instructions are configured to cause the electronic device to perform the following operations when executed individually or jointly by the at least one processor: The stereoscopic image data is stored in the memory; as well as Based on the user input received for reproducing the stereoscopic image, the stereoscopic image is displayed through the flexible display using first depth information.
4. The electronic device according to any one of claims 1 to 3, wherein, The instructions are configured to cause the electronic device to perform the following operations when executed individually or jointly by the at least one processor: Based on the user input received for reproducing the stereo image, information about the three-dimensional space of the stereo image is generated using a first image, a second image, and first depth information; as well as The stereoscopic image is displayed using the information about three-dimensional space.
5. The electronic device according to any one of claims 1 to 4, wherein, The instructions are configured to cause the electronic device to perform the following operations when executed individually or jointly by the at least one processor: The data of the stereoscopic image, including the first depth information, is stored by synchronizing the first depth information with the first image and the second image.
6. The electronic device according to any one of claims 1 to 5, wherein, The instructions are configured to cause the electronic device to perform the following operations when executed individually or jointly by the at least one processor: Based on the horizontal distance between pixels corresponding to the same objects included in each of the first and third images, a disparity map is estimated for the first and third images; as well as Based on the disparity map, first depth information is obtained.
7. The electronic device according to any one of claims 1 to 6, wherein, The instructions are configured to cause the electronic device to perform the following operations when executed individually or jointly by the at least one processor: First depth information is obtained by providing the first and third images to a pre-trained model.
8. The electronic device according to any one of claims 1 to 7, wherein, The instructions are configured to cause the electronic device to perform the following operations when executed individually or jointly by the at least one processor: Identify whether the electronic device is positioned in a specified direction and whether the electronic device is fully deployed; and Based on the identification that the electronic device is set along the specified direction and that the electronic device is fully unfolded, a first image, a second image, and a third image are obtained.
9. The electronic device according to any one of claims 1 to 8, wherein, The instructions are configured to cause the electronic device to perform the following operations when executed individually or jointly by the at least one processor: Based on the identification that the electronic device is not set in the specified direction or the electronic device is not fully deployed, guidance information is displayed to guide the electronic device to be set in the specified direction and the electronic device to be fully deployed.
10. The electronic device according to any one of claims 1 to 9, wherein, The first distance between the first rear camera and the front camera is longer than the second distance between the first rear camera and the second rear camera.
11. A method of operating an electronic device (101), wherein, The electronic device includes: a first housing including a first surface and a second surface opposite to the first surface; a second housing including a third surface and a fourth surface opposite to the third surface; a hinge structure rotatably connecting the first housing and the second housing; a flexible display (200) disposed on the first surface and the third surface; a first rear camera (310) and a second rear camera (320) disposed on the second surface; and a front camera (330) disposed on the fourth surface, wherein the method includes: Based on the received user input for capturing stereo images, a first image is obtained through a first rear camera, a second image is obtained through a second rear camera, and a third image is obtained through the front camera; Based on the first image and the second image, the left-eye image and the right-eye image of the stereoscopic image are obtained; and Based on the first image and the third image, the first depth information of the stereo image is obtained.
12. The method of operating the electronic device according to claim 11, wherein, The accuracy of the first depth information is higher than that of the second depth information obtained based on the first and second images.
13. The method of operating the electronic device according to any one of claims 11 to 12, further comprising: The data of the stereoscopic image is stored in the memory; and Based on the user input received for reproducing the stereoscopic image, the stereoscopic image is displayed through the flexible display using first depth information.
14. The method of operating the electronic device according to any one of claims 11 to 13, wherein, The steps for displaying the stereoscopic image include: Based on the received user input for reproducing the stereoscopic image, information about the stereoscopic image in three-dimensional space is generated using a first image, a second image, and first depth information; and The stereoscopic image is displayed using the information about three-dimensional space.
15. The method of operating the electronic device according to any one of claims 11 to 14, wherein, The steps for storing the data include: The data of the stereoscopic image, including the first depth information, is stored by synchronizing the first depth information with the first image and the second image.
