Method and device for processing image

Abstract

Provided are a method and an electronic device for processing an image. This operation method of the electronic device may include the steps of: acquiring a left eye image by capturing an object using a first camera and acquiring a right eye image by capturing the object using a second camera; adjusting resolutions with respect to a first direction of the left eye image and the right eye image on the basis of calculation performance information of a processor of the electronic device; and acquiring depth information from the resolution-adjusted left eye image and the resolution-adjusted right eye image.

Description

Method and device for processing images

[0001] The present disclosure relates to a method and device for processing images. Specifically, the present disclosure discloses a method and device for adjusting the resolution of an image to obtain depth information or a 3D image from a stereo image.

[0002] Recently, devices that include a depth sensor that obtains depth information representing the spatial sense of an object included in a real space composed of a three-dimensional stereoscopic space are widely used to provide users with a sense of immersion.

[0003] Augmented Reality (AR) is a technology that overlays virtual images onto real-world physical environments or objects. AR devices (e.g., smart glasses) utilizing AR technology are being used in everyday life for purposes such as information retrieval, route guidance, and camera photography. In particular, smart glasses can be worn as a fashion item and can be used for both indoor and outdoor activities.

[0004] Autonomous driving applications, robotic applications, and augmented reality applications running on augmented reality devices often require constant depth information. As the size and resolution of the image used to acquire depth information increase, the complexity of the depth information acquisition process increases, potentially increasing the load on the device. To reduce the complexity or load of the depth information acquisition process, the input image used to acquire depth information can be downscaled to a lower resolution.

[0005] Depth information acquired through low-resolution images may have at least one of lower resolution or lower resolution than depth information acquired through high-resolution images, requiring post-processing of the depth information to improve either resolution or resolution. For example, the resolution of depth information can be improved by utilizing high-resolution images as additional information through methods such as super-resolution. However, if depth information is estimated to be low-resolution, it may be difficult to improve it to high resolution.

[0006] One aspect of the present disclosure provides an electronic device for processing images. An electronic device according to one embodiment of the present disclosure may include a camera including a first camera and a second camera; a memory storing a program or at least one instruction; and at least one processor. By executing the program or at least one instruction stored in the memory by the at least one processor, the electronic device may acquire a left-eye image by photographing an object using the first camera, acquire a right-eye image by photographing the object using the second camera, adjust resolutions of the left-eye image and the right-eye image in a first direction based on computational performance information of the processor of the electronic device, and acquire depth information from the left-eye image with the adjusted resolution and the right-eye image with the adjusted resolution.

[0007] Another aspect of the present disclosure provides a method for processing an image by an electronic device. The method for processing an image according to one embodiment of the present disclosure may include: obtaining a left-eye image by photographing an object using a first camera, obtaining a right-eye image by photographing the object using a second camera, adjusting the resolution of the left-eye image and the right-eye image in a first direction based on computational performance information of a processor of the electronic device, and obtaining depth information from the left-eye image with the adjusted resolution and the right-eye image with the adjusted resolution.

[0008] Another aspect of the present disclosure may provide a computer-readable recording medium having recorded thereon a program for executing the method on a computer.

[0009] The present disclosure can be readily understood by the combination of the following detailed description and the accompanying drawings, wherein reference numerals refer to structural elements.

[0010] FIG. 1 is a diagram illustrating an electronic device processing an image according to an embodiment of the present disclosure.

[0011] FIG. 2 is a diagram illustrating an electronic device according to an embodiment of the present disclosure changing its resolution.

[0012] FIG. 3 is a diagram illustrating an electronic device according to an embodiment of the present disclosure changing its resolution.

[0013] FIG. 4 is a diagram illustrating an electronic device according to an embodiment of the present disclosure changing its resolution.

[0014] FIG. 5 is a diagram illustrating an electronic device according to an embodiment of the present disclosure changing its resolution.

[0015] FIG. 6 is a diagram illustrating an electronic device according to an embodiment of the present disclosure changing its resolution.

[0016] FIG. 7 is a diagram illustrating an electronic device according to an embodiment of the present disclosure changing its resolution.

[0017] FIG. 8 is a flowchart illustrating a method for processing an image according to an embodiment of the present disclosure.

[0018] FIG. 9 is a block diagram illustrating components of an electronic device according to an embodiment of the present disclosure.

[0019] The terms used in the embodiments of the specification have been selected from widely used general terms as much as possible while taking into account the functions of the present disclosure, but these may vary depending on the intention of a technician working in the field, precedents, the emergence of new technologies, etc.

[0020] Additionally, in certain cases, the applicant may arbitrarily select terms, and in such cases, their meanings will be described in detail in the description of the relevant embodiments. Therefore, the terms used in this specification should not be defined simply as names, but rather based on their inherent meaning and the overall content of the present disclosure.

[0021] Throughout this disclosure, whenever a part is said to "include" a certain component, this does not mean that it excludes other components, but rather that it may include other components, unless specifically stated otherwise.

[0022] It should be understood that the blocks and combinations of flowcharts in each flowchart can be executed by one or more computer programs containing computer-executable instructions. The one or more computer programs may be stored entirely in a single memory, or may be stored in separate portions across multiple different memories.

[0023] All functions or operations described in this document may be performed by a single processor or a combination of processors. A single processor or a combination of processors is a circuitry that performs processing, and may include circuitry such as an Application Processor (AP), a Communication Processor (CP), a Graphical Processing Unit (GPU), a Neural Processing Unit (NPU), a Microprocessor Unit (MPU), a System on Chip (SoC), or an Integrated Chip (IC).

[0024] In addition, terms such as "unit", "module", etc. described in this specification mean a unit that processes at least one function or operation, which may be implemented as hardware, software, or a combination of hardware and software. In the present disclosure, "module" means a unit that processes a function or operation performed by a processor, which may be implemented as software such as instructions, algorithms, data structures, or program codes.

[0025] The expression “configured to” used in the present disclosure can be used interchangeably with, for example, “suitable for”, “having the capacity to”, “designed to”, “adapted to”, “made to”, or “capable of”, depending on the context.

[0026] The term "configured (or set up) to" may not necessarily mean "specifically designed to" hardware. Instead, in some contexts, the phrase "a system configured to" may mean that the system, in conjunction with other devices or components, is "capable of" doing something.

[0027] For example, the phrase "a processor configured (or set) to perform A, B, and C" may include a dedicated processor (e.g., an embedded processor) for performing those operations, or a generic-purpose processor (e.g., a CPU or application processor) that enables the device to perform those operations by executing one or more program codes, instructions, or software stored in memory.

[0028] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art described herein.

[0029] In this disclosure, functions related to "artificial intelligence" are operated through a processor and memory. The processor may be comprised of one or more processors. In this case, one or more processors may be a general-purpose processor such as a CPU, AP, or DSP (Digital Signal Processor), a graphics-only processor such as a GPU or VPU (Vision Processing Unit), or an artificial intelligence-only processor such as an NPU.

[0030] One or more processors control the processing of input data according to predefined operating rules or artificial intelligence models stored in memory. Alternatively, if one or more processors are dedicated AI processors, the AI ​​processors may be designed with a hardware structure specialized for processing a specific AI model.

[0031] The predefined operation rules or artificial intelligence models are characterized by being created through learning. Here, "created through learning" means that the basic artificial intelligence model is trained using a learning algorithm using a large amount of learning data, thereby creating a predefined operation rules or artificial intelligence model configured to perform a desired characteristic (or purpose). This learning may be performed on the device itself on which the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and / or system.

[0032] Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0033] In the present disclosure, the "artificial intelligence model" may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and performs neural network operations through operations between the computational results of the previous layer and the multiple weight values.

[0034] The multiple weights of multiple neural network layers can be optimized based on the learning results of the AI ​​model. For example, during the learning process, the multiple weights can be updated to reduce or minimize the loss or cost values ​​obtained by the AI ​​model.

[0035] The artificial intelligence model may include a deep neural network (DNN), such as a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a transformer, attention, or deep Q-networks, but is not limited to the examples described above.

[0036] Additionally, when a component is referred to as being "connected" or "connected" to another component in the present disclosure, it should be understood that the component may be directly connected or connected to the other component, but may also be connected or connected via another component in between, unless otherwise specifically stated.

[0037] The electronic device of the present disclosure can be implemented as various electronic devices such as a laptop computer, a desktop, an e-book reader, a digital broadcasting terminal, a PDA (Personal Digital Assistants), a PMP (Portable Multimedia Player), a navigation device, an MP3 player, a camcorder, an IPTV (Internet Protocol Television), a DTV (Digital Television), a TV, a set-top box, a smart monitor, a tablet PC, a laptop, a digital signage, a large display, a 360-degree projector, a MS (Mobile Station), a vehicle, a satellite, an airborne vehicle, a cellular phone, a smart phone, a wearable device, etc.

[0038] In an embodiment of the present disclosure, the electronic device may be an augmented reality device. An 'augmented reality device' is a device capable of expressing augmented reality, and may be implemented as, for example, augmented reality glasses in the shape of glasses worn on the user's face. However, the augmented reality device is not limited thereto, and may also be implemented as a head-mounted display apparatus (HMD) worn on the user's head, an augmented reality helmet, or the like.

[0039] Augmented Reality (AR) is a technology that overlays virtual objects onto the physical environment or objects of the real world, presenting them together. Its advantage lies in the integration of virtual objects and information within the real world. AR devices (e.g., smart glasses) utilizing AR technology can be useful in everyday life, for example, for information retrieval, route guidance, and camera photography. In particular, AR devices can be worn as fashion items and utilized for both indoor and outdoor activities.

[0040] The camera of the present disclosure is configured to acquire an image of an object by photographing the object in a real space. The camera may include a lens module, an image sensor, and an image processing module.

[0041] A camera can capture still images or moving images captured by an image sensor (e.g., CMOS or CCD). An image processing module can process still images or moving images captured by the image sensor, extract necessary information, and transmit the extracted information to a processor, where the images can be stored in memory.

[0042] Electronic devices can acquire depth information about real-world objects through images captured by a camera. Depth information may include depth maps, disparity maps, and other data, but these are merely examples and are not limited to the examples mentioned.

[0043] A disparity map may contain information about the disparity, i.e., the distance between pixels that are aligned with each other in the left and right eye images. A disparity map may include an image that expresses the horizontal displacement between a pixel and its corresponding pixel on a pixel-by-pixel basis.

[0044] A depth map is an image that contains depth values ​​that indicate how far away an object represented by a pixel in the image is from the camera. A depth map contains multiple pixels with depth values, and each of the pixels may have a depth value or distance value that is the same as or similar to the depth of a real-world object.

[0045] Resolution represents the ability to distinguish different depth values. Resolution can be expressed in terms of bitrate, the error range for a given depth value, or the units of depth value identification.

[0046] The higher the resolution, the more quantized values ​​can be represented between given depth values. For example, a case where depth values ​​are identified in 10 mm increments has greater resolution than a case where depth values ​​are identified in 100 mm increments.

[0047] Resolution can be expressed as the size of the image or the number of pixels contained in the image. For example, it can be expressed as (horizontal pixels x vertical pixels) or (horizontal pixels x vertical pixels), but these are only examples and are not limited to the examples mentioned.

[0048] While the present disclosure may illustrate an image processing method for convenience of explanation below, this is merely an example and can be applied to other graphical content, such as video or games. For example, the electronic device of the present disclosure can be applied to adjusting the resolution of video or other graphical content.

[0049] In the attached drawings, some components are exaggerated, omitted, or schematically depicted. Furthermore, the dimensions of each component do not entirely reflect its actual size. Furthermore, the numbers used throughout the description (e.g., "First," "Second," etc.) are merely identifiers used to distinguish one component from another.

[0050] Below, embodiments of the present disclosure are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.

[0051] FIG. 1 is a diagram illustrating an electronic device processing an image according to an embodiment of the present disclosure.

[0052] Referring to FIG. 1, the electronic device (100) may include a first camera (111) and a second camera (113). The first camera (111) and the second camera (113) may be configured as stereo cameras.

[0053] The first camera (111) and the second camera (113) may represent binocular cameras, such as a left-eye camera and a right-eye camera. However, this is merely an example, and the first camera (111) and the second camera (113) may represent cameras positioned or arranged in various directions, such as up and down, left and right, or diagonally, on the electronic device (100).

[0054] The electronic device (100) can acquire stereo images (10L, 10R) by photographing an object using a first camera (111) and a second camera (113). The stereo images (10L, 10R) can include binocular images, for example, a left-eye image (10L) of an object photographed using the first camera (111) and a right-eye image (10R) of the object photographed using the second camera (113).

[0055] Although FIG. 1 illustrates an electronic device (100) as including two cameras, including a first camera (111) and a second camera (113), the present disclosure is not limited thereto. In one embodiment of the present disclosure, the electronic device (100) may not include a camera, the electronic device (100) may include only one camera, or the electronic device (100) may include three or more multi-cameras.

[0056] In the embodiment illustrated in FIG. 1, the electronic device (100) is illustrated as acquiring a left eye image (10L) and a right eye image (10R) using a first camera (111) and a second camera (113), but the present disclosure is not limited thereto.

[0057] The electronic device (100) may also receive stereo images (10L, 10R) captured by a camera other than the camera (110, see FIG. 4) of the electronic device (100) from another electronic device. However, this is only an example, and the electronic device (100) may obtain stereo images (10L, 10R) through other methods.

[0058] In one embodiment of the present disclosure, the left eye image (10L) and the right eye image (10R) may be acquired in advance and stored in the image storage (147). The electronic device (100) may acquire the left eye image (10L) and the right eye image (10R) from the image storage (147).

[0059] Hereinafter, the resolution change module (141), the depth information acquisition module (143), and the resolution enhancement module (145) may represent software modules composed of commands or program codes that process the functions and / or operations of the electronic device (100).

[0060] Descriptions of the functions and / or operations of the resolution change module (141), the depth information acquisition module (143), and the resolution enhancement module (145) can be performed by the processor (130, see FIG. 9) of the electronic device (100) executing software such as commands or program codes of the modules.

[0061] The resolution change module (141) can obtain a stereo image (20L, 20R) with an adjusted resolution from a stereo image (10L, 10R). The stereo image (20L, 20R) with an adjusted resolution can include binocular images, for example, a left-eye image (20L) with an adjusted resolution and a right-eye image (20R) with an adjusted resolution.

[0062] A resolution change module (141) may be included in the electronic device (100). The resolution change module (141) receives a left-eye image (10L) from the first camera (111), receives a right-eye image (10R) from the second camera (113), and may adjust the resolution of the left-eye image (10L) and the right-eye image (10R). The resolution change module (141) may transmit the stereo images (20L, 20R) with adjusted resolution to the depth information acquisition module (143) or to another device including the depth information acquisition module (143).

[0063] The resolution may include at least one of horizontal resolution and vertical resolution. In the present disclosure, horizontal may refer to a case where the resolution is parallel to or coincides with the baseline between the first camera (111) and the second camera (113), and vertical may refer to a case where the resolution is perpendicular to the straight line connecting the first camera (111) and the second camera (113).

[0064] The resolution change module (141) can adjust the resolution of the left-eye image (10L) and the right-eye image (10R) in the first direction based on the computational performance information of the processor (130) of the electronic device (100). The resolution in the first direction can represent the vertical resolution. For example, the resolution change module (141) can adjust the resolution of the left-eye image (10L) and the right-eye image (10R) in the first direction from H to H. s can be changed. In one embodiment of the present disclosure, the resolution for the second direction of the left eye image (10L) and the right eye image (10R) may be maintained.

[0065] The processor (130) may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), a Tensor Processing Unit (TPU), a Neural Processing Unit (NPU), an application processor (AP), a Digital Signal Processor (DSP), a Vision Processing Unit (VPU), a Microprocessor Unit (MPU), a System on Chip (SoC), an Integrated Chip (IC), or an AI Accelerator, which are merely examples and are not limited to the examples mentioned. The processor (130) may be used to acquire depth information.

[0066] For the convenience of explanation, one processor, such as AP, may be used as an example, but this is only an example, and the same applies to cases involving other processors.

[0067] For convenience of explanation, the computational performance information of the processor (130) may be explained using one processor information, such as FLOPS, as an example, but this is only an example and may be applied to other processor information as well.

[0068] The computational performance information of the processor can be identified based on at least one of computational performance, number of cores, cache capacity, MIPS (Million Instructions Per Second), FLOPS (Floating-point Operations Per Second), IPC (Instructions Per Cycle), CPI (Cycles Per Instruction), thread count, instruction set, clock speed, benchmark score or benchmark result, power consumption while the processor is operating, energy efficiency, performance per watt ratio of power consumed compared to the amount of work that can be processed, TDP (Thermal Design Power), active efficiency, idle efficiency, processor architecture, memory bandwidth, processor generation, memory type and capacity, computational precision, unit of processor manufacturing process, manufacturer, model, purpose, or cache hit rate. However, this is only an example and the computational performance information of the processor is not limited to the examples mentioned.

[0069] The resolution change module (141) can obtain processor information through user input or obtain processor information that is preset or stored in the electronic device (100). The resolution change module (141) can change the resolution (H) for the first direction to H based on the obtained processor information. s It can be determined to adjust to H. The resolution change module (141) determines that the resolution (H) for the first direction is Hs A left eye image (20L) with an adjusted resolution and a right eye image (20R) with an adjusted resolution can be obtained.

[0070] For example, if the computational performance of the processor (1600 FLOPS) is received as input, the resolution change module (141) determines the vertical size Hs=H*k ( ), or at least one of the vertical resize ratios k can be identified. The resolution change module (141) can obtain a left eye image (20L) with the resolution adjusted to the identified vertical size Hs and a right eye image (20R) with the resolution adjusted.

[0071] The resolution change module (141) can identify the power status of the electronic device (100). The resolution change module (141) can adjust the resolution of the left-eye image (10L) and the right-eye image (10R) based on the power status. The power status can indicate information about at least one of the power consumption of the electronic device (100), the power consumption, the remaining battery level, whether the electronic device (100) is charging, or whether power consumption reduction is necessary. In one embodiment of the present disclosure, the resolution of the second direction of the left-eye image (10L) and the right-eye image (10R) may be maintained.

[0072] For example, if the remaining battery level of the electronic device (100) is below a threshold level, if the electronic device (100) is not charging, if the power consumption of the processor (130) is above a threshold level, or if the power consumption of the electronic device (100) needs to be reduced, the resolution change module (141) changes the vertical size Hs=H*k ( ) can obtain a left eye image (20L) with an adjusted resolution and a right eye image (20R) with an adjusted resolution.

[0073] The resolution change module (141) can adjust the resolution of the second direction of the left eye image (10L) and the right eye image (10R) based on the required resolution for the depth information. The resolution of the second direction can represent the horizontal resolution. For example, the resolution change module (141) can adjust the resolution of the second direction of the left eye image (10L) and the right eye image (10R) from W to W. s can be changed to

[0074] The required resolution for depth information indicates the degree to which different depth values ​​must be distinguished. The required resolution can be expressed in terms of bitrate, the error range for a given depth value, or the unit of depth value identification.

[0075] The required resolution for depth information may include at least one of a reference depth required by the system, a reference depth-related resolution required by the system, a reference depth required by the user, or a reference depth-related resolution required by the user.

[0076] For example, if the required resolution is expressed as a reference depth x (mm) and a resolution y (mm) for the reference depth required by the system or application, then the pixel (p) corresponding to the location at depth x (mm) x ) and the pixel (p) corresponding to the position with depth x+y (mm) x+y ) can represent different depth values ​​in this depth map.

[0077] The resolution change module (141) can obtain the required resolution for depth information through user input, or obtain the required resolution information for depth information that is preset or stored in the electronic device (100). The resolution change module (141) can change the resolution (W) for the second direction based on the required resolution. s It can be determined to adjust to the resolution change module (141). The resolution change module (141) determines the resolution (W) for the identified second direction.s ) can obtain a left eye image (20L) with an adjusted resolution and a right eye image (20R) with an adjusted resolution.

[0078] For example, when the horizontal scaling ratio of the image is r, the reference depth is x (mm), the resolution required for the reference depth is y (mm), the focal length is f (pixel), and the distance between the first camera (111) and the second camera (113) is b (mm), the disparity when x mm Disparity in (pixels) and x+y mm (Pixel) can be identified based on [Mathematical Formula 1] and [Mathematical Formula 2].

[0079] [Mathematical Formula 1]

[0080]

[0081] [Equation 2]

[0082]

[0083] When a resolution y mm is required for a reference depth x mm, the horizontal scaling ratio r can be identified based on [Mathematical Formula 3].

[0084] [Equation 3]

[0085]

[0086] Resolution change module (141) Changes the horizontal size W of the stereo image according to the horizontal size adjustment ratio r. s =W*r (r ), a left eye image (20L) with adjusted resolution and a right eye image (20R) with adjusted resolution can be obtained.

[0087] In one embodiment of the present disclosure, the resolution change module (141) adjusts the horizontal size ratio r, the horizontal size W s, or at least one of the images with changed horizontal resolution can be received as input.

[0088] Resolution change module (141) horizontal direction size adjustment ratio r, horizontal direction size W s , or at least one image with a changed horizontal resolution is received as input, the resolution change module (141) can identify a vertical resolution to be adjusted using processor information. The resolution change module (141) can change the vertical resolution of the image to the identified vertical resolution.

[0089] Processor information, horizontal dimension W s , vertical size H s The resolution change module (141) can identify a vertical size according to the input from a data structure to which at least one of the inputs is mapped. The resolution change module (141) can obtain a left-eye image (20L) with a resolution adjusted to the identified vertical size and a right-eye image (20R) with a resolution adjusted to the identified vertical size.

[0090] For example, processor information such as a table in which the processor's computational performance, horizontal size, and vertical size are mapped, as in [Table 1], may be set or stored in the electronic device (100). When the resolution change module (141) receives the processor's computational performance (1600 FLOPS) and horizontal size (W) as input, the resolution change module (141) can identify a vertical size (H / 4) that satisfies the horizontal size or more and the computational performance.

[0091] [Table 1]

[0092]

[0093] For convenience of explanation, a lookup table (LUT) is used as an example, but this is only an example, and the resolution change module (141) can identify the vertical size according to processor information by utilizing a method such as an array, a hash table, a tree structure, a map or dictionary, an index, or a deep learning model.

[0094] The resolution change module (141) changes the vertical size H of the stereo image as Hs=H*k ( ), a stereo image (20L, 20R) with adjusted resolution can be obtained.

[0095] The resolution change module (141) can identify an application being executed by the electronic device (100). The resolution change module (141) can adjust the resolution of the left eye image (10L) and the right eye image (10R) according to the characteristics of the executed application.

[0096] The resolution change module (141) can identify information regarding the depth information acquisition time corresponding to the characteristics of the running application. The characteristics of the application can indicate at least one of the following: the importance of real-time or fast processing operations, the computational speed required by the application, the urgency, the resolution required by the application, or whether object tracking is required.

[0097] The resolution change module (141) can adjust the resolution of the left eye image (10L) and the right eye image (10R) based on the corresponding depth information acquisition times. Information about the depth information acquisition times can include at least one of the depth information acquisition time required by the application, the driving speed required by the application, whether the depth information acquisition time needs to be reduced, or whether the depth information acquisition time required by the application is below a threshold value. The driving speed can be related to the depth map acquisition time, and the shorter the depth map acquisition time, the faster the driving speed.

[0098] The resolution change module (141) changes the vertical size H of the stereo image to be less than the required depth information acquisition time by Hs=H*k ( ), the resolution change module (141) can obtain a stereo image (20L, 20R) with adjusted resolution. In one embodiment of the present disclosure, the resolution for the second direction of the left eye image (10L) and the right eye image (10R) may be maintained.

[0099] The resolution change module (141) can identify the driving speed required by the identified application. The resolution change module (141) can adjust the resolution of the left eye image (10L) and the right eye image (10R) based on the identified driving speed.

[0100] If the driving speed required by the running application is higher than a certain threshold value, if the driving speed required by the running application is faster than the driving speed required by the previously running application, or if the time required to acquire depth information in the running application is less than a threshold time or the time needs to be shortened, the resolution change module (141) changes the vertical size H of the stereo image by Hs=H*k ( ), a stereo image (20L, 20R) with adjusted resolution can be obtained. In one embodiment of the present disclosure, the resolution for the second direction of the left eye image (10L) and the right eye image (10R) may be maintained.

[0101] If a reduction in the acquisition time of depth information is required in the running application, the resolution change module (141) changes the vertical size H of the stereo image (10L, 10R) to Hs=H*k ( ), a stereo image (20L, 20R) with adjusted resolution can be obtained. In one embodiment of the present disclosure, the resolution for the second direction of the left eye image (10L) and the right eye image (10R) may be maintained.

[0102] For example, when a person video application is executed, when an application requiring object tracking is executed, when a traffic application or autonomous driving application providing location information or traffic information is executed, when a communication application or game application that allows real-time interaction with other users is executed, when a medical application such as medical image generation is executed, or when a security application that detects and analyzes movement in a specific area is executed, etc., the resolution change module (141) can obtain a stereo image (20L, 20R) with an adjusted resolution by changing the vertical size H of the stereo image (10L, 10R) to Hs=H*k (k≤1). In one embodiment of the present disclosure, the resolution for the second direction of the left eye image (10L) and the right eye image (10R) may be maintained.

[0103] The resolution change module (141) can identify changes in the application of the electronic device (100). The resolution change module (141) can obtain stereo images (20L, 20R) with adjusted resolution based on the changes in the application.

[0104] When the execution of an application that can operate offline or does not require real-time operation changes to an application that requires real-time operation or an application where real-time operation is relatively important, the resolution change module (141) can acquire stereo images (20L, 20R) with adjusted vertical resolution. In one embodiment of the present disclosure, the resolution of the second direction of the left-eye image (10L) and the right-eye image (10R) may be maintained.

[0105] When changing from the execution of a spatial video application to the execution of a person video application, the resolution change module (141) can obtain a stereo image (20L, 20R) with adjusted resolution by changing the vertical size H of the stereo image (10L, 10R) to Hs=H*k (k≤1). In one embodiment of the present disclosure, the resolution of the second direction of the left eye image (10L) and the right eye image (10R) may be maintained.

[0106] Stereo images (20L, 20R) with adjusted resolution can be used to obtain depth information or 3D images.

[0107] The resolution change module (141) can identify the usage status of the processor (130). The resolution change module (141) can adjust the resolution of the left eye image and the right eye image based on the computational performance information of the processor (130) and the information regarding the usage status of the processor (130).

[0108] The resolution change module (141) can acquire a stereo image (20L, 20R) with an adjusted resolution by taking into account the processing performance information of the processor (130) and the usage status of the processor, such as the application usage status of the electronic device (100).

[0109] For example, if the computational performance of the processor available for acquiring depth information is below a critical level due to background execution or concurrent execution of another application, or if the computational performance allocable to acquiring depth information is below a critical level, the resolution change module (141) can acquire stereo images (20L, 20R) with adjusted vertical resolution.

[0110] As the resolution underlying depth information acquisition increases, the complexity of the depth information acquisition process increases, potentially increasing the load on the device. Furthermore, if horizontal resolution decreases, depth information may be estimated at a low resolution. If depth information is estimated at a low resolution, it may be difficult to improve it to a higher resolution.

[0111] The electronic device (100) according to an embodiment of the present disclosure can reduce instances of horizontal resolution degradation by adaptively changing vertical and horizontal resolutions. Accordingly, the electronic device (100) according to an embodiment of the present disclosure provides the technical effect of rapidly estimating depth information while reducing resolution loss.

[0112] In addition, the electronic device (100) according to the embodiment of the present disclosure can adaptively adjust the complexity of the depth information acquisition process and the load applied to the device by considering the performance of the processor, the usage status, and the type or characteristics of the application being executed.

[0113] Embodiments of the present disclosure can reduce the need to adopt and develop depth information estimation models depending on the performance of the processor, and enable the adoption and development of a single solution.

[0114] The depth information acquisition module (143) can acquire depth information (30) from a stereo image (20L, 20R) with adjusted resolution. The depth information (30) may include a depth map, a disparity map, etc., but this is only an example and is not limited to the examples mentioned.

[0115] The depth information acquisition module (143) can estimate depth information using the overall information of the image through global matching. The depth information acquisition module (143) can estimate depth information using methods such as belief propagation, dynamic programming, semi-global matching, and graph-cut, for example.

[0116] The depth information acquisition module (143) can estimate depth information using partial information of an image through local matching. The depth information acquisition module (143) can estimate depth information using at least one of feature-based local matching and region-based local matching. The depth information acquisition module (143) can estimate depth information using, for example, a method such as SAD (Sum of Absolute difference), SSD (Sum of Squared difference), NCC (Normalized Cross Correlation), or Census transform.

[0117] The depth information acquisition module (143) can acquire depth information using an artificial intelligence model trained to receive stereo images (10, 20) as input and output depth information. The artificial intelligence model can further include at least one of IGEV-Stereo (Iterative Geometry Encoding Volume-Stereo), CREStereo (Cascaded Recurrent Network with Adaptive Correlation), ACVNet (Attention Concatenation Volume), and GC-Net (Geometry and Context Network).

[0118] The stereo images (20L, 20R) with adjusted resolution can be used by the depth information acquisition module (143) to acquire depth information. In the present disclosure, for convenience of explanation, the depth information acquisition module (143) may be described as being included in the same electronic device (100) as the resolution change module (141). However, this is merely an example, and the depth information acquisition module (143) may be included in an electronic device (200, not shown) different from the electronic device (100) including the resolution change module (141).

[0119] For example, if a resolution change module (141) is included in an electronic device (100) and a depth information acquisition module (143) is included in another electronic device (200, not shown), the depth information acquisition module (143) can receive a stereo image (20L, 20R) with an adjusted resolution from the electronic device (100), and based on the stereo image (20L, 20R) with an adjusted resolution, the depth information acquisition module (143) can acquire depth information (30).

[0120] The resolution enhancement module (145) can increase the resolution of the depth information (30) to obtain depth information (40) with improved resolution. A super resolution model can be used for the resolution enhancement module (145) to obtain depth information (40) with improved resolution from the depth information (30).

[0121] The resolution enhancement module (145) estimates values ​​between pixels using a super resolution model that uses an interpolation method such as bicubic interpolation, thereby enabling the resolution enhancement module (145) to obtain depth information (40) with improved resolution from depth information (30).

[0122] A super-resolution model may include an artificial intelligence model trained to receive depth information (30) as input and output depth information (40) with enhanced resolution. The artificial intelligence model may further include at least one of a Super-Resolution Convolutional Neural Network (SRCNN), an Efficient Sub-Pixel Convolutional Neural Network (ESPCN), a Very Deep Super-Resolution (VDSR), a Super-Resolution Generative Adversarial Network (SRGAN), a Fast Super-Resolution Convolutional Neural Network (FSRCNN), or a Laplacian Pyramid Super-Resolution Network (LapSRN).

[0123] For example, the resolution enhancement module (145) increases the vertical size H of the depth information (30). and change the vertical size W to By changing to , it is possible to obtain depth information (40) with improved resolution.

[0124] FIG. 2 is a diagram illustrating an electronic device according to an embodiment of the present disclosure changing its resolution.

[0125] The electronic device (100) of the present disclosure may include a resolution change module (141). The resolution change module (141) includes a unit that processes a function of changing the resolution or an operation of changing the resolution performed by the processor (130), which may be implemented as software such as instructions, algorithms, data structures, or program codes and executed by the processor (130).

[0126] The resolution change module (141) can obtain a stereo image (20L, 20R) with adjusted resolution from a stereo image (10L, 10R).

[0127] The resolution change module (141) receives a left-eye image (10L) from the first camera (111), receives a right-eye image (10R) from the second camera (113), and can adjust the resolution of the left-eye image (10L) and the right-eye image (10R). The resolution change module (141) can transmit the stereo images (20L, 20R) with adjusted resolution to the depth information acquisition module or to a device including the depth information acquisition module.

[0128] The resolution may include at least one of horizontal resolution and vertical resolution. In the present disclosure, horizontal may refer to a case where the resolution is parallel to or coincides with the base line connecting the first camera (111) and the second camera (113), and vertical may refer to a case where the resolution is perpendicular to the line connecting the first camera (111) and the second camera (113).

[0129] The resolution change module (141) can adjust the resolution of the second direction of the left eye image (10L) and the right eye image (10R) based on the required resolution for the depth information. The resolution of the second direction can include the horizontal resolution. For example, the resolution change module (141) can adjust the resolution of the second direction of the left eye image (10L) and the right eye image (10R) from W to W. s can be changed to

[0130] The required resolution for depth information indicates the degree to which different depth values ​​must be distinguished. The required resolution can be expressed in terms of bitrate, the error range for a given depth value, or the unit of depth value identification.

[0131] The required resolution for depth information may include at least one of a reference depth required by the system, a reference depth-related resolution required by the system, a reference depth required by the user, or a reference depth-related resolution required by the user.

[0132] For example, if the required resolution is expressed as a reference depth x (mm) required by the system and the resolution y (mm) for the reference depth, then the pixel (p) corresponding to the location at depth x (mm) x ) and the pixel (p) corresponding to the position with depth x+y (mm) x+y ) can represent different depth values ​​in this depth map.

[0133] The resolution change module (141) can obtain the required resolution for depth information through user input, or obtain the required resolution for depth information set or stored in the electronic device (100). The resolution change module (141) can change the resolution (W) for the second direction based on the required resolution. s It can be determined to adjust to the resolution change module (141). The resolution change module (141) determines the resolution (W) for the identified second direction. s) can obtain a left eye image (20L) with an adjusted resolution and a right eye image (20R) with an adjusted resolution.

[0134] For example, when the horizontal scaling ratio of the image is r, the reference depth is x (mm), the resolution required for the reference depth is y (mm), the focal length is f (pixel), and the distance between the first camera (111) and the second camera (113) is b (mm), the disparity when x mm Disparity in (pixels) and x+y mm (Pixel) can be identified based on [Mathematical Formula 1] and [Mathematical Formula 2].

[0135] [Mathematical Formula 1]

[0136]

[0137] [Equation 2]

[0138]

[0139] When a resolution y mm is required for a reference depth x mm, the horizontal scaling ratio r can be identified based on [Mathematical Formula 3].

[0140] [Equation 3]

[0141]

[0142] The resolution change module (141) adjusts the horizontal size W of the stereo image according to the horizontal size adjustment ratio r. s =W*r (r ), a left eye image (20L) with adjusted resolution and a right eye image (20R) with adjusted resolution can be obtained.

[0143] FIG. 3 is a diagram illustrating an electronic device according to an embodiment of the present disclosure changing its resolution.

[0144] The resolution change module (141) can obtain a stereo image (20L, 20R) with an adjusted resolution from a stereo image (10L, 10R). The stereo image (20L, 20R) with an adjusted resolution can include binocular images, for example, a left-eye image (20L) with an adjusted resolution and a right-eye image (20R) with an adjusted resolution.

[0145] A resolution change module (141) may be included in the electronic device (100). The resolution change module (141) receives a left-eye image (10L) from the first camera (111), receives a right-eye image (10R) from the second camera (113), and may adjust the resolution of the left-eye image (10L) and the right-eye image (10R). The resolution change module (141) may transmit the stereo images (20L, 20R) with adjusted resolution to a depth information acquisition module or to another device including a depth information acquisition module.

[0146] The resolution may include at least one of horizontal resolution and vertical resolution. In the present disclosure, horizontal may refer to a case where the resolution is parallel to or coincides with the baseline between the first camera (111) and the second camera (113), and vertical may refer to a case where the resolution is perpendicular to the straight line connecting the first camera (111) and the second camera (113).

[0147] The resolution change module (141) can adjust the resolution of the left-eye image (10L) and the right-eye image (10R) in the first direction based on the computational performance information of the processor (130) of the electronic device (100). The resolution in the first direction can include the vertical resolution. For example, the resolution change module (141) can adjust the resolution of the left-eye image (10L) and the right-eye image (10R) in the first direction from H to H. scan be changed. In one embodiment of the present disclosure, the resolution for the second direction of the left eye image (10L) and the right eye image (10R) may be maintained.

[0148] The processor (130) may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), a Tensor Processing Unit (TPU), a Neural Processing Unit (NPU), an application processor (AP), a Digital Signal Processor (DSP), a Vision Processing Unit (VPU), a Microprocessor Unit (MPU), a System on Chip (SoC), an Integrated Chip (IC), or an AI Accelerator, which are merely examples and are not limited to the examples mentioned. The processor (130) may be used to acquire depth information.

[0149] For the convenience of explanation, one processor, such as AP, may be used as an example, but this is only an example, and the same applies to cases involving other processors.

[0150] For convenience of explanation, the computational performance information of the processor (130) may be explained using one processor information, such as FLOPS, as an example, but this is only an example and may be applied to other processor information as well.

[0151] The computational performance information of the processor can be identified based on at least one of computational performance, number of cores, cache capacity, MIPS (Million Instructions Per Second), FLOPS (Floating-point Operations Per Second), IPC (Instructions Per Cycle), CPI (Cycles Per Instruction), thread count, instruction set, clock speed, benchmark score or benchmark result, power consumption while the processor is operating, energy efficiency, performance per watt ratio of power consumed compared to the amount of work that can be processed, TDP (Thermal Design Power), active efficiency, idle efficiency, processor architecture, memory bandwidth, processor generation, memory type and capacity, computational precision, unit of processor manufacturing process, manufacturer, model, purpose, or cache hit rate. However, this is only an example and the computational performance information of the processor is not limited to the examples mentioned.

[0152] The resolution change module (141) can obtain processor information through user input or obtain processor information that is preset or stored in the electronic device (100). The resolution change module (141) can change the resolution (H) for the first direction to H based on the obtained processor information. s It can be determined to adjust to H. The resolution change module (141) determines that the resolution (H) for the first direction is Hs A left eye image (20L) with an adjusted resolution and a right eye image (20R) with an adjusted resolution can be obtained.

[0153] For example, if the computational performance of the processor (1600 FLOPS) is received as input, the resolution change module (141) determines the vertical size Hs=H*k ( ), or at least one of the vertical resize ratios k can be identified. The resolution change module (141) can obtain a left eye image (20L) with the resolution adjusted to the identified vertical size Hs and a right eye image (20R) with the resolution adjusted.

[0154] The resolution change module (141) can identify the power status of the electronic device (100). The resolution change module (141) can adjust the resolution of the left eye image (10L) and the right eye image (10R) based on the power status.

[0155] For example, if the remaining battery level of the electronic device (100) is below a threshold level, if the electronic device (100) is not charging, if the power consumption of the processor (130) is above a threshold level, or if the power consumption of the electronic device (100) needs to be reduced, the resolution change module (141) changes the vertical size Hs=H*k ( ) can obtain a left eye image (20L) with an adjusted resolution and a right eye image (20R) with an adjusted resolution.

[0156] FIG. 4 is a diagram illustrating an electronic device according to an embodiment of the present disclosure changing its resolution.

[0157] The resolution change module (141) can obtain a stereo image (20L, 20R) with an adjusted resolution from a stereo image (10L, 10R). The stereo image (20L, 20R) with an adjusted resolution can include binocular images, for example, a left-eye image (20L) with an adjusted resolution and a right-eye image (20R) with an adjusted resolution.

[0158] A resolution change module (141) may be included in the electronic device (100). The resolution change module (141) receives a left-eye image (10L) from the first camera (111), receives a right-eye image (10R) from the second camera (113), and can adjust the resolution of the left-eye image (10L) and the right-eye image (10R).

[0159] The resolution may include at least one of horizontal resolution and vertical resolution. In the present disclosure, horizontal may refer to a case where the resolution is parallel to or coincides with the baseline between the first camera (111) and the second camera (113), and vertical may refer to a case where the resolution is perpendicular to the straight line connecting the first camera (111) and the second camera (113).

[0160] The resolution change module (141) can identify an application being executed by the electronic device (100). The resolution change module (141) can adjust the resolution of the left-eye image (10L) and the right-eye image (10R) according to the characteristics of the executed application. The resolution change module (141) can transmit the stereo images (20L, 20R) with adjusted resolution to the depth information acquisition module or to another device including the depth information acquisition module.

[0161] The characteristics of an application may indicate the importance or urgency of real-time or fast processing operations, the resolution required by the application, or whether object tracking is required.

[0162] The resolution change module (141) can identify information regarding the depth information acquisition time corresponding to the characteristics of the running application. The resolution change module (141) can adjust the resolution of the left eye image (10L) and the right eye image (10R) based on the corresponding depth information acquisition time.

[0163] Information regarding the depth information acquisition time may include at least one of the depth information acquisition time required by the application, the operating speed required by the application, whether the depth information acquisition time must be less than or equal to a threshold time, or whether the depth information acquisition time needs to be reduced. The operating speed may be related to the depth map acquisition time, and the shorter the depth map acquisition time, the faster the operating speed.

[0164] The resolution change module (141) changes the vertical size H of the stereo image to be less than the required depth information acquisition time by Hs=H*k ( ), the resolution change module (141) can obtain a stereo image (20L, 20R) with adjusted resolution. In one embodiment of the present disclosure, the resolution for the second direction of the left eye image (10L) and the right eye image (10R) may be maintained.

[0165] The resolution change module (141) can identify the driving speed required by the identified application. The resolution change module (141) can adjust the resolution of the left eye image (10L) and the right eye image (10R) based on the identified driving speed.

[0166] If the driving speed required by the running application is higher than a certain threshold value, or if the driving speed required by the currently running application is faster than the driving speed required by the previously running application, the resolution change module (141) changes the vertical size H of the stereo image to Hs=H*k ( ), a stereo image (20L, 20R) with adjusted resolution can be obtained. In one embodiment of the present disclosure, the resolution for the second direction of the left eye image (10L) and the right eye image (10R) may be maintained.

[0167] If a reduction in the acquisition time of depth information is required in the running application, the resolution change module (141) changes the vertical size H of the stereo image (10L, 10R) to Hs=H*k ( ), a stereo image (20L, 20R) with adjusted resolution can be obtained. In one embodiment of the present disclosure, the resolution for the second direction of the left eye image (10L) and the right eye image (10R) may be maintained.

[0168] For example, when a person video application is executed, when an application requiring object tracking is executed, when a traffic application or autonomous driving application providing location information or traffic information is executed, when a communication application or game application that allows interaction with other users in real time is executed, when a medical application such as medical image generation is executed, or when a security application that detects and analyzes movement in a specific area is executed, etc., the resolution change module (141) can obtain a stereo image (20L, 20R) with an adjusted resolution by changing the vertical size H of the stereo image (10L, 10R) to Hs=H*k (k≤1).

[0169] The resolution change module (141) can identify changes in the application of the electronic device (100). The resolution change module (141) can obtain stereo images (20L, 20R) with adjusted resolution based on the changes in the application.

[0170] When the execution of an application that can operate offline or does not require real-time operation changes to an application that requires real-time operation or an application where real-time operation is relatively important, the resolution change module (141) can acquire stereo images (20L, 20R) with adjusted vertical resolution. In one embodiment of the present disclosure, the resolution of the second direction of the left-eye image (10L) and the right-eye image (10R) may be maintained.

[0171] When changing from the execution of a spatial video application to the execution of a person video application, the resolution change module (141) can obtain a stereo image (20L, 20R) with an adjusted resolution by changing the vertical size H of the stereo image (10L, 10R) to Hs=H*k (k≤1).

[0172] Stereo images (20L, 20R) with adjusted resolution can be used to obtain depth information or 3D images.

[0173] FIG. 5 is a diagram illustrating an electronic device according to an embodiment of the present disclosure changing its resolution.

[0174] The electronic device (100) can acquire depth information (30, see FIG. 1) based on stereo images (10L, 10R). The electronic device (100), for example, the resolution change module (141), can reduce the complexity of the depth information acquisition process and the load on the device by changing the stereo images (10L, 10R) into stereo images (20a, 20b) adjusted to a low resolution. The depth information may include a depth map, a disparity map, etc., but this is only an example and is not limited to the examples mentioned.

[0175] A disparity map may contain information about the disparity, i.e., the distance between pixels that are aligned with each other in the left and right eye images. A disparity map may include an image that expresses the horizontal displacement between a pixel and its corresponding pixel on a pixel-by-pixel basis.

[0176] A depth map is an image that contains depth values ​​that indicate how far away an object represented by a pixel in the image is from the camera. A depth map contains multiple pixels with depth values, and each of the pixels may have a depth value or distance value that is the same as or similar to the depth of a real-world object.

[0177] Referring to FIG. 5, the resolution change module (141) can obtain a stereo image (20a) adjusted to a low resolution, such as a horizontal resolution Ws=W / 2 and a vertical resolution Hs=H / 2. The stereo image (20a) adjusted to a low resolution can include a left-eye image (20L_a) with an adjusted resolution and a right-eye image (20R_a) with an adjusted resolution.

[0178] The resolution change module (141) can obtain a stereo image (20b) adjusted to a low resolution, such as a horizontal resolution Ws=W and a vertical resolution Hs=H / 2. The stereo image (20a) adjusted to a low resolution can include a left-eye image (20L_b) with an adjusted resolution and a right-eye image (20R_b) with an adjusted resolution.

[0179] For the stereo images (10L, 10R), the focal length f is 50 pixels, the distance between the binocular cameras (base line, b) is 18 mm, and the disparity d for pixel p1 p1 11 pixels, disparity d for pixel p2 p2 When (not shown) is 10 pixels, the depth value D can be obtained based on [Mathematical Formula 4].

[0180] [Equation 4]

[0181]

[0182] Depth value for pixel p1 from the image (20a) adjusted to low resolution ( ) is estimated, Depth information can be obtained as follows. The depth value for pixel p2 from the image (20a) adjusted to low resolution ( ) is estimated, Depth information can be obtained as follows.

[0183] Meanwhile, the depth value for pixel p1 from the image (20b) adjusted to low resolution ( ) is estimated, Depth information can be obtained as follows. The depth value for pixel p2 from the image (20b) adjusted to low resolution ( ) is estimated, Depth information can be obtained as follows.

[0184] When acquiring depth information from a low-resolution adjusted image (20a) having a relatively lower horizontal resolution than a low-resolution adjusted image (20b), the resolution of the depth information acquired from the low-resolution adjusted image (20a) may be lower than the resolution of the depth information acquired from the low-resolution adjusted image (20b).

[0185] An electronic device according to an embodiment of the present disclosure can reduce horizontal resolution degradation by adaptively changing vertical and horizontal resolutions. Accordingly, the embodiment of the present disclosure can estimate depth information at high speed while reducing resolution loss.

[0186] Additionally, the electronic device according to the embodiment of the present disclosure can adaptively adjust the complexity of the depth information acquisition process and the load applied to the device by considering the performance of the processor, the usage status, and the type or characteristics of the application being executed.

[0187] FIG. 6 is a diagram illustrating an electronic device according to an embodiment of the present disclosure changing its resolution.

[0188] The electronic device (100) of the present disclosure may include a resolution change module (141). The resolution change module (141) includes a unit that processes a function of changing the resolution or an operation of changing the resolution performed by the processor (130), which may be implemented as software such as instructions, algorithms, data structures, or program codes and executed by the processor (130).

[0189] The resolution change module (141) can adjust the resolution of stereo images (10L, 10R). The resolution change module (141) receives a left-eye image (10L) and a right-eye image (10R), and can adjust the resolution of the left-eye image (10L) and the right-eye image (10R).

[0190] The resolution change module (141) can adjust the resolution of the left eye image (10L) and the right eye image (10R) based on at least one of the required resolution for depth information, processor information, or device information. The resolution change module (141) can transmit the stereo image with the adjusted resolution to the depth information acquisition module or a device including the depth information acquisition module.

[0191] The resolution may include at least one of horizontal resolution and vertical resolution. In the present disclosure, horizontal may refer to a case where the resolution is parallel to or coincides with the baseline between the first camera and the second camera, and vertical may refer to a case where the resolution is perpendicular to the straight line connecting the first camera and the second camera.

[0192] For the convenience of the following explanation, an example is provided where the required resolution and computational performance differ by a factor of two. However, this is merely an example. Embodiments of the present disclosure are not limited to the examples mentioned.

[0193] Referring to FIG. 6, when the error range of the depth value corresponding to the required resolution 1 is half a time smaller than the error range of the depth value corresponding to the required resolution 2, the bit rate corresponding to the required resolution 1 is twice as large as the bit rate corresponding to the required resolution 2, the computational performance of processor 1 is twice as superior to the computational performance of processor 2, and the computational performance of device 1 is twice as superior to the computational performance of device 2, the resolution change module (141) can adaptively adjust the resolution of the left eye image (10L) and the right eye image (10R) in each case.

[0194] If the required resolution is 1, processor 1, and device 1, the resolution change module (141) changes the vertical resolution , stereo image A (21L, 21R) adjusted to horizontal resolution W can be obtained.

[0195] If the required resolution is 1, processor 2, and device 2, the resolution change module (141) changes the vertical resolution , stereo images B (23L, 23R) adjusted to horizontal resolution W can be acquired.

[0196] If the required resolution is 2, processor 1, and device 1, the resolution change module (141) has a vertical resolution H and a horizontal resolution A stereo image C (25L, 25R) adjusted to .

[0197] If the required resolution is 2, processor 2, and device 2, the resolution change module (141) changes the vertical resolution , horizontal resolution Stereo images D (27L, 27R) adjusted to .

[0198] When acquiring depth information from stereo image B (23L, 23R), the time required to acquire depth information can be shortened or the load on the device acquiring depth information can be reduced compared to when acquiring depth information from stereo image A (21L, 21R) or stereo image C (25L, 25R).

[0199] When acquiring depth information from stereo image C (25L, 25R) or stereo image D (27L, 27R) which have a relatively smaller horizontal resolution than stereo image A (21L, 21R), the resolution of the depth information acquired from stereo image C (25L, 25R) or stereo image D (27L, 27R) may be lower than the resolution of the depth information acquired from stereo image A (21L, 21R).

[0200] An electronic device according to an embodiment of the present disclosure can reduce horizontal resolution degradation by adaptively changing vertical and horizontal resolutions. Accordingly, the embodiment of the present disclosure can estimate depth information at high speed while reducing resolution loss.

[0201] FIG. 7 is a diagram illustrating an electronic device according to an embodiment of the present disclosure changing its resolution.

[0202] The electronic device (100) of the present disclosure may include a resolution change module (141). The resolution change module (141) includes a unit that processes a function of changing the resolution or an operation of changing the resolution performed by the processor (130), which may be implemented as software such as instructions, algorithms, data structures, or program codes and executed by the processor (130).

[0203] The resolution change module (141) can adjust the resolution of stereo images (10L, 10R). The resolution change module (141) receives a left-eye image (10L) and a right-eye image (10R), and can adjust the resolution of the left-eye image (10L) and the right-eye image (10R).

[0204] The resolution change module (141) can adjust the resolution of the left eye image (10L) and the right eye image (10R) based on at least one of the required resolution for depth information and the characteristics of the running application. The resolution change module (141) can transmit the stereo image with the adjusted resolution to the depth information acquisition module or a device including the depth information acquisition module.

[0205] The resolution may include at least one of horizontal resolution and vertical resolution. In the present disclosure, horizontal may refer to a case where the resolution is parallel to or coincides with the baseline between the first camera and the second camera, and vertical may refer to a case where the resolution is perpendicular to the straight line connecting the first camera and the second camera.

[0206] Referring to FIG. 7, when the error range of the depth value corresponding to the required resolution 1 is half a time smaller than the error range of the depth value corresponding to the required resolution 2, or the bit rate corresponding to the required resolution 1 is twice as large as the bit rate corresponding to the required resolution 2, and the time for acquiring depth information in application 2 needs to be shortened compared to application 1, or the load for the operation for acquiring depth information in application 2 needs to be reduced compared to application 1, the resolution change module (141) can adaptively adjust the resolution of the left eye image (10L) and the right eye image (10R) in each case.

[0207] Referring to FIG. 7, when real-time operation or fast processing is relatively more important for application 2 than for application 1, or the depth information acquisition time of application 2 is shorter than the depth information acquisition time of application 1, or the driving speed required by application 2 is faster than the driving speed required by application 1, or when object tracking is required in application 2, the electronic device can adaptively change the vertical resolution and the horizontal resolution.

[0208] (In case of required resolution 1, application 1), the resolution change module (141) can obtain a stereo image E (22L, 22R) adjusted to vertical resolution H and horizontal resolution W.

[0209] (Requested resolution 1, Application 2) If the resolution change module (141) is set to vertical resolution , stereo images F (24L, 24R) adjusted to horizontal resolution W can be acquired.

[0210] (Requested resolution 2, Application 1) In case of resolution change module (141), vertical resolution H, horizontal resolution A stereo image G (26L, 26R) adjusted to .

[0211] (Requested resolution 2, Application 2) If the resolution change module (141) is vertical resolution , horizontal resolution A stereo image H (28L, 28R) adjusted to .

[0212] When acquiring depth information from stereo image F (24L, 24R), the time required to acquire depth information can be shortened or the load on the device acquiring depth information can be reduced compared to when acquiring depth information from stereo image E (22L, 22R) or stereo image G (24L, 24R).

[0213] When acquiring depth information from a stereo image E (22L, 22R) or a stereo image G (26L, 26R) with a relatively smaller horizontal resolution, or from a stereo image H (28L, 28R), the resolution of the depth information acquired from the stereo image G (26L, 26R) or the stereo image H (28L, 28R) may be lower than the resolution of the depth information acquired from the stereo image E (22L, 22R) or the stereo image F (24L, 24R).

[0214] An electronic device according to an embodiment of the present disclosure can reduce horizontal resolution degradation by adaptively changing vertical and horizontal resolutions. Accordingly, the embodiment of the present disclosure can estimate depth information at high speed while reducing resolution loss.

[0215] FIG. 8 is a flowchart illustrating a method for processing an image according to an embodiment of the present disclosure.

[0216] In step S810, the electronic device (100) obtains a left eye image by photographing an object using a first camera, and obtains a right eye image by photographing the object using a second camera.

[0217] The electronic device (100) can acquire stereo images. The stereo images can include images captured by one or more cameras from different viewpoints.

[0218] The first camera and the second camera may be configured as stereo cameras. The first camera and the second camera may represent binocular cameras, such as a left-eye camera and a right-eye camera. However, this is merely an example, and the first camera (111) and the second camera (113) may represent cameras positioned or arranged in various directions, such as up and down, left and right, or diagonally, on the electronic device (100).

[0219] Although the electronic device (100) has been described as including two cameras, including a first camera and a second camera, the present disclosure is not limited thereto. In one embodiment of the present disclosure, the electronic device (100) may not include a camera, may include only one camera, or may include three or more multi-cameras. For example, the first camera and the second camera may represent a case where one camera takes pictures by changing the viewpoint.

[0220] Although the electronic device (100) has been described as acquiring a left eye image and a right eye image using the first camera and the second camera, the present disclosure is not limited thereto.

[0221] The electronic device (100) may also receive stereo images (10L, 10R) captured by a camera other than the camera of the electronic device (100) from another electronic device. However, this is only an example, and the electronic device (100) may obtain stereo images through other methods.

[0222] In one embodiment of the present disclosure, the left-eye image and the right-eye image may be acquired in advance and stored in the image storage. The electronic device (100) may acquire the left-eye image and the right-eye image from the image storage.

[0223] In step S820, the electronic device (100) may adjust the resolution of the left-eye image and the right-eye image based on the computational performance information of the processor of the electronic device. The resolution in the first direction may include the vertical resolution.

[0224] The processor may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), a Tensor Processing Unit (TPU), a Neural Processing Unit (NPU), an application processor (AP), a Digital Signal Processor (DSP), a Vision Processing Unit (VPU), a Microprocessor Unit (MPU), a System on Chip (SoC), an Integrated Chip (IC), or an AI Accelerator, but these are only examples and are not limited to the examples mentioned.

[0225] For convenience of explanation, the present disclosure may illustrate the computational performance information of a processor, such as FLOPS, as an example, but this is only an example and may be applied to other processor information as well.

[0226] The computational performance information of the processor can be identified based on at least one of computational performance, number of cores, cache capacity, MIPS (Million Instructions Per Second), FLOPS (Floating-point Operations Per Second), IPC (Instructions Per Cycle), CPI (Cycles Per Instruction), thread count, instruction set, clock speed, benchmark score or benchmark result, power consumption while the processor is operating, energy efficiency, performance per watt ratio of power consumed compared to the amount of work that can be processed, TDP (Thermal Design Power), active efficiency, idle efficiency, processor architecture, memory bandwidth, processor generation, memory type and capacity, computational precision, unit of processor manufacturing process, manufacturer, model, purpose, or cache hit rate. However, this is just an example and is not limited to the examples mentioned.

[0227] The electronic device (100) can obtain processor information through user input or obtain processor information that is preset or stored in the electronic device (100). The electronic device (100) can set the resolution (H) for the first direction to H based on the obtained processor information. s It can be decided to adjust it.

[0228] For example, if the computational performance of the processor (1600 FLOPS) is received as input, the resolution change module (141) may change the vertical size Hs, or the vertical resize ratio k ( ) can identify at least one of the following. The electronic device (100) can acquire a left-eye image and a right-eye image whose resolution is adjusted to the identified vertical size Hs=H*k. In one embodiment of the present disclosure, the resolution of the left-eye image (10L) and the right-eye image (10R) in the second direction may be maintained.

[0229] The electronic device (100) can identify the power status of the electronic device (100). The resolution of the left eye image and the right eye image can be adjusted based on the power status of the electronic device (100).

[0230] For example, if the power status of the electronic device (100) is below a critical value, if the electronic device (100) is not charging, or if power saving of the electronic device (100) is required, the resolution change module (141) changes the vertical size Hs=H*k ( ) can obtain a left eye image (20L) with an adjusted resolution and a right eye image (20R) with an adjusted resolution.

[0231] The electronic device (100) can adjust the resolution of the left-eye image and the right-eye image in the second direction based on the required resolution for the depth information. The resolution in the second direction can include the horizontal resolution. For example, the electronic device (100) can adjust the resolution of the left-eye image and the right-eye image in the second direction from W to W. s can be changed to

[0232] The required resolution for depth information indicates the degree to which different depth values ​​must be distinguished. The required resolution can be expressed in terms of bitrate, the error range for a given depth value, or the unit of depth value identification.

[0233] The electronic device (100) adjusts the horizontal size W of the stereo image according to a horizontal size adjustment ratio r. s W*r (r ), a left eye image with adjusted resolution and a right eye image with adjusted resolution can be obtained.

[0234] The electronic device (100) can identify an application being executed by the electronic device (100). The electronic device (100) can adjust the resolution of the left eye image and the right eye image according to the characteristics of the executed application.

[0235] The characteristics of an application may indicate whether it is an AR application, a robotics application, an autonomous driving application, a medical imaging application, the importance of real-time or fast processing operations, the urgency, the computational speed required by the application, the resolution required by the application, or whether object tracking is required.

[0236] The electronic device (100) can identify information regarding the depth information acquisition time corresponding to the characteristics of the running application. The electronic device (100) can adjust the resolution of the left-eye image and the right-eye image based on the corresponding depth information acquisition time.

[0237] Information about the depth information acquisition time may include at least one of a required depth information acquisition time, a required driving speed of the application, or whether a reduction in the depth information acquisition time is required.

[0238] The resolution change module (141) changes the vertical size H of the stereo image to be less than the required depth information acquisition time by Hs=H*k ( ), a stereo image (20L, 20R) with adjusted resolution can be obtained.

[0239] The resolution change module (141) can identify the driving speed required by the identified application. The driving speed can be related to the depth map acquisition time, and the shorter the depth map acquisition time, the faster the driving speed. The resolution change module (141) can adjust the resolution of the left-eye image (10L) and the right-eye image (10R) based on the identified driving speed.

[0240] When the driving speed required by the running application is higher than a certain threshold value, when the depth information acquisition time required by the running application is required to be reduced, or when the driving speed required by the currently running application is required to be faster than the driving speed required by the previously running application, the electronic device (100) sets the vertical size H of the stereo image to Hs=H*k ( ), a stereo image (20L, 20R) with adjusted resolution can be obtained. In one embodiment of the present disclosure, the resolution for the second direction of the left eye image (10L) and the right eye image (10R) may be maintained.

[0241] If a reduction in the acquisition time of depth information is required in a running application, the electronic device (100) may set the vertical size H of the stereo image as Hs=H*k ( ), a stereo image (20L, 20R) with adjusted resolution can be obtained. In one embodiment of the present disclosure, the resolution for the second direction of the left eye image (10L) and the right eye image (10R) may be maintained.

[0242] The electronic device (100) can identify the usage status of the processor (130). The resolution change module (141) can adjust the resolution of the left eye image and the right eye image based on the computational performance information of the processor (130) and the information regarding the usage status of the processor (130).

[0243] The electronic device (100) can acquire a stereo image (20L, 20R) with an adjusted resolution by taking into account the processing performance information of the processor (130) and the usage status of the processor, such as the application usage status of the electronic device (100).

[0244] For example, if the computational performance of the processor available for acquiring depth information is below a critical level due to background execution or concurrent execution of another application, or if the computational performance allocable to acquiring depth information is below a critical level, the resolution change module (141) can acquire stereo images (20L, 20R) with adjusted vertical resolution.

[0245] In step S830, the electronic device (100) obtains depth information from a left eye image with an adjusted resolution and a right eye image with an adjusted resolution.

[0246] Depth information may include a depth map, a disparity map, etc., but these are only examples and are not limited to the examples mentioned.

[0247] The electronic device (100) can estimate depth information using the overall information of the image through global matching. The electronic device (100) can estimate depth information using methods such as belief propagation, dynamic programming, semi-global matching, and graph-cut, for example.

[0248] The electronic device (100) can estimate depth information using partial information of an image through local matching. The electronic device (100) can estimate depth information using at least one of feature-based local matching and region-based local matching. The electronic device (100) can estimate depth information using, for example, a method such as Sum of Absolute Difference (SAD), Sum of Squared Difference (SSD), Normalized Cross Correlation (NCC), or Census Transform.

[0249] An electronic device (100) can acquire depth information using an artificial intelligence model trained to receive stereo images as input and output depth information. The artificial intelligence model may further include at least one of IGEV-Stereo (Iterative Geometry Encoding Volume-Stereo), CREStereo (Cascaded Recurrent Network with Adaptive Correlation), ACVNet (Attention Concatenation Volume), and GC-Net (Geometry and Context Network).

[0250] The electronic device (100) can obtain depth information with improved resolution from the acquired depth information.

[0251] A super resolution model can be used to obtain depth information with improved resolution from depth information by an electronic device (100).

[0252] The electronic device (100) can obtain depth information with improved resolution from depth information by estimating values ​​between pixels using a super resolution model that uses an interpolation method such as bicubic interpolation.

[0253] A super-resolution model may include an artificial intelligence model trained to receive depth information as input and output depth information with enhanced resolution. The artificial intelligence model may further include at least one of a Super-Resolution Convolutional Neural Network (SRCNN), an Efficient Sub-Pixel Convolutional Neural Network (ESPCN), a Very Deep Super-Resolution (VDSR), a Super-Resolution Generative Adversarial Network (SRGAN), a Fast Super-Resolution Convolutional Neural Network (FSRCNN), or a Laplacian Pyramid Super-Resolution Network (LapSRN).

[0254] For example, the electronic device (100) may have a vertical size H of depth information as Ht=H*a (1 a) and change the vertical size W to Wt=W*b (1 By changing to b), depth information with improved resolution can be obtained.

[0255] FIG. 9 is a block diagram illustrating components of an electronic device (100) according to an embodiment of the present disclosure.

[0256] The electronic device (100) can be implemented as various electronic devices such as a laptop computer, a desktop, an e-book reader, a digital broadcasting terminal, a PDA (Personal Digital Assistants), a PMP (Portable Multimedia Player), a navigation device, an MP3 player, a camcorder, an IPTV (Internet Protocol Television), a DTV (Digital Television), a TV, a set-top box, a smart monitor, a tablet PC, a laptop, a digital signage, a large display, a 360-degree projector, a MS (Mobile Station), a vehicle, a satellite, an airborne vehicle, a cellular phone, a smart phone, a wearable device, etc.

[0257] The electronic device (100) may be an augmented reality device. An 'augmented reality device' is a device capable of expressing augmented reality, and may be implemented as, for example, augmented reality glasses in the shape of glasses that a user wears on the face. However, the present invention is not limited thereto, and the augmented reality device may also be implemented as a head-mounted display apparatus (HMD) that is worn on the user's head, an augmented reality helmet, or the like.

[0258] Referring to FIG. 9, the electronic device (100) may include a camera (110), a processor (130), a memory (140), and a display unit (150). The camera (110), the processor (130), the memory (140), and the display unit (150) may each be electrically and / or physically connected to each other.

[0259] The components included in the electronic device (100) are not limited to those illustrated in FIG. 9. In one embodiment of the present disclosure, the electronic device (100) may not include at least one of the camera (110) or the display unit (150). In one embodiment of the present disclosure, the augmented reality device (100) may further include a gaze tracking sensor or a communication unit.

[0260] The camera (110) is configured to capture an image of an object in a real space by photographing the object. In one embodiment of the present disclosure, the camera (110) may include a lens module, an image sensor, and an image processing module. The camera (110) may capture still images or moving images obtained by an image sensor (e.g., a CMOS or CCD). The image processing module may process the still images or moving images captured by the image sensor, extract necessary information, and transmit the extracted information to the processor (130).

[0261] In one embodiment of the present disclosure, the camera (110) may include a first camera (111) and a second camera. The first camera (111) and the second camera (113) may be configured as stereo cameras. The first camera (111) and the second camera (113) may represent binocular cameras, such as a left-eye camera and a right-eye camera. However, this is merely an example, and the first camera (111) and the second camera (113) may represent cameras positioned or arranged in various directions, such as up and down, left and right, and diagonally, in the electronic device (100).

[0262] Although FIG. 9 illustrates an electronic device (100) including two cameras, including a first camera (111) and a second camera (113), the present disclosure is not limited thereto.

[0263] In one embodiment of the present disclosure, the electronic device (100) may not include a camera, the electronic device (100) may include only one camera, or the electronic device (100) may include three or more multi-cameras.

[0264] In an embodiment of the present disclosure, the electronic device of the present disclosure can acquire stereo images by utilizing changes in the viewpoint of a camera, such as movement or rotation of one camera. For example, the first camera (111) and the second camera (113) may represent a case where one camera takes pictures by changing the viewpoint.

[0265] The electronic device (100) can obtain a stereo image by photographing an object using the first camera (111) and the second camera (113). The stereo image can include binocular images, for example, a left-eye image of the object photographed using the first camera (111) and a right-eye image of the object photographed using the second camera (113).

[0266] The processor (130) can execute one or more instructions of a program stored in the memory (140). The processor (130) can be composed of hardware components that perform arithmetic, logic, and input / output operations and signal processing.

[0267] The processor (130) may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), a Tensor Processing Unit (TPU), a Neural Processing Unit (NPU), an application processor (AP), a Digital Signal Processor (DSP), a Vision Processing Unit (VPU), a Microprocessor Unit (MPU), a System on Chip (SoC), an Integrated Chip (IC), a Digital Signal Processing Devices (SPDs), a Programmable Logic Devices (PLDs), a Field Programmable Gate Arrays (FPGAs), or an AI Accelerator, but this is only an example and is not limited to the examples mentioned.

[0268] In one embodiment of the present disclosure, the processor (130) may include an AI processor or AI accelerator that performs artificial intelligence (AI) learning. The AI ​​processor may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or may be manufactured as part of an existing general-purpose processor (e.g., CPU or application processor) or a graphics-only processor (e.g., GPU) and mounted on the electronic device (100).

[0269] Although the processor (130) is illustrated as a single element in FIG. 9, it is not limited thereto. In one embodiment of the present disclosure, the processor (130) may be composed of one or more elements.

[0270] The processor (130) may include various processing circuits, including at least one processor. One or more of the processors may be configured to perform various functions described in the present disclosure, either individually or collectively, in a distributed manner. For example, the processor (130) may be configured to perform various functions described in FIGS. 1 to 9 .

[0271] "Processor," "at least one processor," and "one or more processors" may be configured to perform multiple functions. However, these terms encompass, without limitation, situations where one processor performs some of the functions and other processors perform other parts of the functions, and situations where a single processor can perform all of the functions. Furthermore, the at least one processor may comprise a combination of processors that perform various functions of the disclosed functions in a distributed manner. At least one processor may execute program instructions to achieve or perform various functions.

[0272] For example, a first processor may perform A and B, and a second processor may perform C. Alternatively, the first processor may be configured to perform a portion of A, and the second processor may be configured to perform the remaining portion of A, B, and C.

[0273] The memory (140) may be configured as at least one type of storage medium among, for example, a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), an HBM (High Bandwidth Memory), or an optical disk, and this is only an example and is not limited to the examples mentioned.

[0274] The memory (140) may store instructions related to functions or operations of the electronic device (100) to obtain depth value information of an object. In one embodiment of the present disclosure, the memory (140) may store at least one of instructions, an algorithm, a data structure, a program code, and an application program that can be read by the processor (130). The instructions, algorithms, data structures, and program codes stored in the memory (140) may be implemented in a programming or scripting language such as, for example, C, C++, Java, assembler, Python, etc.

[0275] The memory (140) may store instructions, algorithms, data structures, or program codes related to the resolution change module (141), the depth information acquisition module (143), and the resolution enhancement module (145). The 'module' included in the memory (140) refers to a unit that processes a function or operation performed by the processor (130), and this may be implemented as software such as instructions, algorithms, data structures, or program codes.

[0276] In the embodiment of the present disclosure, the processor (130) may be implemented to cause the electronic device (100) to perform a predetermined function by executing instructions or program codes stored in the memory (140).

[0277] The resolution change module (141) is configured with commands or program codes that execute functions and / or operations for adjusting the resolution of the left-eye image and the right-eye image. The processor (130) can adjust the resolution of the left-eye image and the right-eye image by executing the commands or program codes that constitute the resolution change module (141).

[0278] The depth information acquisition module (143) is configured with commands or program codes that execute functions and / or operations for acquiring depth information from left-eye images and right-eye images. The processor (130) can acquire depth information from left-eye images and right-eye images by executing the commands or program codes of the depth information acquisition module (143).

[0279] The resolution enhancement module (145) is configured with instructions or program codes that execute functions and / or operations that enhance the resolution of depth information or spatial images (e.g., 3D images). The processor (130) can enhance the resolution of depth information or spatial images (e.g., 3D images) by executing the instructions or program codes of the resolution enhancement module (145).

[0280] The image storage (147) can store at least one of an image acquired from a camera, an image received from another device, an image whose resolution has been adjusted through a resolution change module (143), depth information acquired by a depth information acquisition module (143), depth information with improved resolution, or a spatial image with improved resolution.

[0281] The display unit (150) outputs information processed in the electronic device (100). For example, the display unit (150) may display a user interface for photographing the surroundings of the electronic device (100) and information related to services provided based on captured images of the surroundings of the electronic device (100).

[0282] The electronic device (100), according to one embodiment, may provide an AR (Augmented Reality) image through a display unit (150). The display unit (150) according to one embodiment may include a wave guide (not shown) and a display module (not shown). The wave guide (not shown) may be made of a transparent material through which a portion of the back surface is visible when the user wears the electronic device (100). The wave guide (not shown) may be made of a single-layer or multi-layer flat plate made of a transparent material through which light may be reflected and propagated internally. The wave guide (not shown) may receive light of a virtual image projected against the emission surface of the display module. Here, the transparent material means a material through which light may pass, and the transparency may not be 100% and may have a predetermined color. In one embodiment, since the wave guide (not shown) is formed of a transparent material, the user can view not only virtual objects of a virtual image through the display (150), but also an external actual scene, and thus the wave guide (not shown) may be referred to as a see-through display. The display (150) may provide an augmented reality image by outputting virtual objects of a virtual image through the wave guide.

[0283] The electronic device (100) may further include a user input unit (not shown). The user input unit (not shown) refers to a means for a user to input data for controlling the electronic device (100).

[0284] For example, the user input unit (not shown) may include, but is not limited to, at least one of a key pad, a dome switch, a touch pad (contact capacitance type, pressure resistive film type, infrared sensing type, surface ultrasonic conduction type, integral tension measurement type, piezo effect type, etc.), a jog wheel, or a jog switch.

[0285] A user input unit (not shown) can capture images of the surroundings of an electronic device (100) using a camera module (110) and receive user input for providing services from the electronic device (100) or a server (not shown) based on the captured images. For example, the processor (130) can recognize a user's gesture by executing a gesture recognition module (not shown) stored in a memory (140).

[0286] The electronic device (100) may further include a microphone (not shown). The microphone (not shown) receives an external audio signal and processes it into electrical voice data. For example, the microphone (not shown) may receive an audio signal from an external device or a speaker. The microphone (not shown) may utilize various noise removal algorithms to remove noise generated during the process of receiving an external audio signal. The microphone (not shown) may receive a user's voice input for controlling the electronic device (100).

[0287] According to an embodiment of the present disclosure, the electronic device (100) can acquire a left-eye image by photographing an object using the first camera (111) and can acquire a right-eye image by photographing an object using the second camera (113) by executing a program or at least one instruction stored in a memory by at least one processor (130).

[0288] By having at least one processor (130) execute a program or at least one instruction stored in a memory (140), the electronic device (100) can adjust the resolution for the first direction of the left eye image and the right eye image based on the computational performance information of the processor of the electronic device.

[0289] By having at least one processor (130) execute a program or at least one instruction stored in a memory (140), the electronic device (100) can obtain depth information from a left eye image with an adjusted resolution and a right eye image with an adjusted resolution.

[0290] By having at least one processor (130) execute a program or at least one instruction stored in a memory (140), the electronic device (100) can adjust the resolution for the second direction of the left eye image and the right eye image based on the required resolution indicating the ability to identify different depth values.

[0291] By having at least one processor (130) execute a program or at least one instruction stored in a memory (140), the electronic device (100) can identify an application executed by the electronic device (100) and adjust the resolution of the left eye image and the right eye image based on the characteristics of the identified application.

[0292] By having at least one processor (130) execute a program or at least one instruction stored in a memory (140), the electronic device (100) can identify information about a depth information acquisition time corresponding to the characteristics of the identified application, and adjust the resolution of the left eye image and the right eye image based on the information about the identified depth information acquisition time.

[0293] The electronic device (100) can identify the usage status of the processor by having at least one processor (130) execute a program or at least one instruction stored in the memory (140), and can include a step of adjusting the resolution for the first direction of the left eye image and the right eye image based on information about the usage status of the processor.

[0294] By having at least one processor (130) execute a program or at least one instruction stored in a memory (140), the electronic device (100) can identify the power status of the electronic device (100) and adjust the resolution of the left-eye image and the right-eye image based on the power status.

[0295] The electronic device (100) can improve the resolution of depth information by having at least one processor (130) execute a program or at least one instruction stored in the memory (140).

[0296] According to an embodiment of the present disclosure, a method for processing an image by an electronic device (100) may be provided.

[0297] The method may include the steps of obtaining a left eye image by photographing an object using a first camera and obtaining a right eye image by photographing the object using a second camera.

[0298] The method may include a step of adjusting the resolution for a first direction of a left-eye image and a right-eye image based on computational performance information of a processor of an electronic device.

[0299] The method may include obtaining depth information from a left eye image with an adjusted resolution and a right eye image with an adjusted resolution.

[0300] The method may include a step of adjusting the resolution for a second direction of the left eye image and the right eye image based on a required resolution representing the discrimination ability for different depth values.

[0301] The method may include a step of identifying an application executed by an electronic device (100) and a step of adjusting the resolution of a left eye image and a right eye image based on characteristics of the identified application.

[0302] The method may include a step of identifying information about a depth information acquisition time corresponding to a characteristic of the identified application and a step of adjusting the resolution of the left eye image and the right eye image based on the information about the identified depth information acquisition time.

[0303] The method may include a step of identifying a usage state of a processor and a step of adjusting a resolution for a first direction of a left-eye image and a right-eye image based on information about the usage state of the processor.

[0304] The method may include a step of identifying a power state of an electronic device (100) and a step of adjusting the resolution of a left-eye image and a right-eye image based on the power state.

[0305] The method may include a step of improving the resolution of depth information.

[0306] According to an embodiment of the present disclosure, a computer-readable recording medium having recorded thereon a program for executing the method on a computer may be provided.

[0307] The above description of the present disclosure is provided for illustrative purposes only, and those skilled in the art will readily appreciate that modifications to other specific forms can be made without altering the technical spirit or essential features of the present disclosure. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, components described as being single may be implemented in a distributed manner, and similarly, components described as being distributed may be implemented in a combined manner.

[0308] The scope of the present disclosure may be indicated by the claims that follow rather than the detailed description above. Various features mentioned in one claim category of the present disclosure (e.g., in a method claim) may also be claimed in another claim category (e.g., in a system claim). Furthermore, an embodiment of the present disclosure may include not only combinations of features specified in the appended claims, but also various combinations of individual features within the claims. The scope of the present disclosure should be interpreted to include all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts.

Claims

1. In a method for processing an image by an electronic device (100), Step (S810) of obtaining a left eye image by photographing an object using a first camera (111) and obtaining a right eye image by photographing the object using a second camera (113); A step (S820) of adjusting the resolution for the first direction of the left eye image and the right eye image based on the computational performance information of the processor (130) of the electronic device (100); and A method comprising a step (S830) of obtaining depth information from a left eye image with adjusted resolution and a right eye image with adjusted resolution.

2. In paragraph 1, A method further comprising the step of adjusting the resolution of the left eye image and the right eye image in a second direction based on a required resolution representing the discrimination capability for different depth values.

3. In either of paragraphs 1 or 2, A step of identifying an application executed by the electronic device (100); A step of identifying information about depth information acquisition time corresponding to the characteristics of the identified application; and A step of adjusting the resolution of the left eye image and the right eye image based on information about the identified depth information acquisition time; A method further comprising:

4. In paragraph 3, A method wherein the characteristics of the application include information about at least one of the type of the application, the computational speed required by the application, the importance of fast processing operations, the urgency, or whether tracking of objects is required.

5. In any one of clauses 1 to 4, The step (S220) of adjusting the resolution of the left eye image and the right eye image in the first direction is A step for identifying the usage status of the above processor (130); and A step of adjusting the resolution of the left eye image and the right eye image in the first direction based on the computational performance information of the processor (130) and the information regarding the usage status of the processor (130); A method comprising:

6. In any one of clauses 1 to 5, A step of identifying the power status of the electronic device (100); and A step of adjusting the resolution of the left eye image and the right eye image based on the power status is further included; A method wherein the power status indicates at least one of the power consumption of the electronic device (100), the power consumption, the remaining battery level, whether the electronic device (100) is charging, or whether power consumption reduction is necessary.

7. In any one of clauses 1 to 6, the step of obtaining the depth information (S230) is A method further comprising the step of improving the resolution of the depth information.

8. In any one of clauses 1 to 7, The computational performance of the above processor is A method identified based on at least one of an operational performance indicator, number of cores, cache size, clock speed, power consumption, processor architecture, bandwidth, or processor generation.

9. In an electronic device (100) for processing images, Multiple cameras (110); a memory (140) in which a program or at least one instruction is stored; and At least one processor (130); Including, The electronic device (100) executes the program or the at least one instruction stored in the memory (140) by the at least one processor (130). By photographing an object using the first camera (111) among the above multiple cameras, a left eye image is obtained, By photographing the object using the second camera (113) among the above multiple cameras, a right-eye image is obtained, Adjusting the resolution for the first direction of the left eye image and the right eye image based on the computational performance information of the at least one processor (130) of the electronic device (100), An electronic device that obtains depth information from a left eye image with adjusted resolution and a right eye image with adjusted resolution.

10. In the 9th paragraph, the electronic device (100) executes the program or the at least one instruction stored in the memory (140) by the at least one processor (130). An electronic device that adjusts the resolution of the second direction of the left eye image and the right eye image based on a required resolution representing the ability to distinguish different depth values.

11. In any one of the clauses 9 and 10, the electronic device (100) executes the program or the at least one instruction stored in the memory (140) by the at least one processor (130), Identifying an application executed by the above electronic device (100), Identify information about the depth information acquisition time corresponding to the characteristics of the above-identified application, An electronic device that adjusts the resolution of the left eye image and the right eye image based on information about the identified depth information acquisition time.

12. In paragraph 11, An electronic device, wherein the characteristics of the application include at least one of information regarding the type of the application, the computational speed required by the application, the importance of fast processing operations, the urgency, or whether tracking of objects is required.

13. In any one of paragraphs 9 to 12, The electronic device (100) executes the program or the at least one instruction stored in the memory (140) by the at least one processor (130). Identify the usage status of the above processor (130), An electronic device that adjusts the resolution of the left eye image and the right eye image in the first direction based on the computational performance information of the processor (130) and the information regarding the usage status of the processor (130).

14. In any one of the clauses 9 to 13, the electronic device (100) executes the program or the at least one instruction stored in the memory (140) by the at least one processor (130). Identify the power status of the above electronic device (100), Adjusting the resolution of the left eye image and the right eye image based on the power status, An electronic device, wherein the power status indicates at least one of the power consumption of the electronic device (100), the power consumption, the remaining battery level, whether the electronic device (100) is charging, or whether power consumption reduction is necessary.

15. A computer-readable recording medium having recorded thereon a program for executing the method of any one of clauses 1 to 8 on a computer.

Images