Electronic device and control method therefor

Abstract

Disclosed is an electronic device. The electronic device comprises: a memory for storing one or more instructions; and one or more processors, wherein the one or more processors execute the one or more instructions to: obtain optical flow edge information on the basis of optical flow information corresponding to an input image; obtain warped depth information on the basis of depth information of the input image; and obtain depth information corresponding to the input image by inputting the input image, the optical flow edge information, and the warped depth information to a first network model.

Description

Electronic device and method of controlling the same

[0001] The present disclosure relates to an electronic device and a control method thereof, and more particularly, to an electronic device for estimating depth information of an image and a control method thereof.

[0002] Depth estimation (DE) is the process of mapping each pixel in a single input image to a corresponding depth value, and can be used in a variety of fields, from navigation and robotic manipulation to augmented reality (AR) effects and 3D reconstruction.

[0003] An electronic device according to one or more embodiments includes a memory storing one or more commands, and one or more processors, wherein the one or more processors, by executing the one or more commands, obtain optical flow edge information based on optical flow information corresponding to an input image, obtain warped depth information based on depth information of the input image, and input the input image, the optical flow edge information, and the warped depth information into a first network model to obtain depth information corresponding to the input image.

[0004] According to one or more embodiments, the one or more processors can obtain the optical flow edge information based on a value obtained by first differentiating the optical flow information and a value obtained by second differentiating the optical flow information by executing the one or more instructions.

[0005] According to one or more embodiments, the one or more processors may, by executing the one or more instructions, obtain a first value by normalizing a first differential value obtained by first differentiating horizontal optical flow information, obtain a second value by normalizing a second differential value obtained by second differentiating the horizontal optical flow information, obtain a third value by normalizing a third differential value obtained by first differentiating vertical optical flow information, obtain a fourth value by normalizing a fourth differential value obtained by second differentiating the vertical optical flow information, and obtain the optical flow edge information based on the first value, the second value, the third value, and the fourth value.

[0006] According to one or more embodiments, the one or more processors may obtain the optical flow edge information based on a fifth value obtained by binarizing the first value, a sixth value obtained by binarizing the second value, a seventh value obtained by binarizing the third value, and an eighth value obtained by binarizing the fourth value by executing the one or more instructions.

[0007] According to one or more embodiments, the one or more processors may obtain first optical flow edge information by applying an OR gate to the fifth value and the seventh value, and obtain second optical flow edge information by applying an OR gate to the sixth value and the eighth value, by executing the one or more instructions. The first optical flow edge information may include depth gradients information, and the second optical flow edge information may include information focused on object boundaries.

[0008] According to one or more embodiments, the one or more processors can obtain the warped depth information by warping the estimated depth information from a previous frame to the current frame based on optical flow information corresponding to the current frame by executing the one or more commands.

[0009] According to one or more embodiments, the one or more processors can randomly drop out at least one of the learning-use optical flow edge information and the learning-use warped depth information while the learning of the first network model is in progress by executing the one or more instructions, thereby reducing the dependence of depth estimation on the optical flow edge information and the warped depth information.

[0010] According to one or more embodiments, the one or more processors can, by executing the one or more commands, input a previous frame, a current frame, and a next frame into a second network model to obtain camera pose information, and learn parameters of the first network model based on the obtained camera pose information.

[0011] According to one or more embodiments, the one or more processors may, by executing the one or more instructions, obtain warped pose information through forward warping for the camera pose information, obtain warped depth information through forward warping for the depth information corresponding to the input image, and learn parameters of the first network model based on the warped pose information and the warped depth information.

[0012] According to one or more embodiments, the one or more processors may, by executing the one or more instructions, synthesize a previous frame and a next frame based on the warped pose information and the warped depth information, and learn parameters of the first network model based on a first loss value between the previous frame and the synthesized previous frame and a second loss value between the next frame and the synthesized next frame.

[0013] A method for controlling an electronic device according to one or more embodiments includes: obtaining optical flow edge information based on optical flow information corresponding to an input image; obtaining warped depth information based on depth information of the input image; and inputting the input image, the optical flow edge information, and the warped depth information into a first network model to obtain depth information corresponding to the input image.

[0014] A non-transitory computer-readable medium storing computer instructions that, when executed by a processor of an electronic device according to one or more embodiments, cause the electronic device to perform an operation, the operation comprising: obtaining optical flow edge information based on optical flow information corresponding to an input image; obtaining warped depth information based on depth information of the input image; and inputting the input image, the optical flow edge information, and the warped depth information into a first network model to obtain depth information corresponding to the input image.

[0015] The above and other aspects, features and advantages of specific embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings.

[0016] FIG. 1A and FIG. 1B are drawings for explaining a method for estimating the depth of an input image according to one or more embodiments.

[0017] FIG. 2A is a block diagram showing the configuration of an electronic device according to one embodiment.

[0018] FIG. 2b is a block diagram specifically illustrating a configuration of an electronic device according to one or more embodiments.

[0019] FIG. 3 is a flowchart illustrating a method for acquiring depth information according to one or more embodiments.

[0020] FIGS. 4A and 4B are drawings for explaining a depth estimation method according to one or more embodiments.

[0021] FIG. 5 is a diagram illustrating the operation of an electronic device according to one or more embodiments.

[0022] FIG. 6 is a diagram illustrating the operation of an electronic device according to one or more embodiments.

[0023] FIG. 7 is a diagram illustrating a method for obtaining optical flow edge information according to one or more embodiments.

[0024] FIG. 8 is a drawing for explaining in detail a method for acquiring depth information according to one or more embodiments.

[0025] FIG. 9 is a diagram illustrating the operation of a pose estimation module according to one or more embodiments.

[0026] FIG. 10 is a diagram illustrating the operation of an image synthesis module according to one or more embodiments.

[0027] FIGS. 11A to 11C and FIGS. 12A to 12C are drawings for explaining effects according to one or more embodiments.

[0028] The terms used in this specification will be briefly explained, and the present disclosure will be described in detail.

[0029] The terms used in the embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of this disclosure. However, these terms may vary depending on the intentions or cases of those skilled in the art, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant disclosure. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on the meanings of the terms and the overall content of this disclosure.

[0030] In this specification, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a feature (e.g., a number, function, operation, or component such as a part), and do not exclude the presence of additional features.

[0031] In this disclosure, expressions such as “A or B,” “at least one of A and / or B,” or “one or more of A or / and B” can include all possible combinations of the listed items. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” can all refer to cases where (1) only A is included, (2) only B is included, or (3) both A and B are included.

[0032] As used herein, the expressions “first,” “second,” “first,” or “second,” etc., may describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0033] When it is said that a component (e.g., a first component) is “operatively or communicatively coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that the component may be directly coupled to the other component, or may be connected through another component (e.g., a third component).

[0034] The expression "configured to" as used in the present disclosure may be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" may not necessarily mean only "specifically designed to" in terms of hardware.

[0035] In some contexts, the phrase "a device configured to" may mean that the device, in conjunction with other devices or components, is "capable of" performing A, B, and C. For example, the phrase "a processor configured (or set) to perform A, B, and C" may refer to a dedicated processor (e.g., an embedded processor) for performing those operations, or a general-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in a memory device.

[0036] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "consist of" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0037] In the embodiments, a "module" or "part" performs at least one function or operation and may be implemented as hardware or software, or as a combination of hardware and software. Furthermore, a plurality of "modules" or "parts" may be integrated into at least one module and implemented as at least one processor (not shown), excluding any "module" or "part" that needs to be implemented as specific hardware.

[0038] Meanwhile, the various elements and areas in the drawings are schematically drawn. Therefore, the technical concept of the present invention is not limited by the relative sizes or spacing depicted in the attached drawings.

[0039] An embodiment of the present disclosure will be described in more detail with reference to the attached drawings below.

[0040] FIG. 1A and FIG. 1B are drawings for explaining a method for estimating the depth of an input image according to one or more embodiments.

[0041] Optical flow estimation (OFE) densely predicts corresponding pixel locations between two images, and deep-learning-based methods have recently been used. Optical flow can be used in a variety of applications, from video compression and frame interpolation to object tracking and gesture recognition.

[0042] Depth estimation (DE), such as single-view depth estimation, is the task of mapping each pixel in a single input image to a corresponding depth value. It can be used in a variety of fields, ranging from navigation and robotic manipulation to augmented reality (AR) effects and 3D reconstruction. Depth estimation tasks can achieve increased attention and high-quality results when trained in a self-supervised manner using only images. Learning depth estimation from monocular video is particularly important because single-camera videos are readily available. However, learning depth estimation from monocular video presents additional challenges, such as estimating the camera pose and accurately estimating the ambiguous depth of moving objects.

[0043] In a moving camera sequence, optical flow can be intuitively linked to depth. The optical flow magnitude (13) of an independently moving object (or rigid flow) (11, 12) is closely related to depth, as illustrated in Fig. 1a. However, raw optical flow makes depth estimation unreliable because moving objects have ambiguous depths (based on overflow), invalidating the fixed scene assumption. That is, fast-moving objects have large overflows regardless of their distance from the camera. However, as illustrated in Fig. 1b, static camera sequences (14, 15) do not induce rigid flow (16). This problem can cause the depth estimation network to perform incorrect pre-training based on overflow, which can lead to a decrease in prediction accuracy.

[0044] Accordingly, below, various embodiments for improving depth estimation performance by using optical flow as input to a depth estimation network are described.

[0045] FIG. 2A is a block diagram showing the configuration of an electronic device according to one embodiment.

[0046] According to FIG. 2a, the electronic device (100) includes a memory (110) and one or more processors (120).

[0047] The electronic device (100) may be implemented as various types of display devices such as a TV, monitor, PC, kiosk, tablet PC, electronic picture frame, mobile phone, HMD (Head mounted Display), NED (Near Eye Display), LFD (Large format display), Digital Signage (digital signage), DID (Digital Information Display), video wall, projector display, etc., or as an image processing device (e.g., set-top box, one connected box) that provides images to the display device.

[0048] The memory (110) can store data required for various embodiments. The memory (110) may be implemented in the form of memory embedded in the electronic device (100') or in the form of memory that can be attached or detached from the electronic device (100), depending on the purpose of data storage. For example, data for driving the electronic device (100) may be stored in a memory embedded in the electronic device (100'), and data for expanding the functions of the electronic device (100) may be stored in a memory that can be attached or detached from the electronic device (100). Meanwhile, in the case of memory embedded in the electronic device (100), it may be implemented as at least one of volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (e.g., one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard drive, or solid state drive (SSD). In addition, in the case of memory that can be attached or detached to the electronic device (100'), it may be implemented as at least one of memory cards (e.g., compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (Mini-SD), extreme digital (xD), multi-media card (MMC), etc.), external memory that can be connected to a USB port (e.g., USB memory), etc. It can be implemented in the form of.

[0049] As an example, the memory (110) may store a computer program including at least one instruction or instructions for controlling the electronic device (100).

[0050] In another example, the memory (110) may store an image received from an external device (e.g., a source device), an external storage medium (e.g., USB), an external server (e.g., a web hard drive), or the like, i.e., an input image. Alternatively, the memory (110) may store an image acquired through a camera provided in the electronic device (100). Here, the image may be a 2D video, but is not limited thereto.

[0051] As another example, the memory (110) may store various information required for image quality processing, such as information, algorithms, image quality parameters, etc. for performing at least one of Noise Reduction, Detail Enhancement, Tone Mapping, Contrast Enhancement, Color Enhancement, or Frame Rate Conversion. In addition, the memory (110) may also store an intermediate image generated by image processing and an image generated based on depth information.

[0052] According to one embodiment, the memory (110) may be implemented as a single memory that stores data generated from various operations according to the present disclosure. However, according to another embodiment, the memory (110) may be implemented to include multiple memories that each store different types of data or each store data generated at different stages.

[0053] In the above-described embodiment, it has been described that various data are stored in the external memory (110) of the processor (120), but at least some of the above-described data may be stored in the internal memory of the processor (120) according to an implementation example of at least one of the electronic device (100) or the processor (120).

[0054] One or more processors (120) control the overall operation of the electronic device (100). Specifically, one or more processors (120) may be connected to each component of the electronic device (100) to control the overall operation of the electronic device (100). For example, one or more processors (120) may be operatively connected to the memory (110) to control the overall operation of the electronic device (100). One or more processors (120) may be configured as one or more processors.

[0055] One or more processors (120) may perform operations of the electronic device (100) according to various embodiments by executing at least one instruction stored in the memory (110).

[0056] The one or more processors (120) may include one or more of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), an Accelerated Processing Unit (APU), a Many Integrated Core (MIC), a Digital Signal Processor (DSP), a Neural Processing Unit (NPU), a hardware accelerator, or a machine learning accelerator. The one or more processors (120) may control one or any combination of other components of the electronic device, and may perform operations related to communication or data processing. The one or more processors (120) may execute one or more programs or instructions stored in a memory. For example, the one or more processors may perform a method according to one or more embodiments of the present disclosure by executing one or more instructions stored in a memory.

[0057] When a method according to one or more embodiments of the present disclosure includes multiple operations, the multiple operations may be performed by one processor or by multiple processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor) and the third operation may be performed by the second processor (e.g., an artificial intelligence-specific processor).

[0058] One or more processors (120) may be implemented as a single core processor including one core, or may be implemented as one or more multicore processors including multiple cores (e.g., homogeneous multicores or heterogeneous multicores). When one or more processors (120) are implemented as a multicore processor, each of the multiple cores included in the multicore processor may include an internal processor memory, such as a cache memory or an on-chip memory, and a common cache shared by the multiple cores may be included in the multicore processor. In addition, each of the multiple cores (or some of the multiple cores) included in the multicore processor may independently read and execute a program instruction for implementing a method according to one or more embodiments of the present disclosure, or all (or some) of the multiple cores may be linked to read and execute a program instruction for implementing a method according to one or more embodiments of the present disclosure.

[0059] When a method according to one or more embodiments of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one core among the plurality of cores included in a multi-core processor, or may be performed by the plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by a first core included in the multi-core processor, or the first operation and the second operation may be performed by a first core included in the multi-core processor, and the third operation may be performed by a second core included in the multi-core processor.

[0060] In the embodiments of the present disclosure, a processor may mean a system on a chip (SoC) in which one or more processors and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, a GPU, an APU, a MIC, a DSP, an NPU, a hardware accelerator, or a machine learning accelerator, but the embodiments of the present disclosure are not limited thereto. Hereinafter, for the convenience of explanation, one or more processors (120) will be referred to as a processor (120).

[0061] According to one embodiment, the electronic device (100) can receive various compressed images or images of various resolutions. For example, the electronic device (100) can receive images in a compressed form such as MPEG (Moving Picture Experts Group) (e.g., MP2, MP4, MP7, etc.), JPEG (joint photographic coding experts group), AVC (Advanced Video Coding), H.264, H.265, HEVC (High Efficiency Video Codec), etc. Alternatively, the electronic device (100) can receive any one of SD (Standard Definition), HD (High Definition), Full HD, and Ultra HD images.

[0062] According to one embodiment, the processor (120) can obtain depth information from an input image. Here, the input image may include a still image, a plurality of consecutive still images (or frames), or a video. For example, the input image may be a 2D image.

[0063] Depth information can be in the form of a depth map (or depth map). A depth map represents three-dimensional distance information of an object (or objects) existing in an image, and can be assigned to each pixel of the image. For example, the expression range of 8-bit bit depth can have a grayscale value from 0 to 255. For example, when expressed based on black / white, black (low value) can represent a place far from the viewer (or camera), and white (high value) can represent a place close to the viewer. However, this is only an example, and the depth map can be expressed with various bit depth values ​​(e.g., 9-bit, 10-bit, etc.) according to various criteria.

[0064] For example, the processor (120) may obtain a depth map from a 2D input image based on a learned network model, etc. For example, the network model may be implemented as a neural network including a plurality of neural network layers. The network model may be implemented as a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or a deep Q-network, but is not limited thereto.

[0065] According to an example, the processor (120) may process an input image and then acquire depth information based on the processed image. Here, the image processing may be digital image processing including at least one of image enhancement, image restoration, image transformation, image analysis, image understanding, image compression, image decoding, or scaling.

[0066] For example, various preprocessing may be performed before acquiring depth information for an input image. However, for convenience of explanation, the input image and the preprocessed image will not be distinguished and will be referred to as an input image.

[0067] In this specification, "region" is a term referring to a portion of an image, and means at least one pixel block or a set of pixel blocks. In addition, a pixel block means a set of adjacent pixels that include at least one pixel.

[0068] The processor (120) may obtain depth information corresponding to an input image, for example, a depth map, and store the information in the memory (110). For example, when a first frame and a second frame are sequentially input, the processor (120) may sequentially obtain a first depth map and a second depth map corresponding to the first frame and the second frame while applying preprocessing and / or postprocessing to the first frame and the second frame, and store the obtained first depth map and the second depth map in the memory (110).

[0069] For example, the processor (120) can obtain depth maps of the first frame and the second frame based on various image processing methods, such as algorithms, formulas, network models, etc.

[0070] FIG. 2b is a block diagram specifically illustrating a configuration of an electronic device according to one or more embodiments.

[0071] According to FIG. 2b, the electronic device (100') may include a memory (110), one or more processors (120), a display (130), a communication interface (140), a user interface (150), a speaker (160), and a camera (170). Among the configurations illustrated in FIG. 2b, a detailed description of configurations that overlap with those illustrated in FIG. 2a will be omitted.

[0072] The display (130) may be implemented as a display including a self-luminous element or a display including a non-luminous element and a backlight. For example, it may be implemented as various types of displays such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, an LED (Light Emitting Diodes), a micro LED, a Mini LED, a PDP (Plasma Display Panel), a QD (Quantum dot) display, a QLED (Quantum dot light-emitting diodes), etc. The display (130) may also include a driving circuit, a backlight unit, etc., which may be implemented in a form such as an a-si TFT, an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc. According to an example, a touch sensor that detects a touch operation in the form of a touch film, a touch sheet, a touch pad, etc. may be disposed on the front of the display (130) so as to be implemented so as to detect various types of touch inputs. For example, the display (130) can detect various types of touch inputs, such as a touch input by a user's hand, a touch input by an input device such as a stylus pen, and a touch input by a specific electrostatic material. Here, the input device can be implemented as a pen-type input device that can be referred to by various terms such as an electronic pen, a stylus pen, an S-pen, etc. According to an example, the display (130) can be implemented as a flat display, a curved display, a flexible display that can be folded or / and rolled, etc.

[0073] It goes without saying that the communication interface (140) can be implemented as various interfaces depending on the implementation example of the electronic device (100'). For example, the communication interface (140) can communicate with an external device, an external storage medium (e.g., a USB memory), an external server (e.g., a web hard drive), etc. through a communication method such as Bluetooth, AP-based Wi-Fi (Wireless LAN network), Zigbee, wired / wireless LAN (Local Area Network), WAN (Wide Area Network), Ethernet, IEEE 1394, HDMI (High-Definition Multimedia Interface), USB (Universal Serial Bus), MHL (Mobile High-Definition Link), AES / EBU (Audio Engineering Society / European Broadcasting Union), optical, coaxial, etc. According to one example, the communication interface (160) can communicate with another electronic device, an external server, and / or a remote control device.

[0074] The user interface (150) may be implemented as a device such as a button, a touch pad, a mouse, and a keyboard, or as a touch screen that can also perform the display function and operation input function described above.

[0075] The speaker (160) may be configured to output various audio data as well as various notification sounds or voice messages. The processor (120) may control the speaker (160) to output feedback or various notifications in audio format according to various embodiments of the present disclosure.

[0076] The camera (170) can be turned on and perform shooting based on a preset event. The camera (170) can convert captured images into electrical signals and generate image data based on the converted signals. For example, the camera (170) can include at least one of a standard (or basic) camera and an ultra-wide-angle camera.

[0077] In addition, the electronic device (100') may include sensors, microphones, etc., depending on the implementation example.

[0078] Sensors may include various types of sensors, such as touch sensors, proximity sensors, acceleration sensors (or gravity sensors), geomagnetic sensors, gyro sensors, pressure sensors, position sensors, distance sensors, light sensors, etc.

[0079] A microphone is a device configured to receive user voice or other sounds and convert them into audio data. However, according to another embodiment, the electronic device (100') may receive user voice input via an external device through a communication interface (140).

[0080] FIG. 3 is a flowchart illustrating a method for acquiring depth information according to one or more embodiments.

[0081] According to FIG. 3, in operation 310, the electronic device (100) can obtain optical flow edge information based on optical flow information corresponding to the input image.

[0082] Optical flow information may be movement information obtained over time by tracking how each pixel moves in an image. For example, optical flow information may include information on the direction and speed of movement of an object. For example, optical flow information may be in the form of an optical flow map including an optical flow value corresponding to each pixel, but is not limited thereto. The optical flow map may have a structure in which optical flow values ​​corresponding to each pixel block are arranged in the form of a matrix in units of pixel blocks. In the present disclosure, a "pixel block" refers to a set of adjacent pixels including at least one pixel, and a "region" refers to a part of an image and may refer to at least one pixel block or a set of pixel blocks.

[0083] Optical follow edge information may include movement information corresponding to the edge (or boundary) of an object. For example, optical flow edge information may include information on the direction and speed of movement of an object. An edge may be an area where spatially adjacent pixel values ​​change abruptly. For example, an edge may be an area where the brightness of an image changes abruptly from a low value to a high value or from a high value to a low value.

[0084] According to one embodiment, the electronic device (100) can obtain optical flow edge information by differentiating optical flow information. For example, the electronic device (100) can obtain optical flow edge information by applying a filter to the optical flow information. For example, the electronic device (100) can obtain a value obtained by applying a filter to the optical flow information (or a filtered value, a filter value) as optical flow edge information.

[0085] According to an example, the electronic device (100) may obtain optical flow edge information based on a value obtained by first differentiating optical flow information and a value obtained by second differentiating optical flow information. For example, the electronic device (100) may obtain optical flow edge information based on a value obtained by applying a first differentiation filter to the optical flow information and a value obtained by applying a second differentiation filter. For example, the second differentiation filter may include at least one of a Laplacian filter, a Roberts filter, a Sobel filter, a directional filter, a gradient filter, a difference filter, or a Prewitte filter.

[0086] According to an example, the electronic device (100) may obtain a first value by normalizing (or normalizing) a first differential value obtained by first differentiating horizontal optical flow information, and may obtain a second value by normalizing (or normalizing) a second differential value obtained by second differentiating horizontal optical flow information. In addition, the electronic device (100) may obtain a third value by normalizing (or normalizing) a third differential value obtained by first differentiating vertical optical flow information, and may obtain a fourth value by normalizing a fourth differential value obtained by second differentiating vertical optical flow information. Subsequently, the electronic device (100) may obtain optical flow edge information based on the first value, the second value, the third value, and the fourth value.

[0087] For example, the electronic device (100) may obtain optical flow edge information based on a fifth value obtained by binarizing the first value, a sixth value obtained by binarizing the second value, a seventh value obtained by binarizing the third value, and an eighth value obtained by binarizing the fourth value. For example, the electronic device (100) may obtain first optical flow edge information by applying an OR gate to the fifth and seventh values, and may obtain second optical flow edge information by applying an OR gate to the sixth and eighth values. An OR gate is a type of digital logic circuit that receives two inputs (assumed to be A and B), and if either A or B is 1, the result is 1, and if both are 0, the result is 0.

[0088] Binarization is a technique used in digital image processing that can convert grayscale images into black-and-white (binary) images. Grayscale images are images in which pixel values ​​range from 0 (black) to 255 (white), while binary images are images in which each pixel is represented by only one of two values, 0 or 1. Binarization can compare each pixel value in an image with a threshold value, converting values ​​greater than the threshold value to 1 (white) and values ​​less than the threshold value to 0 (black).

[0089] For example, the first derivative signal has a negative or positive value at the edge portion. Accordingly, the first optical flow edge information may include depth gradients (or depth value slope) information. For example, the second derivative signal has a zero crossing characteristic where the sign changes at the edge portion. Specifically, the second derivative signal has a negative value when the input signal decreases strongly or increases weakly at the edge portion, and conversely, it has a positive value when the input signal decreases weakly or increases strongly. That is, the second derivative signal has a positive value at the inner boundary of a dark object and a negative value at the outer boundary, and a positive value at the outer boundary of a bright object and a negative value at the inner boundary. Accordingly, the second optical flow edge information may include information focused on object boundaries (or object outer boundaries). A boundary can be distinguished from a complex edge with diverse directions in that it refers to an edge with a clear direction and a straight line, and / or an edge with a clear direction and a thickness greater than a threshold. For example, a boundary can be the boundary of an object or text.

[0090] In operation 320, the electronic device (100) may obtain warped depth information based on depth information of an input image. In one example, the electronic device (100) may obtain warped depth information by warping depth information estimated from a previous frame into the current frame based on optical flow information corresponding to the current frame.

[0091] In operation 330, the electronic device (100) can input the input image, the optical flow edge information obtained in operation 310, and the warped depth information obtained in operation 320 into the first network model to obtain depth information corresponding to the input image.

[0092] For example, the network model can be implemented as at least one of a Convolutuional Neural Network (CNN), a Deep Neural Network (DNN), a Recurrent Neural Network (RNN), a Long Short Term Memory Network (LSTM), a Gated Recurrent Units (GRU), a Generative Adversarial Network (GAN), a Restricted Boltzmann Machine (RBM), a Deep Belief Network (DBN), a Bidirectional Recurrent Deep Neural Network (BRDNN), or a Deep Q-Network.

[0093] Here, learning a network model means a process in which a basic learning network model (e.g., an artificial intelligence model including any random parameters) adjusts parameter values ​​from the perspective of optimizing an objective function using a plurality of training data by a learning algorithm, and means that a predefined operation rule or learning network model set to perform a desired characteristic (or objective) is created. Such learning may be performed through a separate server and / or system, but is not limited thereto, and may also be performed in an electronic device (100). Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0094] According to one embodiment, the electronic device (100) can randomly drop out at least one of the learning-use optical flow edge information and the learning-use warped depth information while the first network model is being learned, thereby reducing the dependence of depth estimation on the optical flow edge information and the warped depth information.

[0095] According to one embodiment, the electronic device (100) may acquire camera pose information by inputting a previous frame, a current frame, and a next frame into a second network model, and train the first network model based on the acquired camera pose information. The camera pose information may include external parameters of the camera.

[0096] According to one embodiment, the electronic device (100) may obtain warped pose information through forward warping on camera pose information, and may obtain warped depth information through forward warping on depth information corresponding to an input image. Subsequently, the electronic device (100) may synthesize a previous frame and a next frame based on the warped pose information and the warped depth information. Thereafter, the electronic device (100) may learn parameters (or weights) of a first network model based on a first loss value between the previous frame and the synthesized previous frame, and a second loss value between the next frame and the synthesized next frame.

[0097] Forward warping is a method of moving each pixel of an input image or object to a new location to create a deformed object or image, where the movement of each pixel can be determined based on the shape or location of the deformed object or image.

[0098] Meanwhile, in Fig. 3, the order is mapped for all steps for convenience of explanation, but it is of course not necessarily limited to the order of steps that are not related to the order or can be performed in parallel.

[0099] FIGS. 4A and 4B are drawings for explaining a depth estimation method according to one or more embodiments.

[0100] According to FIG. 4A, the electronic device (100) may include an input module (410) and a depth estimation module (420). For example, each module may be implemented with at least one software, at least one hardware, and / or a combination thereof.

[0101] For example, at least one of the input module (410) and the depth estimation module (420) may be implemented to utilize a predefined algorithm, a predefined formula, and / or a learned network model. The input module (410) and the depth estimation module (420) may be included within the electronic device (100), but may be distributed to at least one external device according to an example.

[0102] The input module (410) can process input information to be provided to the depth estimation module (420). As an example, the input information can include an input image, optical flow edge information, and warped depth information.

[0103] For example, the input module (410) can obtain optical flow edge information based on optical flow information corresponding to the input image. For example, the input module (410) can obtain optical flow edge information by performing edge filtering and binarization on the optical flow information.

[0104] For example, the input module (410) can obtain warped depth information by warping depth information estimated from a previous frame into the current frame based on optical flow information corresponding to the current frame.

[0105] The depth estimation module (420) can estimate the depth of an input image based on input information provided from the input module (410). In one example, the depth estimation module (420) can input the input image, optical flow edge information, and warped depth information into a learned first network model to obtain estimated depth information.

[0106] According to FIG. 4b, the electronic device (100) may include an input module (410), a depth estimation module (420), a pose estimation module (430), and an image synthesis module (440). For example, each module may be implemented with at least one software, at least one hardware, and / or a combination thereof.

[0107] For example, at least one of the input module (410), the depth estimation module (420), the pose estimation module (430), and the image synthesis module (440) may be implemented to utilize a predefined algorithm, a predefined formula, and / or a learned network model. The input module (410), the depth estimation module (420), the pose estimation module (430), and the image synthesis module (440) may be included within the electronic device (100), but may be distributed to at least one external device according to an example.

[0108] The input module (410) can process input information to be provided to the depth estimation module (420). As an example, the input information can include an input image, optical flow edge information, and warped depth information.

[0109] The depth estimation module (420) can estimate the depth of an input image based on input information provided from the input module (410). In one example, the depth estimation module (420) can input an input image, optical flow edge information, and warped depth information into a learned network model to obtain estimated depth information.

[0110] The pose estimation module (430) may obtain camera pose information and provide it to the depth estimation module (420). As an example, the pose estimation module (430) may obtain camera pose information by inputting the previous frame, the current frame, and the next frame to the second network model. The depth estimation module (420) may train the first network model based on the camera pose information provided from the pose estimation module (430).

[0111] The image synthesis module (440) can synthesize previous and next frames based on warped pose information and warped depth information. The synthesized frames can be used to learn parameters of the first network model.

[0112] For example, the image synthesis module (440) may obtain warped pose information through forward warping on camera pose information, and may obtain warped depth information through forward warping on depth information corresponding to the input image. Subsequently, the image synthesis module (440) may synthesize a previous frame and a next frame based on the warped pose information and the warped depth information. For example, it may be used to learn parameters of a first network model based on a first loss value between the previous frame and the synthesized previous frame, and a second loss value between the next frame and the synthesized next frame.

[0113] FIG. 5 is a diagram illustrating the operation of an electronic device according to one or more embodiments.

[0114] According to FIG. 5, the electronic device (100) is a previous frame (I -1 ) to the current frame (I0)(411) optical flow (F 0→-1 ))(412) Edge filtering and binarization (413) can be performed to obtain optical flow edge information (414).

[0115] For example, the electronic device (100) may include the current frame I0 (411) of the input image, optical flow edge information (414), and warped depth information (D -1→0 )(415) can be used as input to the first network model (421).

[0116] For example, the electronic device (100) may include the current frame I0 (411) of the input image, optical flow edge information (412), and warped depth information (D -1→0 )(415) is input into the first network model (421) to estimate the depth volume (D L )(or Depth probability logits)(422) can be obtained.

[0117] In one example, the electronic device (100) includes a depth estimation volume (D L )(422) can be weighted (423) to obtain the final estimated depth map (D^)(424). According to an example, the electronic device (100) can output a single channel D L A sigmoid function can be applied to the final estimated depth map (^D) (424) and scaled within a preset range. For example, the electronic device (100) can obtain a single channel output D L The final estimated depth map (D^) (424) can be obtained by activating the sigmoid σ(·) and scaling it within the min-max disparity range. The sigmoid function has an S-shaped curve that transforms the input value into a value between 0 and 1, which allows the range of the input value to be limited and the output to be interpreted as a probability. The sigmoid function can output a value close to 0 when the input value is a large negative number, and a value close to 1 when the input value is a large positive number.

[0118] FIG. 6 is a diagram illustrating the operation of an electronic device according to one or more embodiments.

[0119] According to the embodiment illustrated in FIG. 6, while learning of the first network model (421) is in progress, the electronic device (100) receives optical flow edge information (414) and warped depth information (D -1→0 )(415) can be randomly dropped out without always being input to the first network model (421). This is to prevent the depth estimation of the first network model (421) from being overly dependent on optical flow edge information.

[0120] FIG. 7 is a diagram illustrating a method for obtaining optical flow edge information according to one or more embodiments.

[0121] According to the embodiment illustrated in FIG. 7, the electronic device (100) can obtain optical flow edge information (711, 712) by performing edge filtering and binarization (413) based on optical flow information (412).

[0122] For example, the electronic device (100) can obtain optical flow edge information (711, 712) based on a value obtained by first differentiating optical flow information (412) and a value obtained by second differentiating optical flow information (412).

[0123] For example, the electronic device (100) may include horizontal optical flow information (hereinafter, Horzontal OF, F) included in the optical flow information (412). x ) and vertical optical flow information (hereinafter, Vertical OF, F y ) can be obtained.

[0124] In one example, the electronic device (100) is F x Differentiate for the first time and find the first derivative ∂ x F x and ∂ y F x can be obtained. In addition, the electronic device (100) is F x Differentiate for the second time to get the second derivative ∂ 2 x F xand ∂ 2 y F x can be obtained.

[0125] In one example, the electronic device (100) is F y Differentiate it for the first time and find the third derivative ∂ x F y and ∂ y F y can be obtained. In addition, the electronic device (100) is F y Differentiate for the second time to get the fourth derivative ∂ 2 x F y and ∂ 2 y F y can be obtained.

[0126] In one example, the electronic device (100) has a first differential value ∂ x F x and ∂ y F x The first value (L2norm) can be obtained by normalizing (or normalizing).

[0127] In one example, the electronic device (100) has a second differential value ∂ 2 x F x and ∂ 2 y F x The second value (L2norm) can be obtained by normalizing (or normalizing).

[0128] In one example, the electronic device (100) has a third derivative value ∂ x F y and ∂ y F y The third value (L2norm) can be obtained by normalizing (or normalizing).

[0129] In one example, the electronic device (100) has a fourth derivative value ∂ 2 x F y and ∂ 2 y F yThe fourth value (L2norm) can be obtained by normalizing (or normalizing).

[0130] In this case, non-binary edges such as the first value (L2norm), the second value (L2norm), the third value (L2norm), and the fourth value (L2norm) may still contain incorrect priors (or prior distributions).

[0131] For example, the electronic device (100) may obtain a fifth value by binarizing the first value (L2norm). For example, the fifth value may be obtained by binarizing the first value (L2norm) by a threshold value γ = 0.5. For example, the fifth value may have 0 or 1.

[0132] For example, the electronic device (100) may obtain a sixth value by binarizing the second value (L2norm). For example, the second value (L2norm) may be binarized by a threshold value γ = 0.5 to obtain the sixth value. For example, the sixth value may have 0 or 1.

[0133] For example, the electronic device (100) may obtain a seventh value by binarizing the third value (L2norm). For example, the seventh value may be obtained by binarizing the third value (L2norm) by a threshold value γ = 0.5. For example, the seventh value may have 0 or 1.

[0134] For example, the electronic device (100) may obtain a seventh value by binarizing the fourth value (L2norm). For example, the eighth value may be obtained by binarizing the fourth value (L2norm) by a threshold value γ = 0.5. For example, the eighth value may have 0 or 1.

[0135] According to an example, the electronic device (100) may obtain first optical flow edge information by applying an OR gate to the fifth value and the seventh value. When the OR gate is applied, if either the fifth value or the seventh value is 1, the result is 1, and if both are 0, the result is 0. The first optical flow edge information is information based on first-order differentiation and may include depth gradient information.

[0136] According to an example, the electronic device (100) may obtain second optical flow edge information by applying an OR gate to the sixth value and the eighth value. When the OR gate is applied, if either the sixth value or the eighth value is 1, the result is 1, and if both are 0, the result is 0. The second optical flow edge information may include information focused on object boundaries based on second-order differentiation.

[0137] FIG. 8 is a drawing for explaining in detail a method for acquiring depth information according to one or more embodiments.

[0138] For example, the first network model (421) may be a model trained to estimate depth information for each pixel in an image or video. The first network model (421) may be implemented as a deep learning model for estimating depth information, and the core structure of such a deep learning model may be referred to as a backbone.

[0139] According to FIG. 7, the first network model (421) can be learned by exchanging information with the pose estimation module (430) and the image synthesis module (440).

[0140] As an example, the first network model (421) can be implemented with an Encoder-Decoder Architecture and Residual Networks (ResNets).

[0141] According to the Encoder-Decoder Architecture, the first network model (421) may include an encoder and a decoder. The encoder may extract high-level features of an input image, and the decoder may use these features to generate a depth map.

[0142] For example, the encoder can be implemented as a ResNet encoder. For example, the encoder can be implemented as a ResNet-34 encoder, but is not limited thereto. The ResNet-34 encoder can be initialized with weights pre-trained on ImageNet image classification.

[0143] For example, the decoder can be implemented as a convolutional decoder with skip connections, but is not limited thereto. For example, the decoder can be trained from scratch using Kaiming Normal initialization.

[0144] As an example, the first network model (421) can be implemented to include a ResNet convolutional encoder and an Upsample-Conv3x3Elu-Skip-Conv3x3Elu decoder. After each Upsample-Conv3x3 operation, the number of channels is reduced by half, and the final network output of the input resolution can be obtained through an additional Upsample-Conv3x3Elu-Conv3x3 step.

[0145] For example, the first network model (421) uses the current frame I0 (411) of the input image, optical flow edge information (414), and warped depth information (D -1→0 )(415) is input, the depth estimation volume (D L )(or Depth probability logits)(422) can be trained to output.

[0146] FIG. 9 is a diagram illustrating the operation of a pose estimation module according to one or more embodiments.

[0147] For example, as illustrated in FIG. 9, the pose estimation block (710) may perform a pose estimation operation using a second network model (431). For example, the second network model (431) may be implemented as a convolutional encoder, but is not limited thereto.

[0148] The second network model (431) can use three consecutive frames as input. For example, the three consecutive frames are the previous frame I -1 , may contain the current (or center) frame I0 and the next frame I1.

[0149] The second network model (431) can output extrinsic parameters of the camera pose based on three consecutive frames. The extrinsic parameters of the camera pose can output camera position and orientation information. The camera position and orientation information indicates where the camera is located and in which direction it faces in the world coordinate system, and can be mainly expressed as rotation and translation. For example, the extrinsic parameters can include a rotation vector (or rotation matrix) and a translation vector. For example, in FIG. 9, the second network model (431) can output a rotation vector R and a translation vector t. For example, the previous frame I -1 , and the first vector (R) based on the current frame I0 -1 , t -1 ), and a second vector ( based on the current frame I0 and the next frame I1 R1, t1) can be output.

[0150] In one embodiment, camera pose estimation is only required during training and can be disabled for inference.

[0151] FIG. 10 is a diagram illustrating the operation of an image synthesis module according to one or more embodiments.

[0152] According to an example, as illustrated in FIG. 10, the image synthesis module (440) can perform an image synthesis operation using forward warping.

[0153] The image synthesis module (440) generates a depth estimation volume (D L ), the external parameters of the camera pose [R t , t t ], a synthetic image Ic can be obtained based on the intrinsic parameters K of the camera pose. For example, the intrinsic parameters may include a focal length, an optical center, and lens distortion parameters. The focal length of the camera lens determines how much of the field of view can be captured by the image sensor, and a shorter focal length may result in a wider field of view, while a longer focal length may result in a narrower field of view. The optical center is the center point of the lens, which may coincide with the center of the image sensor.

[0154] For example, the image synthesis module (440) uses the input image I0 and the pose estimation module (430) to obtain a pose vector (R -1 , t -1 ) and (R1, t1), depth estimation volume (D L ) by forward warping to create a synthetic image I' -1 , I' +1 can be obtained.

[0155] For example, a synthetic image Ic can be obtained based on the following mathematical expression 1.

[0156]

[0157] Here, D P can be defined as in mathematical expression 2.

[0158]

[0159] σ can be a channel-wise softmax. N can be the number of inverse depth levels. g(ㆍ) can be a warping function. For example, g(ㆍ) can be a hypothetical depth plane d. n Lifting in 3D and 3D points in R c / t c Translate and rotate by I or D, and by bilinear or bicubic interpolation n L It can be a backward warping function that warps the image. Backward warping can be a way to map each pixel of the output image to a location in the input image or object.

[0160] d n can be defined as in mathematical formula 3.

[0161]

[0162] According to one embodiment, the image synthesis module (440) synthesizes the image I' -1 , I' +1 and corresponding input image I -1 , I +1 Based on this, the first network model (421) can be learned.

[0163] For example, synthetic image I' -1 and corresponding input image I -1 Liver loss, synthetic image I' +1 and corresponding input image I +1 The first network model (421) can be trained (or optimized) so that the liver loss is minimized.

[0164] For example, a first network model (421) can be trained based on total objective loss functions such as the following mathematical expression 4.

[0165]

[0166] Here, α ds is l A rec and l B rec can be empirically set to have a small contribution to .

[0167] This total loss l consists of the synthesis losses between the previous and next synthetic images, and the corresponding ground truth images, and can be a combination of perceptual loss and L1 loss. For example, perceptual loss is one of the loss functions used in deep learning, and it is a method of measuring the difference between the original image and the generated image. Perceptual loss can preserve the content and style of the image. Content loss measures the overall similarity between the two images, and style loss can measure the differences in visual features such as texture, color, and pattern between the two images. L1 loss is one of the loss functions used in deep learning, and can also be called mean absolute error (MAE). L1 loss measures the absolute error between the actual value and the predicted value, and can train the model based on the size of the error.

[0168] According to one embodiment, the first network model (421) can be additionally trained based on the photometric reconstruction loss.

[0169] For example, the first network model (421) may be additionally trained based on a photometric reconstruction loss such as Equation 5. For example, the photometric reconstruction loss (average between two reference frames A and B) may be a combination of L1 and perceptual losses.

[0170]

[0171] Here, ⊙ represents the Hadamard product, and O c can be a moving object and an occlusion mask. α p can be set to experience points to achieve a rough balance. φ l is the first pre-trained layer and allows comparison between synthetic and ground truth images in deep feature space through perceptual loss.

[0172] In one embodiment, the first network model (421) may be additionally trained based on a loss (e.g., smoothness loss) that guides the first network model (421) to generate smooth disparity (or parallax) maps at the edges of the image. The smoothness loss is one of the loss functions and may be used to train the model to maintain smoothness in the image. For example, the smoothness loss may promote smoothness in the image and minimize variations between each pixel in the image. This may allow the generated or converted image to have a smooth texture that is natural and free of excessive noise or local fluctuations.

[0173] For example, the first network model (421) can be additionally trained based on smoothness loss such as Equation 6.

[0174]

[0175] Here, ∂x and ∂y can represent image gradients by edge-aware smoothness loss. Edge-aware smoothness loss is one of the loss functions that can be used to preserve edges while maintaining the smoothness of the image. For example, edge-aware smoothness loss is calculated by considering the gradient between surrounding pixels, and when the gradient between surrounding pixels is large (e.g., edge area), the loss is made smaller to preserve the edge, and when the gradient between surrounding pixels is small (e.g., not edge area), the loss increases to maintain smoothness.

[0176] FIGS. 11A to 11C and FIGS. 12A to 12C are drawings for explaining effects according to one or more embodiments.

[0177] Figure 11a illustrates an input image (1110) according to an example.

[0178] FIG. 11b illustrates a reference image (1120) for obtaining overflow information according to an example.

[0179] FIG. 11c illustrates overflow edge information (1130) obtained based on an input image (1110) and a reference image (1120) according to an example.

[0180] FIG. 12a illustrates first depth estimation information (1210) obtained by inputting an input image, overflow edge information, and warped depth information into a first network model according to an example.

[0181] FIG. 12b illustrates second depth estimation information (1220) obtained by inputting input image and overflow edge information into the first network model according to an example.

[0182] FIG. 12c illustrates third depth estimation information (1230) obtained by inputting an input image into a first network model according to an example.

[0183] According to FIGS. 12a to 12c, it can be confirmed that the first depth estimation information (120) obtained according to one embodiment of the present disclosure is a more consistent depth map in terms of time and space compared to the second depth estimation information (1220) and the third depth estimation information (1230).

[0184] According to the various embodiments described above, it is possible to improve network performance and enhance the accuracy of depth estimation by using optical flow as network input.

[0185] Meanwhile, the methods according to the various embodiments of the present disclosure described above can be implemented only with a software upgrade or a hardware upgrade for an existing electronic device and / or server.

[0186] Additionally, the various embodiments of the present disclosure described above can also be performed through an embedded server provided in an electronic device, or an external server of the electronic device.

[0187] Meanwhile, according to a temporary example of the present disclosure, the various embodiments described above can be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device is a device that can call instructions stored from the storage medium and operate according to the called instructions, and may include an electronic device (e.g., electronic device (A)) according to the disclosed embodiments. When an instruction is executed by a processor, the processor can perform a function corresponding to the instruction directly or by using other components under the control of the processor. The instruction may include code generated or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium does not contain a signal and is tangible, but does not distinguish between data being stored semi-permanently or temporarily in the storage medium.

[0188] Furthermore, according to one embodiment of the present disclosure, the method according to the various embodiments described above may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0189] In addition, each of the components (e.g., modules or programs) according to the various embodiments described above may be composed of a single or multiple entities, and some of the corresponding sub-components described above may be omitted, or other sub-components may be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity, which may perform the same or similar functions as those performed by each of the corresponding components prior to integration. Operations performed by modules, programs or other components according to various embodiments may be executed sequentially, in parallel, iteratively or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0190] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person having ordinary skill in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. In electronic devices, A memory storing one or more instructions; and comprising one or more processors; The one or more processors, by executing the one or more instructions, Obtain optical flow edge information based on optical flow information corresponding to the input image, Obtain warped depth information based on the depth information of the input image, An electronic device that inputs the input image, the optical flow edge information, and the warped depth information into a first network model to obtain depth information corresponding to the input image.

2. In paragraph 1, The one or more processors, by executing the one or more instructions, An electronic device that obtains the optical flow edge information based on a value obtained by first differentiating the optical flow information and a value obtained by second differentiating the optical flow information.

3. In paragraph 2, The one or more processors, by executing the one or more instructions, The first value is obtained by normalizing the first derivative value obtained by first differentiating the horizontal optical flow information, and The second value is obtained by normalizing the second differential value obtained by second differentiating the above horizontal optical flow information, The third value is obtained by normalizing the third derivative value obtained by first differentiating the vertical optical flow information, and The fourth value is obtained by normalizing the fourth derivative value obtained by second differentiating the above vertical optical flow information, and An electronic device that obtains the optical flow edge information based on the first value, the second value, the third value, and the fourth value.

4. In paragraph 3, The one or more processors, by executing the one or more instructions, An electronic device that obtains the optical flow edge information based on a fifth value obtained by binarizing the first value, a sixth value obtained by binarizing the second value, a seventh value obtained by binarizing the third value, and an eighth value obtained by binarizing the fourth value.

5. In paragraph 4, The one or more processors, by executing the one or more instructions, By applying an OR gate to the fifth value and the seventh value, the first optical flow edge information is obtained, By applying an OR gate to the sixth value and the eighth value, the second optical flow edge information is obtained. The above first optical flow edge information is, Contains depth gradients information, The above second optical flow edge information is, An electronic device containing information focused on object boundaries.

6. In paragraph 1, The one or more processors, by executing the one or more instructions, An electronic device that obtains warped depth information by warping depth information estimated from a previous frame into the current frame based on optical flow information corresponding to the current frame.

7. In paragraph 1, The one or more processors, by executing the one or more instructions, An electronic device that reduces the dependence of depth estimation on the optical flow edge information and the warped depth information by randomly dropping out at least one of the learning-use optical flow edge information and the learning-use warped depth information while the learning of the first network model is in progress.

8. In paragraph 1, The one or more processors, by executing the one or more instructions, Obtain camera pose information by inputting the previous frame, current frame, and next frame to the second network model, An electronic device that learns parameters of the first network model based on the acquired camera pose information.

9. In paragraph 8, The one or more processors, by executing the one or more instructions, Obtain warped pose information through forward warping of the above camera pose information, Obtain warped depth information through forward warping of the depth information corresponding to the input image, An electronic device that learns parameters of the first network model based on the warped pose information and the warped depth information.

10. In paragraph 9, The one or more processors, by executing the one or more instructions, Synthesize the previous frame and the next frame based on the warped pose information and the warped depth information, An electronic device that learns parameters of the first network model based on a first loss value between the previous frame and the synthesized previous frame and a second loss value between the next frame and the synthesized next frame.

11. In a method for controlling an electronic device, A step of obtaining optical flow edge information based on optical flow information corresponding to an input image; A step of obtaining warped depth information based on the depth information of the input image; and A control method comprising: a step of inputting the input image, the optical flow edge information, and the warped depth information into a first network model to obtain depth information corresponding to the input image.

12. In paragraph 11, The step of obtaining the above optical flow edge information is: A control method for obtaining the optical flow edge information based on a value obtained by first differentiating the optical flow information and a value obtained by second differentiating the optical flow information.

13. In paragraph 12, The step of obtaining the above optical flow edge information is: A step of obtaining a first value by normalizing a first differential value obtained by first differentiating horizontal optical flow information; A step of obtaining a second value by normalizing a second differential value obtained by second differentiating the above horizontal optical flow information; A step of obtaining a third value by normalizing a third derivative value obtained by first differentiating vertical optical flow information; A step of obtaining a fourth value by normalizing a fourth differential value obtained by second differentiating the above vertical optical flow information; and A control method, comprising: a step of obtaining the optical flow edge information based on the first value, the second value, the third value, and the fourth value.

14. In paragraph 13, The step of obtaining the optical flow edge information based on the first value, the second value, the third value and the fourth value comprises: A control method for obtaining the optical flow edge information based on a fifth value obtained by binarizing the first value, a sixth value obtained by binarizing the second value, a seventh value obtained by binarizing the third value, and an eighth value obtained by binarizing the fourth value.

15. A non-transitory computer-readable medium storing computer instructions that, when executed by a processor of an electronic device, cause the electronic device to perform an operation, The above actions are, A step of obtaining optical flow edge information based on optical flow information corresponding to an input image; A step of obtaining warped depth information based on the depth information of the input image; and A non-transitory computer-readable medium comprising: a step of inputting the input image, the optical flow edge information, and the warped depth information into a first network model to obtain depth information corresponding to the input image.

Images