Image processing apparatus and method of operation thereof

Abstract

An image processing apparatus for performing three-dimensional (3D) transformations and a method performed by the image processing apparatus are provided. The image processing apparatus includes a memory for storing one or more instructions and at least one processor configured to execute the one or more instructions stored in the memory. The processor may be configured to: analyze the content categories of an input image; obtain, in real time, a depth estimation model corresponding to the content categories of the input image by using on-device learning, based on the results of the analysis of the content categories of the input image; obtain a depth map of the input image reflecting the estimated depth information based on the depth estimation model learned on-device; and perform a 3D transformation on the input image based on the depth map.

Description

Technical Field

[0001] This disclosure relates to an image processing apparatus and a method performed by the image processing apparatus, and more specifically, to an image processing apparatus for performing three-dimensional (3D) transformations and a method performed by the image processing apparatus. Background Technology

[0002] Recently, various 3D displays (e.g., light field displays, holographic displays, etc.) have been released, offering features for experiencing 3D games, 3D movies, 3D models, and more. However, aside from some 3D-produced content, there is a lack of content that can be viewed in 3D using 3D displays. Furthermore, when users view images, they increasingly prefer to view 2D images in 3D to enhance realism and immersion.

[0003] Therefore, companies that provide some 3D displays have been actively researching technologies to convert 2D images into 3D images when a 2D image is input.

[0004] The most important technique for converting 2D images into 3D images is extracting depth information similar to that of the real world from 2D images that lack depth information. Thanks to deep learning methods utilizing artificial intelligence and the increasing availability of various image-depth map (DB) databases for learning, the accuracy of techniques for estimating depth information from 2D images is steadily improving.

[0005] However, without a depth sensor, it is difficult to estimate the absolute depth of objects in an image; only the relative depth between objects or between an object and the background can be estimated. Due to the limitations of this relative depth estimation and the differences in display specifications, there is a problem of depth estimation errors in 3D images converted from 2D images, which differ from the stereoscopic effect perceived in the real world.

[0006] In addition, many depth estimation errors occur when the input includes 2D images (e.g., complex images with low correlation) that contain various types of content (e.g., graphics, games, documents, comments, captions, seminar materials, etc.).

[0007] Because of various depth estimation errors that occur when converting from 2D to 3D images, users experience increased fatigue and discomfort when viewing content for extended periods. To address this issue, some companies offer sliding user interfaces (UIs) for manually adjusting the overall intensity of the stereoscopic effect in 3D converted images.

[0008] However, in order to further improve the satisfaction and convenience of users using 3D displays, a technology is needed to further improve the accuracy of depth estimation of input 2D images.

[0009] This disclosure provides a method for dynamically applying a depth estimation model in real time based on the analysis of the content categories of an input 2D image, and a method for non-linearly changing the depth map by analyzing the objects included in each scene of the input 2D image. Summary of the Invention

[0010] Solution to the problem

[0011] An image processing apparatus according to embodiments of the present disclosure includes a memory storing one or more instructions and at least one processor configured to execute one or more instructions. The processor may be configured to execute one or more instructions to analyze the content categories of an input image. The processor may be configured to: based on the results of analyzing the content categories of the input image, obtain in real time a depth estimation model corresponding to the content categories of the input image by using on-device learning. The processor may be configured to: obtain a depth map of the input image reflecting the estimated depth information based on the depth estimation model learned on-device. The processor may be configured to: perform a 3D transformation on the input image based on the depth map of the input image.

[0012] A method performed by an image processing device according to embodiments of the present disclosure includes: analyzing the content categories of an input image. The method may include: obtaining, in real time, a depth estimation model corresponding to the content categories of the input image by using on-device learning, based on the result of analyzing the content categories of the input image. The method may include: obtaining a depth map of the input image reflecting estimated depth information based on the depth estimation model learned on-device. The method may include: performing a 3D transformation on the input image based on the depth map of the input image.

[0013] Embodiments of this disclosure provide a computer-readable recording medium having a program recorded thereon for executing at least one of the disclosed method embodiments on a computer, as a technical means for achieving the above-described technical objectives.

[0014] Other technical features will be readily understood by those skilled in the art based on the following figures, description and claims. Attached Figure Description

[0015] Figure 1 These are reference figures used to describe the concept of an image processing apparatus according to embodiments of the present disclosure.

[0016] Figures 2A, 2B, and 2C are diagrams illustrating operations performed by an image processing device, according to an example, for performing a three-dimensional (3D) transformation on a two-dimensional (2D) input image.

[0017] Figure 3This is a schematic diagram illustrating an operation performed by an image processing device to perform a 3D conversion on a 2D input image according to an embodiment of the present disclosure.

[0018] Figure 4 This is an internal block diagram of an image processing apparatus according to an embodiment of the present disclosure.

[0019] Figure 5 This is a diagram illustrating the process of analyzing content categories performed by an image processing device according to embodiments of the present disclosure.

[0020] Figure 6 This is a diagram used to describe the process, according to an embodiment, of obtaining a depth estimation model corresponding to a 2D input image from a cloud server by an image processing device.

[0021] Figure 7 This is a diagram illustrating a process performed by an image processing device according to embodiments of the present disclosure to obtain a depth estimation model by using on-device learning.

[0022] Figure 8 This is a diagram illustrating a process performed by an image processing device according to embodiments of the present disclosure to obtain a depth estimation model by using on-device learning.

[0023] Figure 9 This is a flowchart describing a method for performing 3D conversion on a 2D input image by an image processing device according to embodiments of the present disclosure.

[0024] Figure 10 This is an internal block diagram of an image processing apparatus according to an embodiment of the present disclosure.

[0025] Figure 11 This is a diagram illustrating the process performed by an image processing device to analyze scene objects of a 2D input image according to embodiments of the present disclosure.

[0026] Figure 12A is a diagram illustrating a process performed by an image processing device to dynamically change a depth map according to an embodiment of the present disclosure.

[0027] Figure 12B is a diagram illustrating an example of a process performed by an image processing device to dynamically change a depth map according to an embodiment of the present disclosure.

[0028] Figure 12C is a diagram illustrating an example of the result of a process performed by an image processing device to dynamically change a depth map according to an embodiment of the present disclosure.

[0029] Figure 13 This is a diagram illustrating the process of controlling a stereoscopic effect performed by an image processing device according to embodiments of the present disclosure.

[0030] Figure 14 This is a diagram illustrating an example of the result of a process performed by an image processing device to control a stereoscopic effect according to embodiments of the present disclosure.

[0031] Figure 15 This is a flowchart describing a method for performing 3D conversion on a 2D input image by an image processing device according to embodiments of the present disclosure.

[0032] Figure 16 This is a diagram illustrating an example of the effect of an image processing apparatus according to an embodiment of the present disclosure.

[0033] Figure 17 This is a block diagram illustrating an image processing apparatus according to an embodiment of the present disclosure. Detailed Implementation

[0034] Although the terminology used herein is of general usage and is currently widely used in consideration of the functions described herein, these terms may vary depending on the intent of a person skilled in the art, precedent, the emergence of new technologies, etc. Furthermore, in certain cases, these terms may be arbitrarily chosen by the applicant, and in such cases, the meaning of these terms will be described in detail in the corresponding sections of the detailed description. Therefore, the terminology used herein should be defined based on its meaning and the description throughout this specification, rather than on its simple appellation.

[0035] Furthermore, the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure.

[0036] Unless the context clearly indicates otherwise, the singular form used herein is intended to include the plural form as well. All terms used herein, including technical and scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art.

[0037] It will be understood that throughout this specification, when a portion is referred to as "comprising" or "including" a structural element, unless otherwise stated, that portion may include, rather than exclude, another structural element in addition to that structural element. Furthermore, terms such as "...unit," "...part," and "...module" as used herein refer to a unit for performing at least one function or operation, and that unit may be implemented in hardware, software, or a combination of hardware and software.

[0038] A processor may include various processing circuitry and / or multiple processors. For example, as used herein (including the claims), the term "processor" may include various processing circuitry, including at least one processor, wherein one or more of the at least one processor may be configured individually and / or collectively in a distributed manner to perform the various functions described herein. As used herein, when "processor," "at least one processor," and "one or more processors" are described as being configured to perform multiple functions, these terms cover, for example, but not limited to, a situation where one processor performs some of the functions while another processor performs other functions, and also cover a situation where a single processor can perform all of the functions. Additionally, at least one processor may include, for example, a combination of processors performing the various stated / disclosed functions in a distributed manner. At least one processor may execute program instructions to implement or perform the various functions.

[0039] In addition to their own primary functions, each element described below may additionally perform some or all of the functions performed by another element, and some of the primary functions of each element may be performed entirely by another element.

[0040] The phrase “configured (or set) to” as used in this document may be used interchangeably with, depending on the context, “suitable for,” “capable of,” “designed to,” “suitable for,” “made for,” or “capable of.”

[0041] The phrase "configured (or set) to" does not inherently mean "specifically designed to be" in hardware. Rather, in some cases, the phrase "the system is configured to" can mean that the system can operate in conjunction with another device or component.

[0042] When this document describes an element as being “connected” or “coupled” to another element, it should be understood that the element may be directly connected or directly coupled to the other element, but unless otherwise expressly stated, the element may be “connected” or “coupled” to the other element through another intermediate element. In this document, when an element is referred to as being “connected” to another element, it may be “directly connected” to the other element, or it may be “electrically connected” to the other element through one or more intermediate elements.

[0043] In the context of describing this disclosure (especially in the context of the claims), the use of the term "described" and similar designations should be understood to encompass both the singular and plural. Furthermore, all methods described herein may be performed in any suitable order unless otherwise indicated herein or explicitly stated otherwise by the context. This disclosure is not limited to the described order of operations.

[0044] Phrases such as “some embodiments” or “in embodiments” appearing in various places in this document do not necessarily refer to the same embodiments.

[0045] Additionally, the expression "at least one of a, b, and c" in this document may refer to "a", "b", "c", "a and b", "a and c", "b and c", "all of a, b, and c", or variations thereof. The numbers used in the description of this specification (e.g., first, second, third, etc.) are merely identifiers used to distinguish one element from another.

[0046] Embodiments of this disclosure can be represented by functional block configurations and various processing operations. Some or all of the functional blocks can be implemented in various numbers of hardware and / or software configurations performing a specific function. For example, the functional blocks of this disclosure can be implemented by one or more microprocessors or by a circuit configuration for a given function. Additionally, for example, the functional blocks of this disclosure can be implemented using various programming or scripting languages. The functional blocks can be implemented using algorithms running on one or more processors. Furthermore, this disclosure can employ existing techniques for electronic configuration, signal processing, and / or data processing.

[0047] To clearly illustrate this disclosure in the accompanying drawings, parts unrelated to the description have been omitted, and similar components are given similar reference numerals throughout this specification. Furthermore, the reference numerals used in each drawing are for illustrative purposes only, and the different reference numerals used in each of different drawings are not intended to indicate different elements. Additionally, the connecting lines or connecting members between elements shown in the drawings are only for illustrating functional connections and / or physical or electrical connections. In actual devices, connections between elements can be represented by various functional connections, physical connections, or electrical connections that can be substituted or added.

[0048] In the following description, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings to facilitate implementation by those skilled in the art. However, the present disclosure may be implemented in many different forms and should not be construed as limited to the embodiments set forth herein. In describing embodiments, detailed descriptions will be omitted where it is determined that such detailed descriptions of related known art may unnecessarily obscure the gist of the present disclosure.

[0049] Additionally, in this document, the term "user" refers to a person who uses the image processing equipment, and may include consumers, evaluators, inspectors, managers, or installation engineers. Furthermore, in this document, the term "manufacturer" may refer to a manufacturer that produces the image processing equipment and / or the components included in the image processing equipment.

[0050] In this article, the term "image" can refer to a still image, picture, frame, moving image including multiple consecutive still images, or video.

[0051] In this paper, the term "two-dimensional (2D) image" is an image in which each pixel is configured in a 2D planar form corresponding to rows and columns, and may not include depth / height information.

[0052] In this paper, the term "three-dimensional (3D) image" is an image in which each pixel is configured in a 3D spatial form corresponding to row, column and depth / height information, and may include depth / height information.

[0053] In this paper, the term "scene" can refer to a series of consecutive image frames that make up an image. An image can include various scenes, and one scene can be connected to the next scene to form an overall flow of images. Each scene that makes up an image can be divided into an event, a specific theme, or a specific story unit that occurs at a specific location and time.

[0054] In this paper, the term "neural network" is a representative example of a computational system that simulates the brain's neural pathways, and is not limited to artificial neural network models using specific algorithms. Neural networks can also be referred to as "artificial neural networks (ANNs)" or "deep neural networks (DNNs)." Neural networks can include, but are not limited to, convolutional neural networks (CNNs), DNNs, recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), deep Q-networks, histogram of oriented gradients (HOGs), scale-invariant feature transforms (SHIFTs), long short-term memory (LSTMs), support vector machines (SVMs), SoftMax, etc.

[0055] In this paper, the term "neural network model" can refer to a neural network that is generated / trained to perform operations for a specific purpose. A neural network generated / trained to perform a specific function can be represented as "function" + "model" (e.g., content category analysis model, depth estimation model, object analysis model, depth dynamic change model, stereo effect control model, etc.).

[0056] In this paper, when a neural network model is regenerated / trained / obtained, this can be referred to as "updating the neural network model" or "updating the parameters of the neural network model." The expression "updating" the neural network model can also be referred to as "updating," "adapting," "adjusting," "modifying," or "changing" the neural network model.

[0057] In this paper, the term "machine learning" can refer to an algorithm that allows training a neural network model from data, or an algorithm that allows a neural network model to receive input data and predict output data. Deep learning can be interpreted as performing machine learning by using a deep neural network model.

[0058] In this paper, the terms "parameter" or "weight" refer to elements included in a matrix corresponding to a neural network model, and can refer to values ​​applied to input data for inference using the neural network model. Each of the multiple layers forming a neural network model can have multiple parameters or weights, and inference can be performed through operations between the results of previous layers and the multiple parameters or weights. The multiple parameters or weights included in the multiple layers constituting the neural network model can be optimized by training the neural network model. For example, multiple parameters or weights can be updated to reduce or minimize the loss or cost values ​​obtained from the neural network model during the training process.

[0059] Figure 1 This is a reference diagram used to describe the concept of an image processing apparatus 100 according to an embodiment of the present disclosure.

[0060] refer to Figure 1 The image processing device 100 can be an electronic device capable of receiving a 2D image 110 as input, converting the 2D image 110 into a 3D image 120, and outputting the 3D image 120. In embodiments of this disclosure, the image processing device 100 can be implemented as various types of electronic devices including a display.

[0061] The image processing device 100 can be fixed or mobile, and can be a 3D display (e.g., a light field display), but is not limited thereto.

[0062] Image processing device 100 may include at least one of the following: digital TV capable of receiving digital broadcasts, desktop computer, intelligent computer, tablet PC, mobile phone, video phone, or e-book reader, laptop PC, netbook computer, digital camera, personal digital assistant (PDA), portable multimedia player (PMP), camcorder, navigation, wearable device, smartwatch, home network system, security system, or medical device.

[0063] The image processing device 100 can be implemented not only as a flat panel display device, but also as a curved display device (which is a screen with curvature) or a flexible display device with adjustable curvature.

[0064] In embodiments of this disclosure, the image processing device 100 may utilize artificial intelligence (AI) technology to convert a 2D image 110 into a 3D image 120. In embodiments of this disclosure, the image processing device 100 may be an edge device, wherein AI is combined with an electronic device that provides the 3D image 120 to a user.

[0065] In embodiments of this disclosure, the image processing device 100 can obtain a 2D image 110 by inputting or receiving a 2D image 110.

[0066] In embodiments of this disclosure, the image processing device 100 can analyze the content categories of the acquired 2D image 110. In embodiments of this disclosure, the image processing device 100 can analyze the content categories of each scene constituting the acquired 2D image 110. This will... Figure 5 A detailed description will be provided in the following section.

[0067] In this disclosure, the "content category" of the 2D input image 110 may include, but is not limited to, movies, TV series, first-person shooter (FPS) games, role-playing games (RPGs), real-time strategy (RTS) games, massively multiplayer online RPGs (MMORPGs), documents, complex content, presentation materials, etc. For example, complex content may refer to single content that includes various characteristics, such as 2D animations, PowerPoint presentations (PPTs), commentary images, documents, etc., such as images from online lectures. The "content category" may also be referred to as "content type" or "content category".

[0068] In embodiments of this disclosure, the image processing device 100 can generate / obtain one or more depth estimation models in real time corresponding to the 2D input image 110 or corresponding to each scene constituting the 2D input image 110, based on the results of analyzing the content categories of the obtained 2D input image 110. This will refer to Figures 6 to 8 Provide a detailed description.

[0069] In this disclosure, "depth estimation model" can refer to an artificial neural network model trained to predict the depth information of each pixel constituting the 2D image 110. For example, a depth estimation model can refer to an artificial neural network model trained to predict the depth information of each pixel constituting the 2D image 110 using techniques such as CNN, DNN, RNN, RBM, DBN, BRDNN or Deep Q Network, HOG, SHIFT, LSTM, SVM, SoftMax, etc., but is not limited thereto.

[0070] Depth information can refer to information indicating the position of a corresponding pixel in 3D space relative to a reference plane when performing a 3D transformation from a 2D image 110, and can be represented in meters or pixels.

[0071] In embodiments of this disclosure, the image processing device 100 can acquire or generate one or more depth estimation models in real time that correspond to the 2D image 110 or to each scene constituting the 2D image 110 by using cloud-based AI technology or on-device AI technology.

[0072] In the context of cloud-based AI technology, the training of neural network models or inference using neural network models can be performed by cloud servers. In embodiments of this disclosure, image processing device 100 can obtain, in real time from a cloud server, one or more depth estimation models corresponding to each scene constituting the 2D input image 110, based on the results of analyzing the content categories of the 2D image 110.

[0073] In the case of on-device AI technology, data can be processed in real time by the edge device itself. Therefore, the training of neural network models and inference using neural network models can be performed by the edge device. In embodiments of this disclosure, the image processing device 100 can generate / obtain one or more depth estimation models corresponding to the 2D image 110 or each scene constituting the 2D image 110 in real time, based on the results of analyzing the content categories of the 2D image 110, by collecting data and training a depth estimation model itself (i.e., using on-device learning).

[0074] In embodiments of this disclosure, on-device AI technology can be executed by at least one processor included in the image processing device 100. In embodiments of this disclosure, on-device AI technology can be referred to as on-device learning.

[0075] In embodiments of this disclosure, the image processing device 100 can obtain a depth map (e.g., a first depth map) of the 2D input image 110 based on one or more depth estimation models corresponding to each scene constituting the 2D image 110, obtained in real time from a cloud server or learned on-device. A "depth map" can refer to a 2D image where the depth information of each pixel constituting the image is represented by values ​​such as brightness or color for each pixel. The depth estimation model can receive the 2D image 110 as input data and output the depth map of the 2D image 110 as output data.

[0076] In this disclosure, the depth map of the 2D image 110 initially obtained by the image processing device 100 (i.e., the depth map before the depth map is non-linearly changed) may be referred to as the "depth map" or the "first depth map".

[0077] In embodiments of this disclosure, the image processing device 100 can obtain a modified depth map (e.g., a second depth map) by non-linearly altering a depth map (e.g., a first depth map) based on the results of analyzing the size and distribution of objects included in each scene constituting the 2D image 110. This will refer to... Figures 11 to 13 Provide a detailed description.

[0078] In this disclosure, a depth map obtained by nonlinearly altering a first depth map of an input / received 2D image 110 initially acquired by an image processing device 100 may be referred to as a "modified depth map" or a "second depth map".

[0079] In embodiments of this disclosure, the image processing device 100 may perform 3D transformation on the 2D image 110 based on a depth map (e.g., a first depth map) or a modified depth map (e.g., a second depth map).

[0080] In embodiments of this disclosure, the image processing device 100 can obtain a 3D image 120 converted from a 2D image 110. In embodiments of this disclosure, the image processing device 100 can control a display to output the 3D image 120.

[0081] In embodiments of this disclosure, the image processing device 100 can control the display to generate and output a user interface (UI) indicating the content category corresponding to each scene constituting the 3D image 120 converted from the 2D image 110 and the stereoscopic effect of each scene. A user can determine the content category of each scene in the 3D image 120 being viewed and the degree of stereoscopic effect of each scene through the UI provided by the image processing device 100.

[0082] According to embodiments of this disclosure, the image processing device 100 can dynamically apply a depth estimation model in real time based on the analysis of the content categories of the input 2D image, analyze the objects included in each scene, and non-linearly change the depth map. Therefore, it can reduce the depth estimation error, thereby further improving the accuracy of depth estimation and increasing the satisfaction of users using 3D displays.

[0083] However, the effects that can be obtained from this disclosure are not limited to those described above, and other effects not mentioned will be clearly understood by those skilled in the art based on the following description.

[0084] Figures 2A, 2B, and 2C are diagrams used to illustrate the operation of performing a 3D transformation on a 2D input image by an image processing device according to an example.

[0085] Figure 2A is a diagram illustrating the operation of performing a 3D transformation on a 2D input image by an image processing device according to an example.

[0086] Referring to Figure 2A, the image processing device can receive a 2D image and perform a 3D conversion according to boxes 210 to 240 to output a 3D image.

[0087] When a 2D image is input, the image processing device can estimate (210) the depth of objects included in the image based on the input 2D image, generate (220) a new viewpoint view based on the depth map as a result of the depth estimation, perform hole filling (230) to fill empty areas (e.g., occluded areas) around objects in the image according to the generated new viewpoint view, and perform pixel mapping (240) in which the pixels of the display are set to match the generated hole-filled new viewpoint view in real time.

[0088] Figure 2B is a diagram for describing in detail the process 220 of generating a new viewpoint view according to the example image processing device.

[0089] Referring to Figure 2B, the principle of generating a new viewpoint is based on generating a stereoscopic image from a 2D image using a depth map. A stereoscopic image is an image with two different viewpoints, which is combined by using the positional differences of objects in each image viewed with the left and right eyes, creating a visual illusion that allows the viewer to perceive the depth of objects in the image.

[0090] For example, in the case of parallax 222 (focal length a < focal length b) behind the screen, the image processing device can generate a stereoscopic image 220A by generating and combining an image corresponding to the left eye and an image corresponding to the right eye based on the depth map obtained in 210, such that the angle between the left and right eyes is ... The angle between the eyes is smaller than 221 times that of reality. This is done so that the object appears behind its actual location.

[0091] For example, in the case of spurious (negative parallax) 223 (focal length a > focal length c), the image processing device can generate a stereoscopic image 220B by generating and combining an image corresponding to the left eye and an image corresponding to the right eye based on the depth map obtained in 210, such that the angle between the left and right eyes is such that... The angle between the eyes is greater than 221 in reality. This is to make the object stand out in its actual position.

[0092] Figure 2C is a diagram illustrating a sliding UI for adjusting stereoscopic effects provided by an image processing device, based on an example.

[0093] Referring to Figure 2C, a user can adjust the intensity of the stereoscopic effect of a 3D converted image using UI 250A or UI 250B provided by the image processing device. However, these types of UIs are only used to manually adjust the stereoscopic effect of the entire image when the user is viewing the image, and are not used to automatically adjust the stereoscopic effect of the image to suit the content category of the image, or to suit the content category of each scene when the scene of the image changes, as in the embodiments of this disclosure.

[0094] This disclosure provides a method that, based on the results of analyzing the content categories of an input 2D image in the depth estimation process 210, dynamically applies a depth estimation model using cloud-based AI technology or on-device AI technology to improve the accuracy of depth estimation by reducing depth estimation error.

[0095] This disclosure provides a method for non-linearly altering a depth map and controlling stereoscopic effects based on the results of analyzing the content category of an input 2D image during the process 220 of generating a new viewpoint, in order to improve the satisfaction and convenience of users using 3D displays.

[0096] However, the effects that can be obtained from this disclosure are not limited to those described above, and other effects not mentioned will be clearly understood by those skilled in the art based on the following description.

[0097] Figure 3 This is a schematic diagram illustrating the operation of performing a 3D conversion on a 2D input image 110 by an image processing device 100 according to an embodiment of the present disclosure.

[0098] refer to Figure 3 In block 310, the image processing apparatus 100 according to embodiments of the present disclosure can analyze the content category of the 2D input image 110. This will refer to Figure 5 Provide a detailed description.

[0099] Image processing device 100 can use the results of content category analysis obtained in box 310 to obtain a depth estimation model corresponding to the 2D input image 110 in real time in box 320. Image processing device 100 can use the results of content category analysis obtained in box 310 to non-linearly change the depth map in box 350. Image processing device 100 can use the results of content category analysis obtained in box 310 to control the stereoscopic effect of the 2D input image 110 in box 360.

[0100] In block 320, the image processing apparatus 100 according to an embodiment of the present disclosure can obtain a depth estimation model corresponding to the 2D input image 110 in real time based on the result of analyzing the content category of the 2D input image 110.

[0101] Image processing device 100 can obtain a depth estimation model corresponding to 2D input image 110 in real time via a cloud server based on the results of analyzing the content categories of 2D input image 110. Image processing device 100 can also obtain a depth estimation model corresponding to 2D input image 110 in real time through on-device learning based on the results of analyzing the content categories of 2D input image 110. This will refer to... Figures 6 to 8 Provide a detailed description.

[0102] Image processing device 110 can use the depth estimation model corresponding to 2D input image 110 obtained in block 320 to obtain a depth map of 2D input image 110 by performing depth estimation in block 330.

[0103] Box 330 can be operated similarly to box 210 in Figure 2A. Descriptions that are repeated in reference to Figure 2A are omitted here.

[0104] In block 340, the image processing apparatus 100 according to embodiments of the present disclosure can analyze the size and distribution of objects included in each scene constituting the 2D input image 110 based on the 2D input image 110 and a depth map of the 2D input image 110. This will refer to Figure 11 Provide a detailed description.

[0105] The image processing device 100 can use the results obtained in box 340, which analyze the size and distribution of objects included in each scene, to non-linearly change the depth map in box 350. The image processing device 100 can use the results obtained in box 340, which analyze the size and distribution of objects included in each scene, to control the stereoscopic effect of the 2D input image 110 in box 360.

[0106] In box 350, the image processing device 100 according to an embodiment of the present disclosure can non-linearly change the depth. Figure 10 (e.g., a first depth map) to obtain a modified depth map 20 (e.g., a second depth map).

[0107] Image processing device 100 can, based on at least one of the following: the results of analyzing the size and distribution of objects included in each scene constituting the 2D input image 110 obtained in box 340; the results of analyzing the content categories of the 2D input image 110 obtained in box 310; or additional information, non-linearly change the depth... Figure 10 (e.g., a first depth map) to obtain a modified depth map 20 (e.g., a second depth map). This will refer to Figures 12A to... Figure 13 Provide a detailed description.

[0108] Image processing device 100 can use a modified depth map 20 (e.g., a second depth map) of the 2D input image 110 obtained in box 350 to generate a new viewpoint view in box 370.

[0109] In frame 360, the image processing device 100 according to an embodiment of the present disclosure can determine the relative positions of objects included in each scene constituting the 2D input image 110 with respect to a virtual convergence plane corresponding to the screen, thereby controlling the stereoscopic effect of the 2D input image 110.

[0110] The image processing device 100 can determine the relative positions of objects in each scene constituting the 2D input image 110 with respect to a virtual convergence plane corresponding to the screen, based on at least one of the following: results obtained in box 340 from analyzing the size and distribution of objects included in each scene constituting the 2D input image 110; results obtained in box 310 from analyzing the content categories of the 2D input image 110; or additional information. This will refer to... Figures 14 to 15 Provide a detailed description.

[0111] The image processing device 100 can use the information obtained in box 360 regarding the relative positions of objects included in each scene constituting the 2D input image 110 with respect to the convergence plane to generate a new viewpoint view in box 370.

[0112] Boxes 370 to 390 can be operated similarly to boxes 220 to 240 in Figure 2A. Descriptions that are repeated in reference to Figure 2A are omitted here.

[0113] according to Figure 3 In frames 310 to 390, the image processing apparatus 100 according to embodiments of the present disclosure can obtain a 3D output image 120 by performing a 3D conversion on a 2D input image 110. The image processing apparatus 100 can control a display to output the 3D output image 120.

[0114] This disclosure provides a method in boxes 310 to 330 to dynamically apply a depth estimation model based on the results of analyzing the content category of an input 2D image, using cloud-based AI technology or on-device AI technology, in order to improve the accuracy of depth estimation by reducing depth estimation error.

[0115] This disclosure provides a method, in blocks 340 to 370, to non-linearly change the depth map and control the stereoscopic effect of the 2D input image based on the results of analyzing the content category of the input 2D image, by determining the relative positions of objects included in each scene constituting the 2D input image with respect to a virtual convergence plane corresponding to the screen, in order to improve the satisfaction and convenience of users using 3D displays.

[0116] However, the effects that can be obtained from this disclosure are not limited to those described above, and other effects not mentioned will be clearly understood by those skilled in the art based on the following description.

[0117] Figure 4 This is an internal block diagram of an image processing apparatus 100 according to an embodiment of the present disclosure.

[0118] refer to Figure 4The image processing apparatus 100 according to embodiments of the present disclosure may include a content category analyzer 410, a depth estimation model obtainr 420, a depth estimator 430, and a 3D conversion executor 440.

[0119] The content category analyzer 410, depth estimation model obtainr 420, depth estimator 430, and 3D transformation executor 440 can be implemented using at least one processor. The content category analyzer 410, depth estimation model obtainr 420, depth estimator 430, and 3D transformation executor 440 can be implemented using memory (e.g., ...). Figure 17 Operate on at least one instruction stored in (102).

[0120] Figure 4 Content category analyzer 410, depth estimation model obtainr 420, depth estimator 430, and 3D transformation executor 440 are shown respectively, but these components can be implemented using a single processor. In this case, the content category analyzer 410, depth estimation model obtainr 420, depth estimator 430, and 3D transformation executor 440 can be implemented using a dedicated processor, or using a combination of a general-purpose processor (e.g., an application processor (AP), central processing unit (CPU), or graphics processing unit (GPU)) and software. Additionally, the dedicated processor may include memory implementing the embodiments of this disclosure or a storage processing unit using external memory. Furthermore, AI-specific processors (e.g., neural processing units (NPUs)) may be designed with hardware architectures specifically designed to process particular AI models.

[0121] The content category analyzer 410, depth estimation model obtainr 420, depth estimator 430, and 3D transformation executor 440 can be configured with multiple processors. In this case, the content category analyzer 410, depth estimation model obtainr 420, depth estimator 430, and 3D transformation executor 440 can be implemented by a combination of dedicated processors, or by a combination of multiple general-purpose processors (e.g., AP, CPU, or GPU) and software.

[0122] In embodiments of this disclosure, the content category analyzer 410 can analyze the content categories of the 2D input image 110. In embodiments of this disclosure, the content category analyzer 410 may include appropriate logic, circuitry, interfaces, and / or code capable of operating to analyze the content categories of the 2D input image 110. The content category analyzer 410 can utilize an artificial neural network trained to analyze the content categories of the 2D input image 110 (e.g., Figure 5The content category analyzer 410 analyzes the content category of the 2D input image 110 using at least one of a content category analysis model 500 or a policy-based algorithm. In embodiments of this disclosure, the content category analyzer 410 can obtain information related to the probability of the 2D input image 110 corresponding to each predefined content category as a result of the content category analysis.

[0123] In embodiments of this disclosure, when a scene transition occurs in the 2D input image 110, the content category analyzer 410 can analyze the content category of each scene constituting the 2D input image 110. A "scene transition" can refer to a change in the scene of the image, i.e., a change from one specific scene to another. The results of the content category analysis can include the results of analyzing the content category of each scene constituting the 2D input image 110.

[0124] In embodiments of this disclosure, the content category analyzer 410 may send the results of analyzing the content category of the 2D input image 110 to the depth estimation model obtainr 420.

[0125] In embodiments of this disclosure, the depth estimation model obtainr 420 may receive results from the content category analyzer 410 analyzing the content category of the 2D input image 110.

[0126] In embodiments of this disclosure, the depth estimation model obtainr 420 can obtain a depth estimation model corresponding to the 2D input image 110 in real time based on the results of analyzing the content categories of the 2D input image 110. In embodiments of this disclosure, the depth estimation model obtainr 420 may include appropriate logic, circuitry, interfaces, and / or code capable of operating to obtain the depth estimation model corresponding to the 2D input image 110 in real time.

[0127] In embodiments of this disclosure, the depth estimation model obtainr 420 can obtain a depth estimation model corresponding to the 2D input image 110 in real time from a cloud server based on the results of analyzing the content category of the 2D input image 110.

[0128] In embodiments of this disclosure, the depth estimation model obtainr 420 may, based on the results of analyzing the content categories of the 2D input image 110, train / generate / obtain one or more depth estimation models corresponding to the 2D input image 110 in real time by using on-device learning.

[0129] In embodiments of this disclosure, when there is a scene transition in the 2D input image 110, the depth estimation model obtainr 420 can obtain / generate in real time a depth estimation model corresponding to each scene constituting the 2D input image 110, based on the results of analyzing the content category of each scene constituting the 2D input image 110. The depth estimation models corresponding to each scene constituting the 2D input image 110 can be the same as or different from each other. Even if some parameter values ​​of the filters used in the layers constituting the depth estimation models are different, the depth estimation models can still be considered different.

[0130] The depth estimation model obtained in real time from the cloud server can be dynamically applied in real time based on the results of analyzing the content categories of the 2D input image 110 or the results of analyzing the content categories of each scene constituting the 2D input image 110 due to scene transformation. However, the depth estimation model acquirer 420 does not update the depth estimation model itself. Alternatively, the depth estimation model obtained in real time by learning on the device can be dynamically applied in real time based on the results of analyzing the content categories of the 2D input image 110 or the results of analyzing the content categories of each scene constituting the 2D input image 110 due to scene transformation. The depth estimation model acquirer 420 can update the depth estimation model itself in real time.

[0131] In embodiments of this disclosure, the depth estimation model acquirer 420 may send one or more depth estimation models acquired / generated to the depth estimator 430.

[0132] In embodiments of this disclosure, the depth estimator 430 may receive one or more depth estimation models from the depth estimation model obtainr 420.

[0133] In embodiments of this disclosure, the depth estimator 430 can obtain the depth of the 2D input image 110 by performing depth estimation on the 2D input image 110 using one or more received depth estimation models. Figure 10 (For example, a first depth map). In embodiments of this disclosure, the depth estimator 430 may include operable to obtain the depth of the 2D input image 110. Figure 10 Appropriate logic, circuitry, interfaces, and / or code (e.g., the first depth map).

[0134] In embodiments of this disclosure, when there is a scene transition in the 2D input image 110, the depth estimator 430 can obtain the depth of the 2D input image 110 by applying a depth estimation model corresponding to each scene constituting the 2D input image 110 and performing depth estimation for each scene constituting the 2D input image 110. Figure 10 (For example, the first depth map).

[0135] In embodiments of this disclosure, the depth estimator 430 may send the obtained depth to the 3D transformation actuator 440. Figure 10 (For example, the first depth map).

[0136] In embodiments of this disclosure, the 3D transformation actuator 440 can receive the depth of the 2D input image 110 from the depth estimator 430. Figure 10 (For example, the first depth map).

[0137] In embodiments of this disclosure, the 3D transformation actuator 440 can transform based on the received depth. Figure 10 (For example, a first depth map) A 3D transformation is performed on the 2D input image 110 to obtain a 3D output image 120. In embodiments of this disclosure, the 3D transformation actuator 440 may include appropriate logic, circuitry, interfaces, and / or code operable to obtain the 3D output image 120 by performing a 3D transformation on the 2D input image 110.

[0138] 3D transformation actuator 440 can perform depth-based transformations. Figure 10 (For example, a first depth map) A new viewpoint view is generated, hole filling is performed, and pixel mapping is performed to obtain a 3D output image 120 from the 2D input image 110. The process of generating a new viewpoint view, performing hole filling, and performing pixel mapping can be similar to that in boxes 220 to 240 of FIG. 2A. Descriptions that are repeated with reference to FIG. 2A are omitted here.

[0139] The image processing apparatus 100 according to embodiments of the present disclosure can provide a method for dynamically applying a depth estimation model in real time based on the results of analyzing the content category of a 2D input image. This can reduce depth estimation errors, thereby improving the accuracy of depth estimation and increasing the satisfaction and convenience of users using 3D displays.

[0140] However, the effects that can be obtained from this disclosure are not limited to those described above, and other effects not mentioned will be clearly understood by those skilled in the art based on the following description.

[0141] The specific operation of the image processing apparatus 100 performing 3D conversion on an input 2D image according to an embodiment of the present disclosure is described in more detail below with reference to the accompanying drawings and description.

[0142] Figure 5 This is a diagram illustrating the process of analyzing content categories performed by an image processing device 100 according to an embodiment of the present disclosure.

[0143] refer to Figure 5According to the embodiment, the image processing apparatus 100 (e.g., the content category analyzer 410 of the image processing apparatus 100) can analyze the content category of the 2D input image 110. The image processing apparatus 100 can obtain the result 510 of analyzing the content category of the 2D input image 110 by analyzing the probability that the 2D input image 110 corresponds to each predefined content category.

[0144] Predefined content categories can include, but are not limited to, movies, TV series, FPS games, RPGs, RTS games, MMORPGs, documents, complex content, presentation materials, etc.

[0145] According to embodiments of this disclosure, the result 510 of the content category analysis may include information related to the probability that the 2D input image 110 corresponds to each predefined content category. For example, the result 510 of the content category analysis may include information indicating the specific content category among the predefined content categories that has the highest probability of corresponding to the 2D input image 110. For example, the result 510 of the content category analysis may include probability values ​​indicating the probability that the 2D input image 110 corresponds to each predefined content category.

[0146] According to embodiments of this disclosure, the image processing device 100 can analyze the content category of the 2D input image 110 by using the content category analysis model 500.

[0147] The content category analysis model 500 can refer to an artificial neural network model trained to predict the content category of a 2D image 110. For example, the content category analysis model 500 can refer to an artificial neural network model trained to predict the content category of a 2D image 110 using techniques such as CNN, DNN, RNN, RBM, DBN, BRDNN or Deep Q Network, HOG, SHIFT, LSTM, SVM, SoftMax, etc.

[0148] The content category analysis model 500 can receive a 2D input image 110 as input data and output a result 510 analyzing the content category of the 2D input image 110 as output data. The image processing device 100 can obtain the trained content category analysis model 500 from a cloud server. The image processing device 100 can retrain / generate / obtain the content category analysis model 500 by using on-device learning.

[0149] According to embodiments of this disclosure, the image processing device 100 can analyze the content category of a 2D input image 110 using a policy-based algorithm. The policy-based algorithm for analyzing the content category of the 2D image 110 can be predefined / set by the manufacturer of the image processing device 100.

[0150] According to embodiments of this disclosure, the image processing device 100 can analyze the content category of a 2D input image 110 by using a hybrid content category analysis model 500 and a policy-based algorithm.

[0151] For example, when the 2D input image 110 is a TV series, the image processing device 100 can obtain the results of analyzing the content category by using the content category analysis model 500 and the policy-based algorithm, including information indicating that the content category with the highest probability corresponding to the 2D input image 110 is a TV series, or indicating the probability value of the 2D input image 110 corresponding to each of the following: movie, TV series, FPS game, PRG, RTS game, MMORPG, document, complex content, and presentation material.

[0152] According to embodiments of this disclosure, when there is a scene transition of the 2D input image 110, the image processing device 100 can analyze the content category of the 2D input image 110 by analyzing the content category of each scene constituting the 2D input image 110.

[0153] For example, the image processing device 100 can detect scene transitions in the 2D input image 110. When the image processing device 100 detects a scene transition, it can analyze the content category of each scene that constitutes the 2D input image 110.

[0154] In this case, the result 510 of the content category analysis may include information related to the probability that each scene constituting the 2D input image 110 corresponds to a predefined specific content category. For example, the result 510 of the content category analysis may include information indicating the specific content category with the highest probability among the predefined content categories corresponding to each scene constituting the 2D input image 110. For example, the result 510 of the content category analysis may include probability values ​​for each scene constituting the 2D input image 110 corresponding to a predefined content category.

[0155] For example, it can be assumed that the 2D input image 110 is a complex image comprising four scenes S1, S2, S3, and S4, and the content categories of scenes S1, S2, S3, and S4 are TV series, document, TV series, and FPS game, respectively. The image processing device 100 can detect the scene transition S1->S2->S3->S4 of the 2D input image 110. The image processing device 100 can obtain the results of content category analysis by using a content category analysis model 500 or a policy-based algorithm, including information indicating that the content categories with the highest probability corresponding to scenes S1, S2, S3, and S4 are TV series, document, TV series, and FPS game, respectively, or indicating the probability value of each of scenes S1, S2, S3, and S4 corresponding to movie, TV series, FPS game, RPG, RTS game, MMORPG, document, complex content, or presentation material, etc. (Reference) Figure 5 The description of the 2D input image 110 includes four scenes, but this is only an example, and the 2D input image 110 may include multiple scenes or one scene.

[0156] Image processing device 100 can use the results 510 of analyzing the obtained content categories to obtain a depth estimation model corresponding to the 2D input image 110 in real time (see...). Figures 6 to 8 ), Non-linearly change the depth map (see) Figure 11 and Figure 13 ), control the stereoscopic effect of the 2D input image 110 (see Figure 14 and Figure 15 ).

[0157] The image processing apparatus 100 according to embodiments of the present disclosure can provide a method for dynamically applying a depth estimation model and nonlinearly changing the depth map based on the results of analyzing the content category displayed in a 2D input image. This can reduce depth estimation errors, thereby improving the accuracy of depth estimation and increasing the satisfaction and convenience of users using 3D displays.

[0158] However, the effects that can be obtained from this disclosure are not limited to those described above, and other effects not mentioned will be clearly understood by those skilled in the art based on the following description.

[0159] "Depth estimation model" can refer to an artificial neural network model trained to predict the depth information of each pixel constituting the 2D image 110. For example, a depth estimation model can refer to an artificial neural network model trained to predict the depth information of each pixel constituting the 2D image 110 using techniques such as CNN, DNN, RNN, RBM, DBN, BRDNN or Deep Q Network, HOG, SHIFT, LSTM, SVM, SoftMax, etc., but is not limited to this.

[0160] In the following text, see references Figure 6 The operation of obtaining a depth estimation model by using cloud-based AI technology, performed by an image processing device 100 according to an embodiment of the present disclosure, is described below, with reference to Figure 7 and Figure 8 The operation of obtaining a depth estimation model by using on-device AI technology is described by an image processing device 100 according to an embodiment of the present disclosure.

[0161] Figure 6 This is a diagram used to describe the process by which an image processing device 100 obtains a depth estimation model corresponding to a 2D input image 110 from a cloud server 600, according to an embodiment.

[0162] In the case of cloud-based AI technology, the neural network model itself and its training can be performed by a cloud server. In embodiments of this disclosure, the image processing device 100 can obtain one or more depth estimation models corresponding to the 2D image 110 or corresponding to each scene constituting the 2D image 110 from the cloud server 600 in real time based on the results of analyzing the content categories of the 2D image 110.

[0163] In embodiments of this disclosure, the cloud server 600 can pre-train a depth estimation model for each content category using a training database of 2D image-depth map pairs for each predefined content category. This can be represented as an offline-trained depth estimation model. The offline-trained depth estimation model can be installed in the image processing device 100 or a server.

[0164] In embodiments of this disclosure, the image processing device 100 may request the cloud server 600 to send offline-trained depth estimation models for each content category, and when a request exists from the image processing device 100, the cloud server 600 may send only some of the requested depth estimation models to the image processing device 100. In embodiments of this disclosure, the image processing device 100 may receive offline-trained depth estimation models for each content category in advance from the cloud server 600 and store the depth estimation models in the image processing device 100, and may immediately use some necessary depth estimation models without issuing a separate request to the cloud server 600.

[0165] For example, the offline-trained depth estimation models Model 1, Model 2, Model 3, and Model 4 can be stored in cloud server 600 or image processing device 100. Depth estimation model Model 1 can be a model pre-trained using a training database of movie or TV series image-depth graph pairs, depth estimation model Model 2 can be a model pre-trained using a training database of FPS game image-depth graph pairs, depth estimation model Model 3 can be a model pre-trained using a training database of document-depth graph pairs, and depth estimation model Model 4 can be a model pre-trained using a training database of complex content-depth graph pairs.

[0166] refer to Figure 6 In 610, the image processing apparatus 100 according to an embodiment of the present disclosure (e.g., the depth estimation model obtainr 420 of the image processing apparatus 100) can process the results 510 of the content category analysis received from the content category analyzer 410.

[0167] For example, image processing device 100 can process the results 510 of content category analysis using an infinite impulse response (IIR) filter. The probability values ​​indicating the probability of each scene constituting the 2D input image 110 corresponding to each predefined content category, and included in the results 510 of content category analysis, can be a time-varying sequence of probability values. Image processing device 100 can calculate a cumulative weighted average over any time period (e.g., a time interval corresponding to a specific frame number, a time interval where scene transitions occur, etc.) by applying specific weights to the results of content category analysis (which is a time-varying sequence of probability values) using a moving average or an exponentially weighted moving average. In this respect, image processing device 100 can adjust the degree to which the current average is affected by new probability value data through weights. Depth estimation model acquirer 420 can reduce the variability of the input probability value data by processing the results 510 of content category analysis using an IIR filter, thereby reducing flickering and improving stability.

[0168] In 620, the image processing apparatus 100 according to an embodiment of the present disclosure (e.g., the depth estimation model acquirer 420 of the image processing apparatus 100) can determine / predict the content category with the highest probability corresponding to the 2D input image 110 based on the result 510 of the processed analysis of the content category.

[0169] In 630, the image processing apparatus 100 according to an embodiment of the present disclosure can obtain from the cloud server 600 one or more depth estimation models corresponding to the content category with the highest probability corresponding to the 2D input image 110 in real time.

[0170] The image processing apparatus 100 according to embodiments of the present disclosure can receive and pre-store depth estimation models for each content category pre-trained by the cloud server 600. The image processing apparatus 100 can obtain a depth estimation model corresponding to the 2D input image 110 in real time by selecting the depth estimation model corresponding to the content category with the highest probability from the pre-stored depth estimation models for each content category.

[0171] For example, the image processing device 100 can select / obtain a depth estimation model Mx corresponding to the 2D input image 110 according to [Equation 1].

[0172] [Equation 1]

[0173]

[0174] n can represent the total number of depth estimation models for each content category obtained by the image processing device 100 from the cloud server 600. It can represent the probability that the 2D input image 110 corresponds to each of the corresponding content categories in the depth estimation models M1, M2, M3, M4, ..., Mn, and can correspond to the data included in the results 510 of the content category analysis and processed in 610.

[0175] For example, image processing device 100 can determine / predict the content category with the highest probability corresponding to 2D input image 110 as TV series based on the processed analysis content category result 510. Image processing device 100 can select / obtain a depth estimation model M1 (i.e., Model 1) corresponding to TV series from cloud server 600, where TV series is the content category with the highest probability corresponding to 2D input image 110.

[0176] According to embodiments of the present disclosure, the image processing device 100 can obtain one or more corresponding depth estimation models from the cloud server 600 in real time for each scene constituting the 2D input image 110.

[0177] According to an embodiment of the present disclosure, the image processing device 100 can select one or more depth estimation models corresponding to the content category with the highest probability corresponding to each scene constituting the 2D input image 110 from a pre-stored depth estimation model for each content category, and obtain one or more corresponding depth estimation models in real time for each scene constituting the 2D input image 110.

[0178] Therefore, the image processing device 100 can dynamically apply a depth estimation model for each scene based on the content category of each scene constituting the 2D input image 110, rather than applying a single depth estimation model to the 2D input image 110. Specifically, when the input includes 2D images of various scenes with different content categories, the image processing device 100 can reduce depth estimation errors and improve the accuracy of depth estimation.

[0179] For example, when a scene transition 640 of the 2D input image 110 is detected, at 610, the image processing device 100 can process the result 510 of the content category analysis by initializing the accumulated content category analysis result data. Whenever a scene transition 640 of the 2D input image 110 occurs, the image processing device 100 can initialize the accumulated content category analysis result data. At 620, the image processing device 100 can determine / predict the content category with the highest probability corresponding to each scene constituting the 2D input image 110 based on the processed result 510 of the content category analysis.

[0180] In 630, the image processing device 100 can obtain one or more corresponding depth estimation models from the cloud server 600 in real time for each scene constituting the 2D input image 110.

[0181] The image processing device 100 can obtain, in real time, a depth estimation model corresponding to the scene constituting the 2D input image 110 by selecting one or more depth estimation models corresponding to the content category with the highest probability from the pre-stored depth estimation models for each content category, for each scene constituting the 2D input image 110.

[0182] When the content category with the highest probability corresponding to each scene constituting the 2D input image 110 is the same, the depth estimation model corresponding to each scene can be the same, and when the content category with the highest probability corresponding to each scene is different, the depth estimation model corresponding to each scene can be different.

[0183] For example, it can be assumed that the 2D input image 110 is a complex image comprising four scenes S1, S2, S3, and S4, and the content categories of scenes S1, S2, S3, and S4 are TV series, document, TV series, and FPS game, respectively. The image processing device 100 can detect the scene transition S1->S2->S3->S4 of the 2D input image 110. Among the depth estimation models Model 1, Model 2, Model 3, and Model 4 obtained from the cloud server 600, the image processing device 100 can select / obtain depth estimation model Model 1 as the depth estimation model corresponding to "TV series" with the highest probability corresponding to scene S1, select / obtain depth estimation model Model 4 as the depth estimation model corresponding to "document" with the highest probability corresponding to scene S2, select / obtain depth estimation model Model 1 as the depth estimation model corresponding to "complex content" with the highest probability corresponding to scene S3, and select / obtain depth estimation model Model 2 as the depth estimation model corresponding to "FPS game" with the highest probability corresponding to scene S4. The image processing device 100 can select / obtain in real time one or more depth estimation models, Model 1, Model 2, Model 3 and Model 4, respectively corresponding to the scenes S1, S2, S3 and S4 constituting the 2D input image 110.

[0184] The image processing device 100 can obtain the depth of the 2D input image 110 using a depth estimation model corresponding to the obtained 2D input image 110 or one or more depth estimation models corresponding to the scene constituting the 2D input image 110. Figure 10 (For example, the first depth map).

[0185] For example, the image processing device 100 can obtain the depth of the 2D input image 110 by using the obtained depth estimation model Model 1 corresponding to the 2D input image 110. Figure 10 (For example, the first depth map).

[0186] For example, the image processing device 100 can obtain the depth of the 2D input image 110 in the following manner. Figure 10 (For example, the first depth map): The depth map of scene S1 is obtained by using the depth estimation model Model 1 corresponding to scene S1, the depth map of scene S2 is obtained by using the depth estimation model Model 4 corresponding to scene S2, the depth map of scene S3 is obtained by using the depth estimation model Model 1 corresponding to scene S3, and the depth map of scene S4 is obtained by using the depth estimation model Model 2 corresponding to scene S4.

[0187] Figure 6 Only four depth estimation models for each content category are shown, but this is merely an example, and the image processing device 100 can obtain fewer or more than four depth estimation models for each content category from the cloud server 600. Additionally, see reference... Figure 6 The description of the 2D input image 110 includes four scenes, but this is only an example, and the 2D input image 110 may include multiple scenes or one scene.

[0188] The image processing apparatus 100 according to embodiments of the present disclosure can provide a method for dynamically applying a depth estimation model in real time using cloud-based AI technology based on the content category of an input 2D image or the content category of each scene of the input 2D image. This can reduce depth estimation errors, thereby improving the accuracy of depth estimation and enhancing the satisfaction and convenience of users using 3D displays.

[0189] However, the effects that can be obtained from this disclosure are not limited to those described above, and other effects not mentioned will be clearly understood by those skilled in the art based on the following description.

[0190] Figure 7 This is a diagram illustrating the process performed by an image processing device 100 according to embodiments of the present disclosure to obtain a depth estimation model by using on-device learning.

[0191] In the case of on-device AI technology, data can be processed in real time by the edge device itself. Therefore, the training of neural network models and inference using neural network models can be performed by the edge device. In embodiments of this disclosure, the image processing device 100 can generate / obtain one or more depth estimation models corresponding to the 2D image 110 or corresponding to each scene constituting the 2D input image 110 in real time, based on the results of analyzing the content categories of the 2D input image 110, by collecting data without the cloud server 600 and training the depth estimation model itself (i.e., using on-device learning).

[0192] Image processing device 100 can train / generate new depth estimation models using a general-purpose processor (e.g., CPU and GPU) installed in image processing device 100 or a dedicated processor (e.g., NPU) by using on-device learning.

[0193] refer to Figure 7In 710, the image processing apparatus 100 according to embodiments of the present disclosure (e.g., the depth estimation model acquirer 420 of the image processing apparatus 100) can process the result 510 of content category analysis received from the content category analyzer 410. For example, the image processing apparatus 100 can process the result 510 of content category analysis by using an IIR filter. (Note: The last sentence appears to be incomplete and possibly refers to a reference.) Figure 6 The description given in 610 is a duplicate.

[0194] In 720, the image processing device 100 according to an embodiment of the present disclosure can obtain a depth estimation model Z corresponding to the 2D input image 110 in real time based on the result 510 of the processed analysis content category by updating the parameters of the depth estimation model 701 installed in the image processing device 100 or the server and retraining the depth estimation model 701.

[0195] The depth estimation model 701 for each content category can be pre-trained offline using a training database of 2D image-depth map pairs for each predefined content category, and can be installed in the image processing device 100 or a server.

[0196] In embodiments of this disclosure, the image processing device 100 may request a cloud server to send offline-trained depth estimation models for each content category in order to obtain newly trained depth estimation models. When a request arises from the image processing device 100, the cloud server may send only some of the requested depth estimation models to the image processing device 100. Alternatively, in embodiments of this disclosure, the image processing device 100 may receive offline-trained depth estimation models for each content category from the cloud server in advance and store these models in the image processing device 100 to obtain newly trained depth estimation models. Furthermore, it may be able to immediately use some necessary depth estimation models without making a separate request to the cloud server.

[0197] For example, the offline-trained depth estimation models Model 1, Model 2, Model 3, and Model 4 can be stored on a cloud server or in the image processing device 100. Model 1 can be a model pre-trained using a training database of movie or TV show image-depth graph pairs; Model 2 can be a model pre-trained using a training database of FPS game image-depth graph pairs; Model 3 can be a model pre-trained using a training database of document-depth graph pairs; and Model 4 can be a model pre-trained using a training database of complex content-depth graph pairs.

[0198] The image processing device 100 can determine / predict the content category with the highest probability corresponding to the 2D input image 110 based on the results 510 of the processed analysis of content categories.

[0199] Image processing device 100 can utilize a 2D input image 110 generated by image processing device 100 and the depth of the 2D input image 110. Figure 10 (For example, the first depth map) The training data 700 of the depth estimation model 701 is updated with parameters of the depth estimation model Model X corresponding to the content category with the highest probability corresponding to the 2D input image 110, and the depth estimation model Model Z corresponding to the 2D input image 110 is trained / generated in real time.

[0200] Image processing device 100 can obtain the depth of 2D input image 110 output from depth estimation model 701 by inputting 2D input image 110 included in training data 700 into depth estimation model Model X through a forward propagation process. Figure 10 (For example, the first depth map).

[0201] Image processing device 100 can process the depth of 2D input image 110 through a backpropagation process. Figure 10 (For example, a first depth map) is compared with the 2D input image 110 included in the training data 700 to calculate the loss and adjust the parameters of the depth estimation model Model X such that the loss is minimized. For example, the image processing device 100 can calculate the slope of the loss by differentiating the loss with respect to the parameters of the depth estimation model Model X. The image processing device 100 can update the parameters of the depth estimation model Model X in the direction that the calculated slope of the loss decreases, and can repeat the update until the loss is minimized. For example, the image processing device 100 can optimize the parameters of the depth estimation model 701 in the image processing device 100 by using a gradient descent algorithm. The image processing device 100 can repeatedly update the parameters of the depth estimation model Model X such that the loss is minimized over several iterations.

[0202] For example, based on the processed analysis of content category results 510, image processing device 100 can determine / predict that the content category with the highest probability corresponding to 2D input image 110 is a TV series. Image processing device 100 can use data including 2D input image 110 and depth information of 2D input image 110. Figure 10The training data 700 of the first depth map (for example) is used to update the parameters of the depth estimation model Model1 in the depth estimation model 701 in the image processing device 100 that corresponds to "TV series" (which is the content category with the highest probability corresponding to the 2D input image 110) through forward and backward propagation processes, and the depth estimation model Model Z corresponding to the 2D input image 110 is obtained in real time.

[0203] According to the embodiment, the image processing device 100 can obtain one or more corresponding depth estimation models in real time for each scene constituting the 2D input image 110 by using on-device learning.

[0204] According to the embodiment, the image processing device 100 can obtain one or more corresponding depth estimation models for each content category in real time for each scene constituting the 2D input image 110 by updating the parameters of one or more depth estimation models corresponding to the content category with the highest probability corresponding to each scene constituting the 2D input image 110.

[0205] Therefore, the image processing device 100 can dynamically apply a depth estimation model for each scene based on the content category of each scene constituting the 2D input image 110, rather than applying a single depth estimation model to the 2D input image 110. Specifically, when the input includes 2D images of various scenes with different content categories, the image processing device 100 can reduce depth estimation errors and improve the accuracy of depth estimation.

[0206] For example, when a scene transition 730 of the 2D input image 110 is detected, in 710, the image processing device 100 can process the analysis content category result 510 by initializing the accumulated content category analysis result data. Whenever a scene transition 730 of the 2D input image 110 occurs, the image processing device 100 can initialize the accumulated content category analysis result data. In 720, the image processing device 100 can determine / predict the content category with the highest probability corresponding to each scene constituting the 2D input image 110 based on the processed analysis content category result 510. The image processing device 100 can obtain one or more corresponding depth estimation models in real time for each scene constituting the 2D input image 110 by updating the parameters of one or more depth estimation models corresponding to the content category with the highest probability corresponding to each scene constituting the 2D input image 110.

[0207] For example, we can assume that the 2D input image 110 is a complex image comprising four scenes S1, S2, S3, and S4, and that the content categories of scenes S1, S2, S3, and S4 are TV series, document, TV series, and FPS game, respectively. The image processing device 100 can detect the scene transition S1->S2->S3->S4 of the 2D input image 110. The image processing device 100 can generate / obtain the depth estimation model Model by updating the parameters of the depth estimation model Model 1 corresponding to the most probable scene "TV series" in scene S1. The depth estimation model Model can be generated / obtained by updating the parameters of the depth estimation model Model 4 corresponding to the "document" with the highest probability corresponding to scene S2. The depth estimation model Model can be generated / obtained by updating the parameters of the depth estimation model Model 3, which corresponds to the most probable "complex content" in scenario S3. Furthermore, the depth estimation model Model can be generated / obtained by updating the parameters of the depth estimation model Model 2, which corresponds to the "FPS game" with the highest probability in scene S4. The image processing device 100 can generate / obtain in real time one or more depth estimation models corresponding to the scenes S1, S2, S3, and S4 constituting the 2D input image 110, respectively. Model Mode and Model .

[0208] The image processing device 100 can obtain the depth of the 2D input image 110 using a depth estimation model corresponding to the obtained 2D input image 110 or one or more depth estimation models corresponding to the scene constituting the 2D input image 110. Figure 10 (For example, the first depth map).

[0209] For example, the image processing device 100 can obtain the depth of the 2D input image 110 by using the obtained depth estimation model Model Z corresponding to the 2D input image 110. Figure 10 (For example, the first depth map).

[0210] For example, the image processing device 100 can obtain the depth of the 2D input image 110 in the following manner. Figure 10 (For example, the first depth map): using the obtained depth estimation model Model corresponding to scene S1 To obtain the depth map of scene S1, the depth estimation model corresponding to scene S2 is used. To obtain the depth map of scene S2, the depth estimation model corresponding to scene S3 is used. To obtain the depth map of scene S3, and then using the obtained depth estimation model corresponding to scene S4... To obtain the depth map of scene S4.

[0211] Figure 7 Only four depth estimation models for each content category are shown, but this is merely an example, and fewer or more than four depth estimation models for each content category can be stored in the image processing device 100. Additionally, see reference... Figure 7 The description of the 2D input image 110 includes four scenes, but this is only an example, and the 2D input image 110 may include multiple scenes or one scene.

[0212] The image processing apparatus 100 according to embodiments of the present disclosure can provide a method for dynamically applying a depth estimation model in real time using AI technology on the device based on the content category of the input 2D image or the content category of each scene of the input 2D image. This can reduce depth estimation errors, thereby improving the accuracy of depth estimation and enhancing the satisfaction and convenience of users using 3D displays.

[0213] However, the effects that can be obtained from this disclosure are not limited to those described above, and other effects not mentioned will be clearly understood by those skilled in the art based on the following description.

[0214] Figure 8 This is a diagram illustrating the process performed by an image processing device 100 according to embodiments of the present disclosure to obtain a depth estimation model by using on-device learning.

[0215] refer to Figure 8 In step 810, the image processing apparatus 100 according to embodiments of the present disclosure (e.g., the depth estimation model acquirer 420 of the image processing apparatus 100) can process the result 510 of content category analysis received from the content category analyzer 410. For example, the image processing apparatus 100 can process the result 510 of content category analysis by using an IIR filter. (Note: The last sentence appears to be incomplete and possibly refers to a reference.) Figure 7 The description given in 710 is a duplicate.

[0216] In 820, the image processing device 100 according to an embodiment of the present disclosure can generate / obtain a depth estimation model Mz corresponding to the 2D input image 110 in real time by interpolating the parameters of the depth estimation model 801 installed in the image processing device 100 or a cloud server based on the result 510 of the processed analysis content category.

[0217] The depth estimation model 801 for each content category can be pre-trained offline using a training database of 2D image-depth map pairs for each predefined content category, and can be installed on the image processing device 100 or a cloud server.

[0218] In embodiments of this disclosure, the image processing device 100 may request a cloud server to send offline-trained depth estimation models for each content category in order to obtain an interpolated depth estimation model. When a request is received from the image processing device 100, the cloud server may send only some of the requested depth estimation models to the image processing device 100. Alternatively, in embodiments of this disclosure, the image processing device 100 may receive offline-trained depth estimation models for each content category from the cloud server in advance and store these models in the image processing device 100 to obtain an interpolated depth estimation model. Furthermore, it may be able to immediately use some of the necessary depth estimation models without making a separate request to the cloud server.

[0219] For example, the offline-trained depth estimation models Model 1, Model 2, Model 3, and Model 4 can be stored on a cloud server or in the image processing device 100. Model 1 can be a model pre-trained using a training database of movie or TV show image-depth graph pairs; Model 2 can be a model pre-trained using a training database of FPS game image-depth graph pairs; Model 3 can be a model pre-trained using a training database of document-depth graph pairs; and Model 4 can be a model pre-trained using a training database of complex content-depth graph pairs.

[0220] Interpolation between neural network models refers to the process of generating a model with intermediate performance between two or more models by using linear interpolation, polynomial interpolation, or various interpolation methods. Interpolation between neural network models is possible when their network structures and training methods are identical. The statement that neural network models have the same network structure can mean that the number of layers, the number of neurons in each layer, the activation function, etc., are the same. The statement that neural network models have the same training method can mean that the training algorithm, hyperparameters, and the preprocessing methods for the training data are the same. In the following text, it is assumed that the depth estimation models (801, e.g., Model 1, Model 2, Model 3, and Model 4) for each content category have the same network structure and training method, and are in a relationship where interpolation between them is possible.

[0221] The image processing device 100 can generate / obtain a depth estimation model Model Mz corresponding to the 2D input image 100 in real time based on the results 510 of the processed analysis of content categories, by adjusting the interpolation weights between models according to the probability of the 2D input image 110 corresponding to each content category and interpolating the parameters of the depth estimation model 801.

[0222] The image processing device 100 can adjust the interpolation weights between models based on the probability of the 2D input image 110 corresponding to each content category using a policy-based algorithm. The policy-based algorithm used to define / adjust the interpolation weights between models can be predefined / set by the manufacturer of the image processing device 100.

[0223] For example, the image processing device 100 can interpolate the two models according to [Equation 2] to generate / obtain a depth estimation model Mz corresponding to the 2D input image 110.

[0224] [Equation 2]

[0225]

[0226] and It can represent the weight value of the interpolation. , ... It can represent the parameters of the depth estimation model Model A, and , ... This can represent the parameters of depth estimation model B. Depth estimation models A and B can each refer to one of the depth estimation models 801 for each content category. ... This can refer to the parameters of the depth estimation model Mz that is generated / obtained corresponding to the 2D input image 110.

[0227] For example, image processing device 100 can determine / predict, based on the processed analysis content category result 510, that the probability of 2D input image 110 corresponding to "TV series" is 20% and the probability of 2D input image 110 corresponding to "FPS game" is 80%. Image processing device 100 can adjust the interpolation weights based on the probability of 2D input image 110 corresponding to "TV series" being 20% ​​and the probability of 2D input image 110 corresponding to "FPS game" being 80% using a policy-based algorithm. and The image processing device 100 can obtain the depth estimation model Mz corresponding to the 2D input image 110 in real time by substituting the depth estimation model Model 1 corresponding to "TV series" and the depth estimation model Model 2 corresponding to "FPS game" into the depth estimation models Model A and Model B respectively, and interpolating the depth estimation models Model 1 and Model 2 according to [Equation 2].

[0228] The image processing apparatus 100 according to embodiments of the present disclosure can obtain one or more corresponding depth estimation models in real time for each scene constituting the 2D input image 110 by using on-device learning.

[0229] According to the embodiment, the image processing device 100 can adjust the interpolation weight values ​​between models and interpolate the parameters of the depth estimation model 801 according to the probability of each scene constituting the 2D input image 110 corresponding to each content category, thereby obtaining one or more corresponding depth estimation models for each content category in real time for each scene constituting the 2D input image 110.

[0230] Therefore, the image processing device 100 can dynamically apply a depth estimation model for each scene based on the content category of each scene constituting the 2D input image 110, rather than applying a single depth estimation model to the 2D input image 110. Specifically, when the input includes 2D images of various scenes with different content categories, the image processing device 100 can reduce depth estimation errors and improve the accuracy of depth estimation.

[0231] For example, when a scene transition 830 of the 2D input image 110 is detected, in step 810, the image processing device 100 can process the analysis content category result 510 by initializing the accumulated content category analysis result data. Whenever a scene transition 830 of the 2D input image 110 occurs, the image processing device 100 can initialize the accumulated content category analysis result data. In step 820, the image processing device 100 can determine / predict the probability that each scene constituting the 2D input image 110 corresponds to each content category based on the processed analysis content category result 510. The image processing device 100 can adjust the interpolation weights between models according to the probability that each scene constituting the 2D input image 110 corresponds to each content category and interpolate the parameters of the depth estimation model 801, thereby obtaining one or more corresponding depth estimation models in real time for each scene constituting the 2D input image 110.

[0232] For example, it can be assumed that the 2D input image 110 is a complex image comprising four scenes S1, S2, S3, and S4, and the content categories of scenes S1, S2, S3, and S4 are TV series, document, TV series, and FPS game, respectively. The image processing device 100 can detect the scene transition S1->S2->S3->S4 of the 2D input image 110. The image processing device 100 can adjust the interpolation weights between depth estimation models Model 1, Model 2, Model 3, and Model 4 based on the probability that scene S1 corresponds to "TV series", the probability that scene S1 corresponds to "FPS game", the probability that scene S1 corresponds to "complex content", and the probability that scene S1 corresponds to "document", and interpolate the depth estimation models Model 1, Model 2, Model 3, and Model 4 to generate / obtain the depth estimation model Model. Similarly, the image processing device 100 can generate / obtain in real time one or more depth estimation models corresponding to the scenes S1, S2, S3, and S4 constituting the 2D input image 110, respectively. Model Mode and Model .

[0233] The image processing device 100 can obtain the depth of the 2D input image 110 using a depth estimation model corresponding to the obtained 2D input image 110 or one or more depth estimation models corresponding to the scene constituting the 2D input image 110. Figure 10 (For example, the first depth map).

[0234] For example, the image processing device 100 can obtain the depth of the 2D input image 110 by using the obtained depth estimation model Model Mz corresponding to the 2D input image 110. Figure 10 (For example, the first depth map).

[0235] For example, the image processing device 100 can obtain the depth of the 2D input image 110 in the following manner. Figure 10 (For example, the first depth map): using the obtained depth estimation model Model corresponding to scene S1 To obtain the depth map of scene S1, the depth estimation model corresponding to scene S2 is used. To obtain the depth map of scene S2, the depth estimation model corresponding to scene S3 is used. To obtain the depth map of scene S3, and then using the obtained depth estimation model corresponding to scene S4... To obtain the depth map of scene S4.

[0236] Figure 8 The image processing device 100 is shown interpolating two depth estimation models, but this is only an example, and the image processing device 100 can interpolate two or more depth estimation models in a similar manner.

[0237] Figure 8 Only four depth estimation models for each content category are shown, but this is merely an example, and fewer or more than four depth estimation models for each content category can be stored in the image processing device 100. Additionally, see reference... Figure 8 The description of the 2D input image 110 includes four scenes, but this is only an example, and the 2D input image 110 may include multiple scenes or one scene.

[0238] The image processing apparatus 100 according to embodiments of the present disclosure can provide a method for dynamically applying a depth estimation model in real time using AI technology on the device based on the content category of the input 2D image or the content category of each scene of the input 2D image. This can reduce depth estimation errors, thereby improving the accuracy of depth estimation and enhancing the satisfaction and convenience of users using 3D displays.

[0239] However, the effects that can be obtained from this disclosure are not limited to those described above, and other effects not mentioned will be clearly understood by those skilled in the art based on the following description.

[0240] Figure 9 This is a flowchart describing a method for performing a 3D conversion on a 2D input image 10 by an image processing device 100 according to an embodiment of the present disclosure.

[0241] refer to Figure 9 In operation 910, the image processing apparatus 100 according to an embodiment of the present disclosure can analyze the content category of the 2D input image 110. For example, the input image 110 may be a 2D image. For example, the content category may include, but is not limited to, movies, TV series, FPS games, RPGs, RTS games, MMORPGs, documents, complex content, presentation materials, etc.

[0242] Image processing apparatus 100 according to embodiments of the present disclosure can analyze the content category of 2D input image 110 using content category analysis model 500. Content category analysis model 500 may refer to an artificial neural network model trained to predict the content category of 2D input image 110.

[0243] The image processing device 100 according to the embodiment can analyze the content category of the 2D input image 110 by using a policy-based algorithm. The policy-based algorithm for analyzing the content category of the 2D input image 110 can be predefined / set by the manufacturer of the image processing device 100.

[0244] According to embodiments of this disclosure, the image processing device 100 can analyze the content category of a 2D input image 110 by using a hybrid content category analysis model 500 and a policy-based algorithm.

[0245] According to embodiments of this disclosure, the result 510 of the content category analysis may include information related to the probability that the 2D input image 110 corresponds to each predefined content category. For example, the result 510 of the content category analysis may include information indicating the specific content category among the predefined content categories that has the highest probability of corresponding to the 2D input image 110. For example, the result 510 of the content category analysis may include probability values ​​indicating the probability that the 2D input image 110 corresponds to each predefined content category.

[0246] In operation 920, the image processing apparatus 100 according to an embodiment of the present disclosure can obtain a depth estimation model corresponding to the content category of the 2D input image 110 in real time based on the result of analyzing the content category of the 2D input image 110.

[0247] A depth estimation model can refer to an artificial neural network model that has been learned / trained to predict the depth information of each pixel that constitutes a 2D input image 110.

[0248] According to embodiments of the present disclosure, the image processing apparatus 100 can obtain a depth estimation model corresponding to the content category of the 2D input image 110 in real time via a cloud server 600 based on the result of analyzing the content category of the 2D input image 110.

[0249] According to embodiments of the present disclosure, the image processing apparatus 100 can obtain a depth estimation model corresponding to the content category of the 2D input image 110 in real time by using on-device learning, based on the result of analyzing the content category of the 2D input image 110.

[0250] According to an embodiment of the present disclosure, the image processing device 100 can obtain a depth estimation model corresponding to the 2D input image 110 in real time by updating the parameters of the depth estimation model installed on the image processing device 100 or a server and training the depth estimation model (720) based on the result 510 of analyzing the content category of the 2D input image 110.

[0251] According to an embodiment of the present disclosure, the image processing device 100 can obtain a depth estimation model corresponding to the 2D input image 110 in real time by interpolating the parameters of a depth estimation model installed on the image processing device 100 or a server based on the result 510 of analyzing the content category of the 2D input image 110.

[0252] According to an embodiment of the present disclosure, the image processing device 100 can obtain in real time depth estimation models corresponding to the scenes constituting the 2D input image 110 based on the result 510 of analyzing the content category of the 2D input image 110.

[0253] In operation 930, the image processing apparatus 100 according to an embodiment of the present disclosure can obtain the depth of the 2D input image 110 based on a depth estimation model learned on the apparatus. Figure 10 (For example, the first depth map). Depth Figure 10 The depth map reflects the depth information estimated by the depth estimation model. The image processing device 100 can input the 2D input image 110 as input data into the depth estimation model obtained in operation 920, and output a depth map of the 2D input image 110 as output data. The depth map can refer to a 2D image in which the depth information of each pixel constituting the image is represented by a value such as brightness or color for each pixel.

[0254] According to embodiments of the present disclosure, the image processing device 100 can obtain the depth of the 2D input image 110 in real time by applying a depth estimation model corresponding to each scene constituting the 2D input image 110 for each scene constituting the 2D input image 110. Figure 10 (For example, the first depth map).

[0255] The depth map of the 2D input image 110 initially obtained by the image processing device 100 (i.e., the depth map before the depth map is non-linearly changed) can be referred to as the "depth map" or "first depth map".

[0256] In operation 940, the image processing device 100 according to an embodiment of the present disclosure can process the image based on the depth of the 2D input image 110. Figure 10 (For example, a first depth map) performs a 3D transformation on the 2D input image 110.

[0257] According to embodiments of the present disclosure, the image processing device 100 can process images based on the depth of a 2D input image 110. Figure 10 (For example, a first depth map), the 2D input image 110 is transformed into 3D by performing a process 370 to generate a new viewpoint view, a hole filling process 380, and a pixel mapping process 390.

[0258] In embodiments of this disclosure, the image processing device 100 can obtain a 3D output image 120 converted from a 2D input image 110. The image processing device 100 can control a display to output the 3D output image 120.

[0259] In embodiments of this disclosure, the image processing device 100 can control a display to generate and output a UI indicating the content category corresponding to each scene constituting the 3D output image 120 converted from the 2D input image 110 and the stereoscopic effect of each scene. A user can determine the content category of each scene in the 3D output image 120 being viewed and the degree of stereoscopic effect of each scene through the UI provided by the image processing device 100.

[0260] According to embodiments of this disclosure, the image processing device 100 can dynamically apply a depth estimation model in real time based on the analysis of the content category of the input 2D image, thereby reducing depth estimation error, further improving the accuracy of depth estimation, and increasing the satisfaction of users using 3D displays.

[0261] However, the effects that can be obtained from this disclosure are not limited to those described above, and other effects not mentioned will be clearly understood by those skilled in the art based on the following description.

[0262] Figure 10 This is an internal block diagram of an image processing apparatus 100 according to an embodiment of the present disclosure.

[0263] refer to Figure 10 The image processing device 100 according to embodiments of the present disclosure may include a content category analyzer 1010, a depth estimation model obtainr 1020, a depth estimator 1030, a scene object analyzer 1040, a depth map dynamic changer 1050, a stereo effect controller 1060, and a 3D conversion executor 1070.

[0264] The content category analyzer 1010, depth estimation model obtainr 1020, depth estimator 1030, scene object analyzer 1040, depth map dynamic changer 1050, stereo effect controller 1060, and 3D transformation executor 1070 can be implemented by at least one processor. The content category analyzer 1010, depth estimation model obtainr 1020, depth estimator 1030, scene object analyzer 1040, depth map dynamic changer 1050, stereo effect controller 1060, and 3D transformation executor 1070 can be implemented based on memory (e.g., ...). Figure 17 Operate on at least one instruction stored in (102).

[0265] Figure 10The diagram illustrates a content category analyzer 1010, a depth estimation model acquirer 1020, a depth estimator 1030, a scene object analyzer 1040, a depth map dynamic transformer 1050, a stereo effect controller 1060, and a 3D transformation executor 1070. However, these components can be implemented using a single processor. In this case, the content category analyzer 1010, depth estimation model acquirer 1020, depth estimator 1030, scene object analyzer 1040, depth map dynamic transformer 1050, stereo effect controller 1060, and 3D transformation executor 1070 can be implemented using a dedicated processor, or a combination of a general-purpose processor (e.g., an application processor, CPU, or GPU) and software. Additionally, the dedicated processor may include memory implementing the embodiments of this disclosure or a storage processing unit using external memory. In addition, AI-specific processors such as NPUs can be designed with hardware architectures specifically designed to process particular AI models.

[0266] The content category analyzer 1010, depth estimation model acquirer 1020, depth estimator 1030, scene object analyzer 1040, depth map dynamic changer 1050, stereo effect controller 1060, and 3D transformation executor 1070 can be configured using multiple processors. In this case, the content category analyzer 1010, depth estimation model acquirer 1020, depth estimator 1030, scene object analyzer 1040, depth map dynamic changer 1050, stereo effect controller 1060, and 3D transformation executor 1070 can be implemented using a combination of dedicated processors, or using a combination of multiple general-purpose processors (e.g., AP, CPU, or GPU) and software.

[0267] In embodiments of this disclosure, the content category analyzer 1010, the depth estimation model obtainr 1020, and the depth estimator 1030 can be respectively connected to... Figure 4 The content category analyzer 410, depth estimation model obtainr 420, and depth estimator 430 operate similarly. Details related to the reference are omitted here. Figure 4 The given description is a duplicate.

[0268] In embodiments of this disclosure, the scene object analyzer 1040 can receive the depth of the 2D input image 110 from the depth estimator 1030. Figure 10 (For example, the first depth map).

[0269] In embodiments of this disclosure, the scene object analyzer 1040 can analyze the size and distribution of objects included in each scene constituting the 2D input image 110. In embodiments of this disclosure, the scene object analyzer 1040 can analyze the size and distribution of objects included in each scene constituting the 2D input image 110. Figure 10 (For example, a first depth map), for each scene constituting the 2D input image 110, the size and distribution of objects included in each scene are analyzed.

[0270] In embodiments of this disclosure, the scene object analyzer 1040 may include appropriate logic, circuitry, interfaces, and / or code operable to analyze the size and distribution of objects included in each scene constituting the 2D input image 110.

[0271] An “object” (which is an independently identifiable subject or object within each scene constituting the 2D input image 110) can refer to an element with visually distinguishable characteristics. For example, an object can refer to a specific object, animal, or person included in each scene constituting the 2D input image 110.

[0272] In embodiments of this disclosure, the scene object analyzer 1040 can utilize an artificial neural network trained to analyze the size and distribution of objects included in each scene constituting the 2D input image 110 (e.g., Figure 11 The scene object analysis model 1100 or at least one of the policy-based algorithms is used to analyze the size and distribution of objects included in each scene constituting the 2D input image 110.

[0273] In embodiments of this disclosure, the scene object analyzer 1040 can obtain information related to the size characteristics of each object included in each scene and the distribution characteristics of each object included in each scene, as a result of analyzing the size and distribution of objects included in each scene.

[0274] In embodiments of this disclosure, the scene object analyzer 1040 may send the results of analyzing the size and distribution of objects included in each scene constituting the 2D input image 110 to the depth map dynamic changer 1050 and the stereo effect controller 1060.

[0275] In embodiments of this disclosure, the depth map dynamic changer 1050 can receive results from the scene object analyzer 1040 analyzing the size and distribution of objects included in each scene constituting the 2D input image 110. The depth map dynamic changer 1050 can receive results 510 from the content category analyzer 1010 analyzing the content categories of the 2D input image 110. The depth map dynamic changer 1050 can receive the depth of the 2D input image 110 from the depth estimator 1030. Figure 10 (For example, the first depth map).

[0276] In embodiments of this disclosure, the depth map dynamic changer 1050 may obtain additional information, including at least one of metadata information about the 2D input image 110, information about the viewing environment, or user settings information related to the 2D input image 110.

[0277] In embodiments of this disclosure, the depth map dynamic transformer 1050 can non-linearly change the depth based on at least one of the following: the results of analyzing the size and distribution of objects included in each scene, the results of analyzing the content categories of the 2D input image 110, or additional information obtained. Figure 10 (e.g., a first depth map) to obtain a modified depth map 20 (e.g., a second depth map).

[0278] In embodiments of this disclosure, the depth map dynamic changer 1050 may include components operable to change depth non-linearly. Figure 10 (e.g., a first depth map) to obtain the appropriate logic, circuitry, interfaces, and / or code for the modified depth map 20 (e.g., a second depth map).

[0279] In embodiments of this disclosure, the depth map dynamic changer 1050 can non-linearly change the depth by using at least one of the following: an artificial neural network trained to non-linearly change the input depth map constituting the 2D input image 110 (e.g., the depth map dynamic change model 1200 of FIG. 12A), a policy-based algorithm, or a lookup table (LUT) method. Figure 10 (e.g., the first depth map), to obtain a modified depth map 20 (e.g., the second depth map).

[0280] In embodiments of this disclosure, the depth map dynamic changer 1050 may send a modified depth map 20 (e.g., a second depth map) to the 3D transformation actuator 1070. The depth map dynamic changer 1050 may also send the modified depth map 20 (e.g., a second depth map) to the stereoscopic effects controller 1060.

[0281] In embodiments of this disclosure, the stereoscopic effects controller 1060 can receive from the scene object analyzer 1040 the results of analyzing the size and distribution of objects included in each scene constituting the 2D input image 110. The stereoscopic effects controller 1060 can receive from the content category analyzer 1010 the results 510 of analyzing the content categories of the 2D input image 110. The stereoscopic effects controller 1060 can receive the depth of the 2D input image 110 from the depth estimator 1030. Figure 10 (e.g., a first depth map). The stereoscopic effect controller 1060 can receive a modified depth map 20 (e.g., a second depth map) of the 2D input image 110 from the depth map dynamic changer 1050.

[0282] In embodiments of this disclosure, the stereoscopic effect controller 1060 may obtain additional information, including at least one of metadata information about the 2D input image 110, information about the viewing environment, or user settings information related to the 2D input image 110.

[0283] In embodiments of this disclosure, the stereoscopic effect controller 1060 may determine the relative position of objects included in each scene constituting the 2D input image 110 relative to a virtual convergence plane corresponding to the screen based on at least one of the following: the result of analyzing the size and distribution of objects included in each scene, the result of analyzing the content category of the 2D input image 110, or additional information.

[0284] In embodiments of this disclosure, the stereoscopic effects controller 1060 may include appropriate logic, circuitry, interfaces, and / or code capable of operating to determine the relative positions of objects included in each scene constituting the 2D input image 110 with respect to a virtual convergence plane corresponding to the screen.

[0285] In embodiments of this disclosure, the stereoscopic effects controller 1060 can utilize an artificial neural network (e.g., trained to determine the relative positions of objects included in each scene constituting the 2D input image 110 with respect to a virtual convergence plane corresponding to the screen) that has been learned / trained. Figure 14 The system uses at least one of a stereoscopic effect control model 1300, a policy-based algorithm, or a LUT method to determine the relative positions of objects included in each scene constituting the 2D input image 110 with respect to a virtual convergence plane corresponding to the screen.

[0286] In embodiments of this disclosure, the stereoscopic effects controller 1060 may send information to the 3D transformation actuator 1070 regarding the relative positions of objects included in each scene with respect to the convergence plane.

[0287] In embodiments of this disclosure, the 3D transformation actuator 1070 may receive a modified depth map 20 (e.g., a second depth map) from the depth map dynamic changer 1050. In embodiments of this disclosure, the 3D transformation actuator 1070 may receive information from the stereoscopic effects controller 1060 regarding the relative positions of objects included in each scene with respect to the convergence plane.

[0288] In embodiments of this disclosure, the 3D conversion actuator 1070 can obtain a 3D output image 120 by performing a 3D conversion on a 2D input image 110 based on a modified depth map 20 (e.g., a second depth map). In embodiments of this disclosure, the 3D conversion actuator 1070 can be coupled with... Figure 4 The 3D transformation actuator 440 operates in a similar manner. Details related to the reference are omitted here. Figure 4 The given description is a duplicate.

[0289] In embodiments of this disclosure, the 3D transformation actuator 1070 may perform a 3D transformation on the 2D input image 110 based on a modified depth map 20 (e.g., a second depth map) and information about the relative positions of objects included in each scene relative to the convergence plane. The 3D transformation actuator 1070 may perform the 3D transformation on the 2D input image 110 by executing a process of generating a new viewpoint view based on information about the relative positions of objects included in each scene relative to the convergence plane.

[0290] According to embodiments of this disclosure, the image processing device 100 can non-linearly change the depth map and control the stereoscopic effect based on the analysis of the content category of the input 2D image and the analysis of the objects included in each scene. Therefore, it can reduce the depth estimation error, thereby further improving the accuracy of depth estimation and increasing the satisfaction of users using 3D displays.

[0291] However, the effects that can be obtained from this disclosure are not limited to those described above, and other effects not mentioned will be clearly understood by those skilled in the art based on the following description.

[0292] Figure 11 This is a diagram illustrating the process performed by an image processing device 100 to analyze scene objects of a 2D input image 110 according to an embodiment of the present disclosure.

[0293] refer to Figure 11 According to embodiments of the present disclosure, the image processing apparatus 100 (e.g., the scene object analyzer 1040 of the image processing apparatus 100) can analyze the size and distribution of objects included in each scene constituting the 2D input image 110.

[0294] Image processing device 100 can process images based on 2D input image 110 and the depth of 2D input image 110. Figure 10 (For example, a first depth map), the size and distribution of objects included in each scene constituting the 2D input image 110 are analyzed. The image processing device 100 can analyze the size and distribution of objects included in each scene constituting the 2D input image 110 and obtain a result 1110 of analyzing the size and distribution of objects included in each scene for each scene. The "result 1110 of analyzing the size and distribution of objects included in each scene" can be referred to as "information related to the size characteristics of each object included in each scene and the distribution characteristics of each object included in each scene".

[0295] The image processing device 100 can analyze the size of the objects in each scene constituting the 2D input image 110 by calculating the area of ​​pixels occupied by each object in each scene, measuring the size of the objects, and comparing the relative sizes of the objects in a scene.

[0296] Image processing device 100 can process images based on the depth of 2D input image 110. Figure 10 (For example, a first depth map) Analyze the spatial distribution of each object included in each scene that constitutes the 2D input image 110.

[0297] For example, in 1120, it can be assumed that the 2D input image 110 includes scenes A through F. The image processing device 100 can process the image based on the 2D input image 110 and depth. Figure 10 (For example, the first depth map), analyzes whether the size of objects included in scenes A through F is large or small, and whether the distribution of objects is near or far. However, 1120 is only an example, and the image processing device 100 can obtain the result 1110 of analyzing the size and distribution of objects included in each scene constituting the 2D input image 110 through numerical expressions, etc., which can indicate the size and distribution of objects included in each scene.

[0298] According to embodiments of this disclosure, the image processing device 100 can analyze the size and distribution of objects included in each scene constituting the 2D input image 110 by using a scene object analysis model 1100.

[0299] Scene object analysis model 1100 can refer to an artificial neural network model trained to analyze the size and distribution of objects included in each scene constituting the 2D input image 110. For example, scene object analysis model 1100 can refer to, but is not limited to, an artificial neural network model learned / trained to analyze the size and distribution of objects included in each scene constituting the 2D input image 110 using techniques such as CNN, DNN, RNN, RBM, DBN, BRDNN or Deep Q Network, HOG, SHIFT, LSTM, SVM, SoftMax, etc.

[0300] Scene object analysis model 1100 can receive 2D input image 110 and the depth of 2D input image 110. Figure 10 (For example, a first depth map) is used as input data, and the output data is the result 1110 analyzing the size and distribution of objects included in each scene. The image processing device 100 can obtain a trained scene object analysis model 1100 from a cloud server. The image processing device 100 can train / generate / obtain the scene object analysis model 1100 by using on-device learning.

[0301] According to embodiments of this disclosure, the image processing device 100 can analyze the size and distribution of objects included in each scene constituting the 2D input image 110 by using a policy-based algorithm. The policy-based algorithm used to analyze the size and distribution of objects included in each scene constituting the 2D input image 110 can be predefined / set by the manufacturer of the image processing device 100.

[0302] According to embodiments of this disclosure, the image processing device 100 can analyze the size and distribution of objects included in each scene constituting the 2D input image 110 using a hybrid scene object analysis model 1100 and a policy-based algorithm.

[0303] According to embodiments of this disclosure, the image processing device 100 can non-linearly change the depth map and control the stereoscopic effect based on the analysis of the size and distribution of objects included in each scene constituting the input 2D image. This can reduce depth estimation errors, thereby further improving the accuracy of depth estimation and increasing the satisfaction of users using 3D displays.

[0304] However, the effects that can be obtained from this disclosure are not limited to those described above, and other effects not mentioned will be clearly understood by those skilled in the art based on the following description.

[0305] Figures 12A to 12C are diagrams illustrating a process of dynamically changing a depth map performed by an image processing device 100 according to an embodiment of the present disclosure.

[0306] Referring to FIG12A, the image processing apparatus 100 according to an embodiment of the present disclosure (e.g., the depth map dynamic changer 1050 of the image processing apparatus 100) can non-linearly change the depth Figure 10 (e.g., a first depth map) to obtain a modified depth map 20 (e.g., a second depth map).

[0307] Image processing device 100 can process images based on the depth of 2D input image 110. Figure 10 (For example, a first depth map), the result 1110 of analyzing the size and distribution of objects included in each scene constituting the 2D input image 110, the result 510 of analyzing the content categories of the 2D input image 110, or at least one of additional information 1201, by non-linearly changing the depth Figure 10 (e.g., a first depth map) to obtain a modified depth map 20 (e.g., a second depth map).

[0308] In embodiments of this disclosure, the image processing device 100 may obtain additional information 1201. The additional information 1201 may include at least one of metadata information related to the 2D input image 110, information about the viewing environment, or user settings information.

[0309] Metadata information associated with 2D input image 110 can refer to information indicating various details related to 2D input image 110. For example, metadata information associated with 2D input image 110 may include information about the title, description, content category and theme of 2D input image 110; information about the date and time 2D input image 110 was generated; information about the location where 2D input image 110 was taken; information about the creator, editor, copyright owner and usage rights of 2D input image 110; information about the resolution, bitrate and codec of 2D input image 110; information about the color gamut and file format of 2D input image 110; information about the total number of frames or scenes constituting 2D input image 110; and the characters appearing in 2D input image 110, etc.

[0310] Information about the viewing environment can refer to information about the surrounding environment of the user viewing the 3D output image 120. For example, information about the viewing environment may include the illuminance of the user's surrounding environment as measured by sensors in the image processing device 100, or the screen brightness of the display on which the 3D image 120 is displayed.

[0311] User settings information can refer to information reflecting the degree to which a user wants to view the stereoscopic effect of the 2D input image 110, i.e., user preferences related to the stereoscopic effect of the 2D input image 110. For example, user settings information related to the 2D input image 110 may include information about the intensity of the 3D effect manually adjusted by the user through UI 250A and UI 250B of Figure 2C.

[0312] In embodiments of this disclosure, the image processing device 100 may obtain additional information 1201 from another device located outside the image processing device 100, or through a process within the image processing device 100.

[0313] In embodiments of this disclosure, the image processing device 100 can non-linearly change the depth according to 1210A to 1230A. Figure 10 (e.g., a first depth map) to obtain a modified depth map 20 (e.g., a second depth map).

[0314] In 1210A, the image processing apparatus 100 according to an embodiment of the present disclosure can generate an offset for each object included in each scene constituting the 2D input image 110 based on the result 1110 of analyzing the size and distribution of objects included in each scene. The offset may refer to the offset relative to the constituting depth. Figure 10 The deviation of depth information of each object's pixels included in (e.g., a first depth map), and may include positive or negative deviations.

[0315] In 1220A, the image processing apparatus 100 according to embodiments of the present disclosure can adjust the depth of the 2D input image 110 by generating an offset for each object included in each scene. Figure 10 The depth information included in (for example, the first depth map) changes non-linearly. Figure 10 (For example, the first depth map).

[0316] In 1230A, the image processing apparatus 100 according to embodiments of the present disclosure can control the output range of the depth map based on at least one of the result 510 of analyzing the content category of the 2D input image 110 or the obtained supplementary information 1201. The output range of the depth map can refer to the range from the minimum to the maximum value of the depth information of the pixels included in the depth map. In embodiments of the present disclosure, the image processing apparatus 100 can control the output range of the depth map by preferentially considering the information included in the supplementary information 1201 rather than the result 510 of analyzing the content category of the 2D input image 110.

[0317] For example, referring to Figure 12B, the input depth map can be depth Figure 10 (e.g., a first depth map), and the output depth map can be a modified depth map 20 (e.g., a second depth map). The image processing device 100 can obtain the output depth map by non-linearly changing the input depth map via 1210B to 1230B.

[0318] For example, in 1210B, the image processing device 100 can generate positive offsets for some objects included in a specific scene constituting the 2D input image 110 and negative offsets for other objects, based on the result 1110 of analyzing the size and distribution of objects included in each scene constituting the 2D input image 110.

[0319] For example, in 1220B, the image processing device 100 can adjust the depth information of the pixels of the corresponding objects included in the input depth map by the offset generated in 1210B. Adjusting the depth information by a positive offset can mean adjusting the object to be more prominent, and adjusting the depth information by a negative offset can mean adjusting the object to be further back.

[0320] For example, in 1230B, the image processing device 100 can obtain information about the viewing environment as supplementary information 1201, indicating that the user's surrounding environment is a dark room with an illumination close to 0 lux. In this case, the image processing device 100 can control the overall output range of the depth map reflecting the adjusted depth information in 1220B to 0.5 times to reduce user fatigue.

[0321] For example, referring to FIG12C, in 1221, the image processing device 100 can non-linearly change the input depth map by applying a positive bias to the depth information of the pixels constituting the human object and a negative bias to the depth information of the pixels constituting the background object. In 1222, the image processing device 100 can determine that the content category is text based on the result 510 of analyzing the content category or additional information 1201, and control the overall output range of the depth map to 0. In 1223, the image processing device 100 can non-linearly change the input depth map by applying a positive bias to the depth information of the pixels constituting the building object and a negative bias to the depth information of the pixels constituting the background object.

[0322] Referring back to FIG12A, according to an embodiment of the present disclosure, the image processing device 100 can dynamically change the model 1200 by using a depth map, thereby non-linearly changing the depth. Figure 10 (e.g., a first depth map) to obtain a modified depth map 20 (e.g., a second depth map).

[0323] The depth map dynamic change model 1200 can refer to models trained to change depth non-linearly. Figure 10 (For example, the first depth map) an artificial neural network model. For example, a depth map dynamic change model 1200 can refer to a model trained to non-linearly change the depth using techniques such as CNN, DNN, RNN, RBM, DBN, BRDNN or deep Q network, HOG, SHIFT, LSTM, SVM, SoftMax, etc. Figure 10 Artificial neural network models (e.g., the first depth map), but not limited to these.

[0324] The depth map dynamically changes the model 1200, which can receive the depth of a 2D input image 110. Figure 10 The image processing device 100 takes a modified depth map 20 (e.g., a first depth map) as input data and outputs a modified depth map 20 (e.g., a second depth map) of the 2D input image 110 as output data. The image processing device 100 can obtain a trained depth map dynamic modification model 1200 from a cloud server. The image processing device 100 can train / generate / obtain the depth map dynamic modification model 1200 by using on-device learning.

[0325] According to embodiments of this disclosure, the image processing device 100 can non-linearly change the depth by using a policy-based algorithm. Figure 10 (e.g., a first depth map) to obtain a modified depth map 20 (e.g., a second depth map). This is used to non-linearly change the depth. Figure 10 The policy-based algorithm for (e.g., the first depth map) can be predefined / set by the manufacturer of the image processing device 100.

[0326] According to embodiments of this disclosure, the image processing device 100 can non-linearly change the depth by using a LUT method. Figure 10 (e.g., a first depth map) to obtain a modified depth map 20 (e.g., a second depth map). Specific functions or mapping tables used for the LUT method can be predefined / set by the manufacturer of the image processing device 100. For example, the image processing device 100 can use a mapping table that uniformly adjusts the depth using predefined offsets. Figure 10 (For example, a first depth map) Depth information of pixels corresponding to a specific range. For example, in a scene with a specific object distribution, the image processing device 100 can use a mapping table that uniformly adjusts the depth information of pixels constituting scene objects by predefined offsets.

[0327] According to embodiments of this disclosure, the image processing device 100 can dynamically change two or more of the following: model 1200, policy-based algorithm, and LUT method, by mixing depth maps, thereby non-linearly changing the depth. Figure 10 (e.g., a first depth map) to obtain a modified depth map 20 (e.g., a second depth map).

[0328] According to embodiments of this disclosure, the image processing device 100 can non-linearly change the depth map, thereby reducing depth estimation errors, further improving the accuracy of depth estimation, and increasing user satisfaction with 3D displays.

[0329] However, the effects that can be obtained from this disclosure are not limited to those described above, and other effects not mentioned will be clearly understood by those skilled in the art based on the following description.

[0330] Figure 13 and Figure 14 This is a diagram illustrating the process of controlling a stereoscopic effect performed by an image processing device 100 according to an embodiment of the present disclosure.

[0331] The operation of controlling "depth" has been described with reference to Figures 12A to 12C, and it is one of the main factors affecting the stereoscopic effect. Figures 13 to 14This describes the operation of adjusting the "prominence relative to the screen (negative parallax)" and "recession relative to the screen (positive parallax)," which are another major factor affecting stereoscopic effects. Prominence (or popping out, negative parallax) means that the object protrudes from above the screen, while recession (positive parallax) means that the object is located behind the screen. Focus (zero parallax) means that the object is on a virtual convergence plane corresponding to the screen.

[0332] refer to Figure 13 According to embodiments of the present disclosure, the image processing apparatus 100 (e.g., the stereoscopic effect controller 1060 of the image processing apparatus 100) can determine the relative positions of objects included in each scene constituting the 2D input image 110 with respect to a virtual convergence plane corresponding to the screen.

[0333] Image processing device 100 can process images based on at least one of the following: the result 1110 of analyzing the size and distribution of objects included in each scene; the result 510 of analyzing the content categories of the 2D input image 110; or additional information 1201 obtained; or based on the depth of the 2D input image 110. Figure 10 Alternatively, by modifying at least one of the depth maps 20, the position of a virtual convergence plane corresponding to the screen can be determined. When the position of the virtual convergence plane corresponding to the screen is determined, the image processing device 100 can determine the relative position of objects included in each scene constituting the 2D input image 110 relative to the convergence plane. The relative position of objects included in each scene relative to the convergence plane can be determined as prominent, focused, or behind the screen.

[0334] For example, when the object is large, its distribution is in the foreground, and the foreground and background are clearly distinguishable, the image processing device 100 can determine the object's relative position to the convergence plane as prominent based on the analysis results 1110 of the size and distribution of objects included in each scene. For example, when the object is small, its distribution is in the background, the image processing device 100 can determine the object's relative position to the convergence plane as behind the screen based on the analysis results 1110 of the size and distribution of objects included in each scene. For example, when the content category of the 2D input image 110 is a document, the image processing device 100 can determine the relative position of the objects included in the 2D input image 110 relative to the convergence plane based on the analysis results 510 of the content category of the 2D input image 110 or the obtained additional information 1201.

[0335] For example, refer to Figure 14 1410 to 1460 show at least one of the following: results 1110 based on the analysis of the size and distribution of objects included in each scene; results 510 based on the analysis of the content categories of the 2D input image 110; or additional information 1201 obtained; or based on the depth of the 2D input image 110. Figure 10 Alternatively, at least one of the depth maps 20 may be modified to determine the relative positions of objects included in each scene constituting the 2D input image 110 with respect to a virtual convergence plane corresponding to the screen.

[0336] Return to reference Figure 13 According to embodiments of the present disclosure, the image processing device 100 can determine the relative positions of objects included in each scene constituting the 2D input image 110 with respect to a virtual convergence plane corresponding to the screen by using a stereoscopic effect control model 1300.

[0337] The stereoscopic effect control model 1300 can refer to an artificial neural network model trained to determine the relative positions of objects included in each scene constituting the 2D input image 110 with respect to a virtual convergence plane corresponding to the screen. For example, the stereoscopic effect control model 1300 can refer to, but is not limited to, an artificial neural network model learned / trained to determine the relative positions of objects included in each scene constituting the 2D input image 110 with respect to a virtual convergence plane corresponding to the screen using techniques such as CNN, DNN, RNN, RBM, DBN, BRDNN or Deep Q Network, HOG, SHIFT, LSTM, SVM, SoftMax, etc.

[0338] The stereoscopic effect control model 1300 can receive at least one of the following: the results 1110 of analyzing the size and distribution of objects included in each scene; the results 510 of analyzing the content categories of the 2D input image 110; or the obtained additional information 1201; or the depth of the 2D input image 110. Figure 10 Alternatively, at least one of the depth maps 20 can be modified as input data, and information about the relative positions of objects included in each scene with respect to the convergence plane can be output as output data. The image processing device 100 can obtain a trained stereo effect control model 1300 from a cloud server. The image processing device 100 can train / generate / obtain the stereo effect control model 1300 by using on-device learning.

[0339] According to embodiments of this disclosure, the image processing device 100 can determine the relative positions of objects included in each scene constituting the 2D input image 110 with respect to a virtual convergence plane corresponding to the screen by using a policy-based algorithm. The policy-based algorithm for determining the relative positions of objects included in each scene constituting the 2D input image 110 with respect to the virtual convergence plane corresponding to the screen can be predefined / set by the manufacturer of the image processing device 100.

[0340] According to embodiments of this disclosure, the image processing device 100 can determine the relative positions of objects included in each scene constituting the 2D input image 110 with respect to a virtual convergence plane corresponding to the screen by using a LUT method. Specific functions or mapping tables used for the LUT method can be predefined / set by the manufacturer of the image processing device 100. For example, in the case of scenes with specific content categories, the image processing device 100 can use a mapping table that determines the relative positions of objects included in the scene with respect to the virtual convergence plane as predefined positions.

[0341] According to embodiments of this disclosure, the image processing device 100 can determine the relative positions of objects included in each scene constituting the 2D input image 110 with respect to a virtual convergence plane corresponding to the screen by using two or more of the hybrid stereo effect control model 1300, a policy-based algorithm, and a LUT method.

[0342] According to embodiments of this disclosure, the image processing device 100 can determine whether to make objects included in each scene constituting a 2D input image stand out above the screen or behind the screen, thereby reducing depth estimation errors, further improving the accuracy of depth estimation, and increasing user satisfaction with 3D displays.

[0343] However, the effects that can be obtained from this disclosure are not limited to those described above, and other effects not mentioned will be clearly understood by those skilled in the art based on the following description.

[0344] Figure 15 This is a flowchart describing a method 1500 performed by an image processing device 100 to perform 3D conversion on a 2D input image 110 according to embodiments of the present disclosure.

[0345] refer to Figure 15 In operation 1510, the image processing apparatus 100 according to an embodiment of the present disclosure can analyze the content category of the input image 110. Operation 1510 can be combined with... Figure 9 The operation is similar to that in 910. Details related to the reference are omitted here. Figure 9 The given description is a duplicate.

[0346] In operation 1520, the image processing apparatus 100 according to an embodiment of the present disclosure can obtain a depth estimation model corresponding to the input image 110 in real time based on the result of analyzing the content category of the input image 110.

[0347] Image processing device 100 can obtain a depth estimation model corresponding to input image 110 in real time via a cloud server based on the result 510 of analyzing the content category of input image 110. Image processing device 100 can also train / generate / obtain one or more depth estimation models corresponding to input image 110 in real time by using on-device learning based on the result 510 of analyzing the content category of input image 110.

[0348] When the image processing device 100 uses on-device learning, operation 1520 can... Figure 9 The operation is similar to that in 920. Details related to the reference are omitted here. Figure 9 The given description is a duplicate.

[0349] In operation 1530, the image processing device 100 can obtain the depth of the input image 110 based on a depth estimation model. Figure 10 Operation 1530 can be used with... Figure 9 The operation is similar to that in 930. Details related to the reference are omitted here. Figure 9 The given description is a duplicate.

[0350] In operation 1540, the image processing device 100 according to an embodiment of the present disclosure can process the image based on the input image 110 and the depth of the input image 110. Figure 10 The size and distribution of objects in the scene constituting the input image 110 are analyzed respectively.

[0351] According to an embodiment of the present disclosure, the image processing apparatus 100 can analyze the size of objects included in the scenes constituting the 2D input image 110 by calculating the area of ​​pixels occupied by each object included in each scene constituting the input image 110, measuring the size of the objects, and comparing the relative sizes of objects included in a scene.

[0352] According to embodiments of the present disclosure, the image processing device 100 can process an input image 110 based on its depth. Figure 10 The distribution of objects in the scene that constitutes the input image 110 in space is analyzed.

[0353] According to an embodiment of the present disclosure, the image processing apparatus 100 can analyze the size and distribution of objects included in the scene constituting the input image 110 by using a scene object analysis model 1100.

[0354] Scene object analysis model 1100 can refer to an artificial neural network model trained to analyze the size and distribution of objects included in the scene constituting the input image 110.

[0355] The image processing device 100 according to embodiments of the present disclosure can analyze the size and distribution of objects included in the scene constituting the input image 110 using a policy-based algorithm. The policy-based algorithm for analyzing the size and distribution of objects included in the scene constituting the input image 110 can be predefined / set by the manufacturer of the image processing device 100.

[0356] According to embodiments of the present disclosure, the image processing apparatus 100 can analyze the size and distribution of objects included in the scene constituting the input image 110 using a hybrid scene object analysis model 1100 and a policy-based algorithm, respectively.

[0357] In operation 1550, the image processing apparatus 100 according to embodiments of the present disclosure can, based on the results 1110 of analyzing the size and distribution of objects included in the scene, non-linearly change the depth. Figure 10 To obtain the modified depth map 20.

[0358] The image processing apparatus 100 according to embodiments of the present disclosure can obtain additional information 1201. In embodiments of the present disclosure, the additional information 1201 may include at least one of metadata information related to the input image 110, information about the viewing environment, or user settings information.

[0359] According to embodiments of the present disclosure, the image processing device 100 can process an input image 110 based on its depth. Figure 10 The results of analyzing the size and distribution of objects in the scene constituting the input image 110, the results of analyzing the content categories of the input image 110, or additional information 1201, are at least one of the following, and the depth is changed non-linearly. Figure 10 To obtain the modified depth map 20.

[0360] According to an embodiment of the present disclosure, the image processing apparatus 100 can generate offsets for objects included in the scene constituting the input image 110 based on the result 1110 of analyzing the size and distribution of objects included in the scene.

[0361] According to embodiments of the present disclosure, the image processing device 100 can adjust the depth of the input image 110 by generating offsets for objects included in the scene. Figure 10 The depth information included in it changes non-linearly. Figure 10 .

[0362] According to embodiments of the present disclosure, the image processing apparatus 100 can control the output range of a depth map based on at least one of the result 510 of analyzing the content category of the input image 110 or the obtained supplementary information 1201. The output range of the depth map can refer to the range from the minimum to the maximum value of the depth information of the pixels included in the depth map. In embodiments of the present disclosure, the image processing apparatus 100 can control the output range of the depth map by prioritizing the information included in the supplementary information 1201 rather than the result 510 of analyzing the content category of the input image 110.

[0363] According to embodiments of this disclosure, the image processing device 100 can dynamically change the model 1200 using a depth map, by non-linearly changing the depth. Figure 10 To obtain the modified depth map 20.

[0364] The depth map dynamic change model 1200 can refer to models trained to change depth non-linearly. Figure 10 Artificial neural network models.

[0365] According to embodiments of this disclosure, the image processing device 100 can non-linearly change the depth by using a policy-based algorithm. Figure 10 To obtain the modified depth map 20. Used to non-linearly change the depth. Figure 10 The policy-based algorithm can be predefined / set by the manufacturer of the image processing device 100.

[0366] According to embodiments of this disclosure, the image processing device 100 can non-linearly change the depth by using a LUT method. Figure 10 To obtain the modified depth map 20. The specific functions or mapping tables used for the LUT method can be predefined / set by the manufacturer of the image processing device 100.

[0367] According to embodiments of this disclosure, the image processing device 100 can dynamically change two or more of the following: model 1200, policy-based algorithm, and LUT method, by mixing depth maps, thereby non-linearly changing the depth. Figure 10 To obtain the modified depth map 20.

[0368] In operation 1560, the image processing apparatus 100 according to embodiments of the present disclosure may, based on at least one of the results 510 of analyzing the content category of the input image 110 or the results 1110 of analyzing the size and distribution of objects included in the scene, and the depth of the input image 110. Figure 10 Alternatively, by modifying at least one of the depth maps 20, the relative positions of objects included in the scene with respect to a virtual convergence plane corresponding to the screen can be determined.

[0369] According to an embodiment of the present disclosure, the image processing device 100 can determine the relative positions of objects included in the scene constituting the input image 110 with respect to a virtual convergence plane corresponding to the screen by using a stereoscopic effect control model 1300.

[0370] The stereoscopic effect control model 1300 can refer to an artificial neural network model trained to determine the relative positions of objects included in the scene constituting the 2D input image 110 with respect to a virtual convergence plane corresponding to the screen.

[0371] The image processing device 100 according to an embodiment of the present disclosure can determine the relative positions of objects included in a scene constituting an input image 110 with respect to a virtual convergence plane corresponding to the screen by using a policy-based algorithm. The policy-based algorithm for determining the relative positions of objects included in a scene constituting an input image 110 with respect to a virtual convergence plane corresponding to the screen can be predefined / set by the manufacturer of the image processing device 100.

[0372] According to an embodiment of the present disclosure, the image processing device 100 can determine the relative positions of objects included in the scene constituting the input image 110 with respect to a virtual convergence plane corresponding to the screen by using a LUT method.

[0373] The image processing device 100 according to an embodiment of the present disclosure can determine the relative positions of objects included in the scene constituting the input image 110 with respect to a virtual convergence plane corresponding to the screen by two or more of a hybrid stereoscopic effect control model 1300, a policy-based algorithm, and a LUT method.

[0374] In operation 1570, the image processing apparatus 100 according to an embodiment of the present disclosure may perform a 3D transformation on the input image 110 based on a modified depth map 20 and information 1310 regarding the relative positions of objects included in the scene with respect to a virtual convergence plane.

[0375] According to an embodiment of the present disclosure, the image processing apparatus 100 can perform 3D transformation on the input image 110 by executing a process 370 for generating a new viewpoint view, a hole filling process 380, and a pixel mapping process 390, based on a modified depth map 20 and information 1310 about the relative positions of objects included in the scene relative to a virtual convergence plane.

[0376] According to embodiments of this disclosure, the image processing device 100 can non-linearly change the depth map and control the stereoscopic effect based on the analysis of the size and distribution of objects included in the scene constituting the input 2D image. This can reduce depth estimation errors, thereby further improving the accuracy of depth estimation and increasing the satisfaction of users using 3D displays.

[0377] However, the effects that can be obtained from this disclosure are not limited to those described above, and other effects not mentioned will be clearly understood by those skilled in the art based on the following description.

[0378] Figure 16 This is a diagram illustrating an example of the effect of an image processing apparatus 100 according to an embodiment of the present disclosure.

[0379] refer to Figure 16 When the 2D input image 110 is a complex image including scenes with various content categories, the image processing device 100 according to embodiments of the present disclosure can provide a 2D to 3D conversion method that allows users to watch for extended periods of time by providing strong stereoscopic effects in FPS games, providing medium stereoscopic effects in RPG games, appropriately adjusting stereoscopic effects according to scenes in TV series and movie content, and reducing stereoscopic effects to 2D level in documents, taking into account the stereoscopic effects of each scene and user fatigue.

[0380] Specifically, the method provided by the image processing device 100 according to embodiments of this disclosure differs from prior art, which continuously provides a fixed stereoscopic effect to an input image. The image processing device 100 according to embodiments of this disclosure can dynamically control the stereoscopic effect for each content category to provide dynamic depth based on the content category, non-linearly adjust the depth map according to the size and distribution of objects in each scene, and determine the relative position of the screen. Therefore, it can reduce depth estimation errors, thereby improving the accuracy of depth estimation, and reducing user fatigue even during prolonged viewing.

[0381] Figure 17 This is a block diagram of an image processing apparatus 100 according to an embodiment of the present disclosure.

[0382] refer to Figure 17 The image processing device 100 according to embodiments of the present disclosure may include at least one processor 101 and a memory 102.

[0383] Memory 102 may store one or more instructions for performing the 3D conversion functions disclosed herein. Memory 102 may store at least one program executed by processor 101. At least one neural network and / or a predefined policy-based algorithm or neural network model may be stored in memory 102. Additionally, memory 102 may store data input to or output from image processing device 100.

[0384] The memory 102 may include at least one type of storage medium selected from flash memory, hard disk memory, multimedia card micro memory, card memory (e.g., SD or XD memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, and optical disk.

[0385] At least one processor 101 can control the overall operation of the image processing device 100. At least one processor 101 can execute one or more instructions stored in the memory 102 to control the image processing device 100 to perform its functions.

[0386] For example, at least one processor 101 can execute one or more instructions stored in memory 120. Figures 1 to 16 The functions of the image processing device 100 shown.

[0387] At least one processor 101 can be configured as one or more processors. In this case, the one or more processors can be general-purpose processors (e.g., CPU, AP, or digital signal processor (DSP)), graphics-specific processors (e.g., GPU or Visual Processing Unit (VPU)), or artificial intelligence-specific processors (e.g., NPU). For example, when one or more processors are artificial intelligence-specific processors, the artificial intelligence-specific processors can be designed with hardware architectures specifically designed to process specific artificial intelligence models.

[0388] According to embodiments of this disclosure, at least one processor 101 can execute one or more instructions stored in memory 102 to analyze the content categories of an input image. At least one processor 101 can execute one or more instructions stored in memory 102 to obtain, in real time, a depth estimation model corresponding to the content categories of the input image by using on-device learning, based on the results of analyzing the content categories of the input image. At least one processor 101 can execute one or more instructions stored in memory 102 to obtain a depth map of the input image reflecting the estimated depth information based on the depth estimation model. At least one processor 101 can perform a 3D transformation on the input image based on the depth map of the input image.

[0389] According to embodiments of this disclosure, the results of analyzing the content category of an input image may include information related to the probability that the input image corresponds to a predefined content category.

[0390] According to embodiments of this disclosure, by updating the parameters of a depth estimation model and training the depth estimation model on an image processing device based on the results of analyzing the content category of the input image, a depth estimation model corresponding to the input image can be obtained in real time.

[0391] According to embodiments of this disclosure, a depth estimation model corresponding to the input image 110 is obtained in real time by interpolating the depth estimation model on the image processing device 100 based on the results of analyzing the content category of the input image.

[0392] According to embodiments of this disclosure, at least one processor 101 can execute one or more instructions stored in memory 102 to obtain, in real time, depth estimation models corresponding to the scenes constituting the input image, based on the results of analyzing the content categories of the input image. At least one processor 101 can also execute one or more instructions stored in memory 102 to obtain depth maps of the input image in real time by applying depth estimation models corresponding to the scenes constituting the input image to the scenes constituting the input image.

[0393] According to embodiments of this disclosure, at least one processor 101 can execute one or more instructions stored in memory 102 to analyze the size and distribution of objects included in a scene constituting the input image, based on an input image and a depth map of the input image, respectively. At least one processor 101 can execute one or more instructions stored in memory 102 to obtain a modified depth map by non-linearly altering the depth map based on the results of the analysis of the size and distribution of objects included in the scene. At least one processor 101 can execute one or more instructions stored in memory 102 to perform a 3D transformation on the input image based on the modified depth map.

[0394] According to embodiments of this disclosure, at least one processor 101 can execute one or more instructions stored in memory 102 to obtain additional information, including at least one of metadata information about the input image, information about the viewing environment, or user settings information. At least one processor 101 can execute one or more instructions stored in memory 102 to control the output range of a depth map or a modified depth map based on at least one of the obtained additional information or the result of analyzing the content category of the input image.

[0395] According to embodiments of this disclosure, at least one processor 101 can execute one or more instructions stored in memory 102 to generate offsets for objects included in the scene constituting the input image, based on the results of analyzing the size and distribution of objects included in the scene. At least one processor 101 can also execute one or more instructions stored in memory 102 to adjust depth information included in the depth map of the input image using offsets generated for objects included in the scene, thereby obtaining a modified depth map by non-linearly altering the depth map.

[0396] According to embodiments of this disclosure, at least one processor 101 can execute one or more instructions stored in memory 102 to determine the relative positions of objects in the scene relative to a virtual convergence plane corresponding to the screen, based on at least one of the following: results of analyzing the content category of an input image, results of analyzing the size and distribution of objects included in the scene, or additional information obtained. At least one processor 101 can also execute one or more instructions stored in memory 102 to perform a 3D transformation on the input image based on information regarding the relative positions of objects included in the scene relative to the convergence plane.

[0397] According to embodiments of the present disclosure, at least one processor 101 may execute one or more instructions stored in memory 102 to control a display to generate and output a UI that indicates content categories corresponding to scenes constituting a 3D output image converted from an input image and stereoscopic effects for each scene.

[0398] The specific examples used to describe embodiments of this disclosure are merely single combinations of standards, methods, detailed methods, and operations. By combining at least two or more of the various techniques described, the image processing device 100 can dynamically control the stereoscopic effect for each content category to provide dynamic depth according to the content category, non-linearly adjust the depth map according to the size and distribution of objects in each scene, and control the stereoscopic effect, thereby reducing depth estimation errors, improving the accuracy of depth estimation, and reducing user fatigue even when the user views for a long time.

[0399] Furthermore, the 2D to 3D conversion method of this disclosure can be performed according to a method determined by one or at least a combination of the above-described techniques. For example, some operations of one embodiment can be combined with some operations of another embodiment and performed.

[0400] Machine-readable storage media can be provided as non-transitory storage media. Here, "non-transitory" means that the storage medium does not include signals (e.g., electromagnetic waves) and is tangible, but does not distinguish whether the data is stored semi-permanently or temporarily in the storage medium. For example, a "non-transitory storage medium" may include a buffer for temporarily storing data.

[0401] According to embodiments of this disclosure, methods according to various embodiments of this disclosure can be provided in a computer program product. A computer program product is a product tradable between a seller and a buyer. The computer program product can be distributed in the form of a machine-readable storage medium (e.g., a compact disc (CD)-ROM), or distributed online via an app store (e.g., downloaded or uploaded), or directly between two user devices (e.g., smartphones). When distributed online, at least a portion of the computer program product (e.g., a downloadable application) can be temporarily generated or at least temporarily stored in a machine-readable storage medium (e.g., the memory of a manufacturer's server, an app store's server, or a relay server).

Claims

1. An image processing apparatus, comprising: Memory, used to store one or more instructions; as well as At least one processor is configured to execute the one or more instructions stored in the memory to: Analyze the content category of the input image. Based on the analysis of the content categories of the input image, a depth estimation model corresponding to the content categories of the input image is obtained in real time using on-device learning. Based on the depth estimation model learned on the device, a depth map reflecting the estimated depth information of the input image is obtained, and Based on the depth map of the input image, a 3D transformation is performed on the input image.

2. The image processing apparatus according to claim 1, wherein, The results of analyzing the content category of the input image include information related to the probability that the input image corresponds to a predefined content category.

3. The image processing apparatus according to claim 1 or 2, wherein, The depth estimation model is obtained in real time by updating the parameters of the depth estimation model and training the depth estimation model on the image processing device based on the results of analyzing the content category of the input image.

4. The image processing apparatus according to any one of claims 1 to 3, wherein, The depth estimation model is obtained in real time by interpolating the depth estimation model on the image processing device based on the results of analyzing the content category of the input image.

5. The image processing apparatus according to any one of claims 1 to 4, wherein, The at least one processor is configured to execute the one or more instructions to: Based on the analysis of the content categories of the input image, depth estimation models corresponding to the scenes constituting the input image are obtained in real time, and By applying depth estimation models corresponding to the scenes constituting the input image to the scenes constituting the input image, a depth map of the input image is obtained in real time.

6. The image processing apparatus according to any one of claims 1 to 5, wherein, The at least one processor is further configured to execute the one or more instructions to: Based on the input image and its depth map, the size and distribution of objects in the scene constituting the input image are analyzed. Based on the analysis of the size and distribution of objects included in the scene, a modified depth map is obtained by non-linearly altering the depth map. Based on the modified depth map, a 3D transformation is performed on the input image.

7. The image processing apparatus according to any one of claims 1 to 6, wherein, The at least one processor is further configured to execute the one or more instructions to: Obtain additional information, including at least one of metadata information about the input image, information about the viewing environment of the input image, or user settings information, and The output range of the depth map or the modified depth map is controlled based on at least one of the additional information obtained or the results of analyzing the content category of the input image.

8. The image processing apparatus according to claim 6 or 7, wherein, The at least one processor is further configured to execute the one or more instructions to: Based on the analysis of the size and distribution of objects in the scene constituting the input image, offsets are generated for each object in the scene, and... The modified depth map is obtained by adjusting the depth information included in the depth map of the input image based on the offset generated for the objects included in the scene, and by non-linearly changing the depth map.

9. The image processing apparatus according to any one of claims 1 to 8, wherein, The at least one processor is further configured to execute the one or more instructions to: Based on at least one of the following: the results of analyzing the content category of the input image, the results of analyzing the size and distribution of objects included in the scene, or additional information obtained, the relative positions of the objects included in the scene with respect to a virtual convergence plane corresponding to the screen are determined. The input image is transformed into 3D based on information about the relative positions of objects in the scene with respect to the virtual convergence plane.

10. The image processing apparatus according to any one of claims 1 to 9, wherein, The at least one processor is further configured to execute the one or more instructions to: The display is controlled to generate and output a user interface (UI), which indicates the content category corresponding to the scene constituting the 3D output image converted from the input image and the stereoscopic effect of the scene.

11. A method performed by an image processing device, the method comprising: Analyze the content categories of the input image; Based on the analysis of the content categories of the input image, a depth estimation model corresponding to the content categories of the input image is obtained in real time by using on-device learning. Based on the depth estimation model learned on the device, a depth map reflecting the estimated depth information of the input image is obtained; as well as Based on the depth map of the input image, a 3D transformation is performed on the input image.

12. The method according to claim 11, wherein, The depth estimation model is obtained in real time by updating the parameters of the depth estimation model and training the depth estimation model on the image processing device based on the results of analyzing the content category of the input image.

13. The method according to claim 11 or 12, wherein, The depth estimation model is obtained in real time by interpolating the depth estimation model on the image processing device based on the results of analyzing the content category of the input image.

14. The method according to any one of claims 11 to 13, further comprising: Based on the analysis of the content categories of the input image, depth estimation models corresponding to the scenes constituting the input image are obtained in real time. as well as By applying depth estimation models corresponding to the scenes constituting the input image to the scenes constituting the input image, a depth map of the input image is obtained in real time.

15. A computer-readable recording medium having recorded thereon at least one program for implementing the method according to any one of claims 11 to 14.

Images