Electronic device and controlling method of electronic device

Abstract

An electronic device and a method for controlling the electronic device are disclosed. In particular, the method for controlling the electronic device according to the present disclosure comprises the steps of: acquiring a plurality of images through at least one camera; inputting RGB data for each of the plurality of images into a first neural network model to acquire two-dimensional pose information for an object included in the plurality of images; inputting RGB data for at least one of the plurality of images into a second neural network model to identify whether the object is transparent; if the object is a transparent object, performing stereo matching based on two-dimensional pose information for each of the plurality of images to acquire three-dimensional pose information for the object; and if the object is an opaque object, acquiring three-dimensional pose information for the object based on one of the plurality of images and depth information corresponding to that image.

Description

Technology Field

[0001] The present disclosure relates to an electronic device and a method for controlling an electronic device, and specifically refers to a device capable of obtaining three-dimensional pose information for an object included in an image. Background Technology

[0003] Recently, the need for technology to obtain three-dimensional pose information about objects included in images has been highlighted. In particular, the development of technology to detect objects included in images using neural network models such as CNN (Convolutional Neural Network) and to obtain three-dimensional pose information about the detected objects has been accelerating.

[0004] However, it has been pointed out that when obtaining pose information for an object based on a single image according to conventional technology, it is difficult to obtain pose information for an object for which a 3D model has not been built, and in particular, it is difficult to obtain accurate pose information for transparent objects.

[0005] In addition, limitations have been pointed out regarding the acquisition of pose information for an object based on stereo cameras according to conventional technology, such as the fact that the range of distances that can be measured for acquiring pose information is limited because the difference in field of view between the two cameras is narrow, and that a neural network model trained on the premise that the positional relationship between the two cameras is fixed cannot be used when the positional relationship between the two cameras changes. The problem to be solved

[0007] The present disclosure is intended to overcome the limitations of the prior art as described above, and the purpose of the present disclosure is to provide an electronic device and a method for controlling the electronic device that can acquire three-dimensional pose information of an object in an efficient manner according to the characteristics of the object included in an image. means of solving the problem

[0009] According to one embodiment of the present disclosure for achieving the purpose described above, the electronic device includes at least one camera, a memory, and a processor that acquires a plurality of images through the at least one camera, inputs RGB data for each of the plurality of images into a first neural network model to acquire two-dimensional pose information for an object included in the plurality of images, inputs RGB data for at least one of the plurality of images into a second neural network model to identify whether the object is transparent, and if the object is a transparent object, performs stereo matching based on the two-dimensional pose information for each of the plurality of images to acquire three-dimensional pose information for the object, and if the object is an opaque object, acquires three-dimensional pose information for the object based on one of the plurality of images and depth information corresponding to the one image.

[0010] Here, the processor can obtain information regarding whether the object is transparent through the second neural network model and identify whether the object is transparent based on the information regarding whether the object is transparent.

[0011] Meanwhile, the processor can obtain information regarding whether the object is symmetrical through the second neural network model, identify whether the object is symmetrical based on the information regarding whether the object is symmetrical, and if the object is symmetrical, convert the first feature points included in the two-dimensional pose information into second feature points independent of symmetry, and perform stereo matching based on the second feature points to obtain three-dimensional pose information for the object.

[0012] Meanwhile, if the object has symmetry, the first feature points can be identified based on a three-dimensional coordinate system in which the x-axis or y-axis is perpendicular to the at least one camera.

[0013] Meanwhile, the plurality of images may be two images acquired at different times through the first camera among the at least one camera.

[0014] Meanwhile, the plurality of images may be two images acquired at the same time through each of the first camera and the second camera among the at least one camera.

[0015] Here, the processor can obtain first position information regarding the positional relationship between the first camera and the second camera, and perform stereo matching based on the plurality of two-dimensional pose information for each image and the first position information.

[0016] Here, the electronic device further includes a driving unit, and the processor controls the driving unit so that the position of at least one of the first camera and the second camera is changed, obtains second position information regarding the positional relationship between the first camera and the second camera based on the changed position of at least one camera, and can perform stereo matching based on the plurality of two-dimensional pose information for each image and the second position information.

[0017] Meanwhile, the first neural network model and the second neural network model may be included in a single integrated neural network model.

[0018] According to one embodiment of the present disclosure for achieving the purpose described above, a control method for an electronic device comprises the steps of: acquiring a plurality of images through at least one camera; inputting RGB data for each of the plurality of images into a first neural network model to acquire two-dimensional pose information for an object included in the plurality of images; inputting RGB data for at least one of the plurality of images into a second neural network model to identify whether the object is transparent; if the object is a transparent object, performing stereo matching based on the two-dimensional pose information for each of the plurality of images to acquire three-dimensional pose information for the object; and if the object is an opaque object, acquiring three-dimensional pose information for the object based on one of the plurality of images and depth information corresponding to the one image.

[0019] Here, the step of identifying whether the object is transparent may include the step of obtaining information regarding whether the object is transparent through the second neural network model and the step of identifying whether the object is transparent based on the information regarding whether the object is transparent.

[0020] Meanwhile, the step of identifying whether the object is symmetrical includes the step of obtaining information regarding whether the object is symmetrical through the second neural network model and the step of identifying whether the object is symmetrical based on the information regarding whether the object is symmetrical, and the control method of the electronic device may further include the step of converting first feature points included in the two-dimensional pose information into second feature points independent of symmetry and the step of obtaining three-dimensional pose information for the object by performing stereo matching based on the second feature points if the object is an object having symmetry.

[0021] Meanwhile, if the object has symmetry, the first feature points can be identified based on a three-dimensional coordinate system in which the x-axis or y-axis is perpendicular to the at least one camera.

[0022] Meanwhile, the plurality of images may be two images acquired at different times through the first camera among the at least one camera.

[0023] Meanwhile, the plurality of images may be two images acquired at the same time through each of the first camera and the second camera among the at least one camera.

[0024] Here, the control method of the electronic device may further include the step of obtaining first position information regarding the positional relationship between the first camera and the second camera, and the step of performing stereo matching based on the plurality of two-dimensional pose information for each image and the first position information.

[0025] Here, the control method of the electronic device may further include the step of controlling the position of at least one camera among the first camera and the second camera to be changed, the step of obtaining second position information regarding the positional relationship between the first camera and the second camera based on the changed position of at least one camera, and the step of performing stereo matching based on the plurality of two-dimensional pose information for each image and the second position information.

[0026] Meanwhile, the first neural network model and the second neural network model may be included in a single integrated neural network model.

[0027] According to one embodiment of the present disclosure for achieving the purpose described above, a non-transient computer-readable recording medium comprising a program for executing a method of controlling an electronic device, wherein the method of controlling the electronic device comprises the steps of: acquiring a plurality of images through at least one camera; inputting RGB data for each of the plurality of images into a first neural network model to acquire two-dimensional pose information for an object included in the plurality of images; inputting RGB data for at least one of the plurality of images into a second neural network model to identify whether the object is transparent; if the object is a transparent object, performing stereo matching based on the two-dimensional pose information for each of the plurality of images to acquire three-dimensional pose information for the object; and if the object is an opaque object, acquiring three-dimensional pose information for the object based on one of the plurality of images and depth information corresponding to the one image. Brief explanation of the drawing

[0029] FIG. 1 is a flowchart illustrating a method for controlling an electronic device according to one embodiment of the present disclosure, FIGS. 2 and FIGS. 3 are drawings showing a plurality of images and two-dimensional pose information according to an embodiment of the present disclosure, FIG. 4 is a flowchart illustrating in detail a method for controlling an electronic device according to one embodiment of the present disclosure, FIGS. 5 and 6 are drawings for explaining an embodiment in which an object according to the present disclosure has symmetry, FIG. 7 is a block diagram briefly illustrating the configuration of an electronic device according to one embodiment of the present disclosure, FIG. 8 is a block diagram illustrating in detail neural network models and modules according to one embodiment of the present disclosure, and, FIG. 9 is a block diagram showing in detail the configuration of an electronic device according to one embodiment of the present disclosure. Specific details for implementing the invention

[0030] The embodiments described herein are subject to various modifications and may have various forms; specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope of specific embodiments and should be understood to include various modifications, equivalents, and / or alternatives of the embodiments of the present disclosure. In relation to the description of the drawings, similar reference numerals may be used for similar components.

[0031] In describing the present disclosure, if it is determined that a detailed description of related known functions or configurations could unnecessarily obscure the essence of the present disclosure, such detailed description is omitted.

[0032] Additionally, the following embodiments may be modified in various other forms, and the scope of the technical concept of the present disclosure is not limited to the following embodiments. Rather, these embodiments are provided to make the present disclosure more faithful and complete and to fully convey the technical concept of the present disclosure to those skilled in the art.

[0033] The terms used in this disclosure are used merely to describe specific embodiments and are not intended to limit the scope of the rights. The singular expression includes the plural expression unless the context clearly indicates otherwise.

[0034] In the present disclosure, expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of such features (e.g., numerical values, functions, actions, or components such as parts) and do not exclude the presence of additional features.

[0035] In the present disclosure, expressions such as “A or B,” “at least one of A or / and B,” or “one or more of A or / and B” may include all possible combinations of items listed together. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” may refer to cases including (1) at least one A, (2) at least one B, or (3) both at least one A and at least one B.

[0036] Expressions such as "first," "second," "first," or "second" used in this disclosure may modify various components regardless of order and / or importance, and are used only to distinguish one component from another and do not limit said components.

[0037] Where it is stated that a certain component (e.g., a first component) is "(operatively or communicatively) coupled with / to" or "connected to" another component (e.g., a second component), it should be understood that the said certain component may be directly connected to the said other component or connected through another component (e.g., a third component).

[0038] On the other hand, when it is stated that a certain component (e.g., a first component) is "directly connected" or "directly coupled" to another component (e.g., a second component), it may be understood that no other component (e.g., a third component) exists between said certain component and said other component.

[0039] As used in this disclosure, the expression “configured to” may be replaced, depending on the context, with, for example, “suitable for,” “having the capacity to,” “designed to,” “adapted to,” “made to,” or “capable of.” The term “configured to” may not necessarily mean only “specifically designed to” in hardware.

[0040] Instead, in some situations, the expression “device configured to do something” may mean that the device is “capable of doing something” together with other devices or components. For example, the phrase “processor configured (or set) to perform A, B, and C” may mean a dedicated processor for performing those operations (e.g., an embedded processor), or a generic-purpose processor (e.g., a CPU or application processor) capable of performing those operations by executing one or more software programs stored in a memory device.

[0041] In the embodiments, a 'module' or 'part' performs at least one function or operation and may be implemented in hardware or software, or a combination of hardware and software. Additionally, a plurality of 'modules' or a plurality of 'parts' may be integrated into at least one module and implemented by at least one processor, except for the 'module' or 'part' that needs to be implemented in specific hardware.

[0042] Meanwhile, the various elements and areas in the drawings are depicted schematically. Accordingly, the technical concept of the present invention is not limited by the relative sizes or spacing depicted in the attached drawings.

[0043] Hereinafter, embodiments according to the present disclosure are described in detail with reference to the attached drawings so that those skilled in the art can easily implement them.

[0045] FIG. 1 is a flowchart illustrating a control method for an electronic device according to an embodiment of the present disclosure. FIG. 2 and FIG. 3 are drawings illustrating a plurality of images and two-dimensional pose information according to an embodiment of the present disclosure.

[0046] The ‘electronic device’ according to the present disclosure refers to a device capable of obtaining three-dimensional pose information for an object included in an image. In particular, the electronic device according to the present disclosure can obtain three-dimensional pose information in various ways depending on the characteristics of the object included in the image. For example, the electronic device according to the present disclosure can be implemented as a user terminal such as a smartphone or tablet PC, and can also be implemented as a device such as a robot. Hereinafter, the electronic device according to the present disclosure is referred to as the ‘electronic device (100)’.

[0047] As illustrated in FIG. 1, the electronic device (100) according to the present disclosure can acquire a plurality of images through at least one camera (S110).

[0048] The electronic device (100) may include 'at least one camera,' that is, one or two or more cameras. If the electronic device (100) includes two or more cameras, the positional relationship between the two or more cameras may be fixed or changeable. For example, the electronic device (100) may include a camera positioned on the rear left side of the electronic device (100) and a camera positioned on the rear right side of the electronic device (100). If the electronic device (100) is implemented as a robot, at least one camera may include two cameras positioned on the head and hand of the robot, namely a head camera and a hand camera, in which case the positional relationship between the two cameras may change as the position of at least one of the head camera and the hand camera changes.

[0049] "Multiple images" may include the same object and may be images acquired through a single camera or two or more different cameras. Specifically, multiple images may be two images acquired at different points in time through a first camera. Furthermore, multiple images may be two images acquired at the same point in time through the first camera and the second camera, respectively. In other words, in the present disclosure, multiple images may not only be different image frames included in a video sequence acquired through a single camera, but may also be different image frames resulting from capturing the same scene at the same time through cameras with different views.

[0050] For example, the images of FIGS. 2 and FIGS. 3 represent a first image and a second image obtained through a robot’s head camera and hand camera, respectively, when the electronic device (100) according to the present disclosure is implemented as a robot. Specifically, referring to the example of FIGS. 2 and FIGS. 3, the first image and the second image may each include an object called a ‘wine glass,’ and the object called a ‘wine glass’ may be placed in different positions and poses in each of the first image and the second image.

[0051] When multiple images are acquired, the electronic device (100) inputs RGB data for each of the multiple images into a first neural network model to acquire two-dimensional pose information for an object included in the multiple images (S120).

[0052] "The first neural network model" refers to a neural network model trained to output two-dimensional pose information for an object included in an image based on input RGB data. "Two-dimensional pose information" is a general term for information that can specify the pose of an object in two dimensions. In particular, the first neural network model detects a bounding box corresponding to each object included in the image and can obtain two-dimensional coordinate information of each of the pre-set feature points constituting the detected bounding box as two-dimensional pose information.

[0053] Referring to FIG. 2, the first neural network model detects a three-dimensional bounding box corresponding to a 'wine glass' in the first image and can obtain two-dimensional coordinate information corresponding to eight vertices (210, 220, 230, 240, 250, 260, 270, 280) constituting the detected bounding box as two-dimensional pose information. Similarly, referring to FIG. 3, the first neural network model detects a three-dimensional bounding box corresponding to a 'wine glass' in the second image and can obtain two-dimensional coordinate information corresponding to eight vertices (310, 320, 330, 340, 350, 360, 370, 380) constituting the detected bounding box as two-dimensional pose information. In the description of FIGS. 2 and 3, eight vertices constituting the detected bounding box were exemplified as examples of pre-set feature points, but it goes without saying that the present disclosure is not limited thereto.

[0054] Meanwhile, the electronic device (100) can identify whether an object is transparent by inputting RGB data for at least one of a plurality of images into a second neural network model (S130).

[0055] "Second neural network model" refers to a neural network model trained to classify the features of an object included in an input image and output the result. Specifically, based on the input RGB data, the second neural network model can output information regarding the probability that an object included in the image corresponds to each of a plurality of classes (or categories, domains) distinguished according to various features of the image. Furthermore, the electronic device (100) can identify the features of an object included in the input image based on the information regarding the probability output from the second neural network model. Here, the plurality of classes may be defined according to features such as whether the object is transparent or symmetrical.

[0056] Specifically, the electronic device (100) obtains information regarding whether an object is transparent through a second neural network model and can identify whether an object is transparent based on the information regarding whether an object is transparent. In the present disclosure, the statement that an object is "transparent" does not refer only to the case where the transparency of the object is 100%, but refers to the case where the transparency of the object is greater than or equal to a preset threshold value. Here, it may be preferable that the preset threshold value be set to distinguish the boundary between a degree of transparency where depth information can be obtained by a depth sensor as described later and a degree of transparency where it is difficult to obtain depth information by a depth sensor. Meanwhile, the statement that an object is transparent may include not only the case where the entire object is transparent, but also the case where an area greater than or equal to a preset ratio of the entire area of ​​the object is transparent. If a "wine glass" as illustrated in FIGS. 2 and FIGS. 3 is a transparent wine glass, it can be identified as a transparent object through the second neural network model according to the present disclosure.

[0057] Meanwhile, as described above, the second neural network model according to the present disclosure may identify not only whether an object is transparent but also whether an object is symmetrical. However, the process of identifying whether an object is symmetrical through the second neural network model is described in detail with reference to FIG. 5.

[0058] As described above, when information regarding whether an object is transparent is obtained, the electronic device (100) can obtain three-dimensional pose information for the object in different ways depending on whether the object is transparent.

[0059] If the identification result indicates that the object is a transparent object (S140-Y), the electronic device (100) can perform stereo matching based on 2-dimensional pose information for a plurality of images to obtain 3-dimensional pose information for the object (S150). That is, since it is difficult to obtain depth information for the object if the object is a transparent object, the electronic device (100) can obtain 3-dimensional pose information using 2-dimensional pose information for each of the plurality of images.

[0060] "3D pose information" is a general term for information that can specify the pose of an object in three dimensions. Specifically, 3D pose information may include information about the 3D coordinate values ​​of pixels corresponding to pre-set feature points among the pixels constituting the object. For example, 3D pose information may be provided in the form of a depth map that includes information about the depth of the object.

[0061] "Stereo matching" refers to one of the methods for obtaining three-dimensional pose information based on two-dimensional pose information. Specifically, the stereo matching process can be performed by obtaining three-dimensional pose information based on the displacement difference between two-dimensional pose information obtained from each of a plurality of images. Although the stereo matching process in a broad sense may include the process of obtaining two-dimensional pose information from each of a plurality of images, in describing the present disclosure, it is used to refer to the process after the two-dimensional pose information has been obtained.

[0062] For example, when an object included in multiple images is located close to the camera, a large displacement difference appears between the multiple images, and when an object included in multiple images is located far from the camera, a small displacement difference appears between the multiple images. Accordingly, the electronic device (100) can reconstruct 3D pose information based on the displacement difference between the multiple images. Here, 'displacement' may refer to disparity, which represents the distance between corresponding feature points in multiple images.

[0063] Specifically, the electronic device (100) can identify corresponding feature points in a plurality of images. For example, the electronic device (100) can identify corresponding feature points in a plurality of images by calculating the similarity between feature points based on at least one of luminance information, color information, and gradient information of each pixel of the plurality of images. Once the corresponding feature points are identified, the electronic device (100) can obtain disparity information between the corresponding feature points based on 2D coordinate information between the corresponding feature points.

[0064] When disparity information is obtained, the electronic device (100) can obtain 3D pose information based on the disparity information, the focal length of the camera, and information on the positional relationship between the cameras at the time of obtaining a plurality of images. Here, 'information on the focal length of the camera' may be stored in the memory of the electronic device (100), and 'information on the positional relationship between the cameras' may be determined based on a vector amount obtained by subtracting the value representing the position of another camera from the value representing the position of one camera among the cameras, and may be determined differently depending on the number and position of the cameras according to the present disclosure.

[0065] Specifically, when multiple images are acquired through multiple cameras having fixed positions, the electronic device (100) can acquire information about the positional relationship between the multiple cameras based on information previously stored in memory, and perform stereo matching based thereon.

[0066] Meanwhile, when multiple images are acquired through multiple cameras having non-fixed positions, the electronic device (100) may periodically or in real time acquire information regarding the positional relationship between the multiple cameras and perform stereo matching based on the acquired information. For example, if the electronic device (100) is implemented as a robot, the robot may acquire image frames while changing the positions of the head camera and the hand camera. Then, the robot may acquire information regarding the positional relationship between the head camera and the hand camera based on information regarding the joint angles, etc., of the frames connected to each of the head camera and the hand camera, and perform stereo matching based on the acquired information.

[0067] Meanwhile, when the electronic device (100) moves and acquires multiple images through a single camera, the electronic device (100) can determine the positional relationship between the cameras based on the position of the camera at the time of acquiring each of the multiple images.

[0068] If the identification result indicates that the object is an opaque object (S140-N), the electronic device (100) can obtain three-dimensional pose information for the object based on one of the multiple images and depth information corresponding to the image (S160). That is, since it is easy to obtain depth information for the object when the object is an opaque object, the electronic device (100) can obtain three-dimensional pose information using the depth information.

[0069] In the present disclosure, "depth information" refers to information indicating the distance between an object and a camera, specifically information obtained through a depth sensor. The "depth sensor" may be included in the camera and implemented in the form of a depth camera, or it may be included in an electronic device (100) as a separate component from the camera. For example, the depth sensor may be a Time-of-Flight (ToF) sensor or an IR depth sensor, provided that there are no special limitations on the type of depth sensor according to the present disclosure. When the depth sensor according to the present disclosure is implemented as a TOF sensor, the electronic device (100) can obtain depth information by measuring the flight time from when light is irradiated onto the object until the light reflected from the object is received by the depth sensor. Once depth information is obtained through the depth sensor, the electronic device (100) can obtain three-dimensional pose information for the object based on one of a plurality of images and the depth information.

[0070] Meanwhile, although the above description is based on the premise that there is only one object included in multiple images, this is merely for convenience of explanation and the present disclosure is not limited thereto. That is, when multiple images include multiple objects, the electronic device (100) may apply various embodiments according to the present disclosure to each of the multiple objects. Furthermore, when 3D pose information is obtained in different ways for each of the multiple objects, the electronic device (100) may combine the 3D pose information obtained in different ways and output a 3D depth image including the combined 3D pose information.

[0071] Meanwhile, although the above explanation was based on the premise that the first neural network model and the second neural network model are implemented as separate, independent neural network models, the first neural network model and the second neural network model may be included in a single integrated neural network model. Furthermore, when the first neural network model and the second neural network model are implemented as a single integrated neural network model, the entire pipeline of that single integrated neural network model can be jointly trained end-to-end.

[0072] Meanwhile, when acquiring a video sequence through at least one camera, the electronic device (100) may acquire three-dimensional pose information for some of the image frames included in the video sequence in the manner described above, and acquire pose information for all image frames by tracking pose information for the remaining image frames based on the acquired three-dimensional pose information. In this case, a predefined filter such as a Kalman filter may be used when tracking pose information.

[0073] According to the embodiment described above with reference to FIGS. 1 to 3, the electronic device (100) can acquire three-dimensional pose information for an object in an efficient manner depending on whether the object included in the image is a transparent object.

[0075] FIG. 4 is a flowchart illustrating in detail a control method of an electronic device according to one embodiment of the present disclosure. FIG. 5 and FIG. 6 are drawings for explaining an embodiment in which an object according to the present disclosure has symmetry.

[0076] As described above, the electronic device (100) according to the present disclosure can acquire 3D information in different ways depending on whether an object included in a plurality of images is transparent, and can also acquire 3D information in different ways depending on whether the plurality of objects are symmetrical. FIG. 4 is a diagram showing a combination of a method for acquiring 3D information in different ways depending on whether an object is transparent and a method for acquiring 3D information in different ways depending on whether an object is symmetrical. However, since the method for acquiring 3D information in different ways depending on whether an object is transparent has been described above with reference to FIG. 1, the method for acquiring 3D information in different ways depending on whether an object is symmetrical will be described below.

[0077] As illustrated in FIG. 4, the electronic device (100) can identify whether an object is symmetric based on RGB data for at least one of a plurality of images (S143). FIG. 4 illustrates that the symmetrical status of an object is identified only when the object is a transparent object (S140-Y), but the present disclosure is not limited thereto.

[0078] Specifically, the electronic device (100) can obtain information regarding whether an object is symmetrical through a second neural network model and obtain information regarding whether an object is symmetrical based on the information regarding whether an object is symmetrical. In describing the present disclosure, the term "symmetrical" for an object refers to a case where it satisfies symmetry in a 360-degree direction with respect to at least one axis passing through the object.

[0079] As shown in the examples of FIGS. 5 and 6, when an object included in multiple images is a 'wine glass', it satisfies symmetry in the 360-degree direction with respect to an axis passing through the center of the 'wine glass', and can be identified as a symmetrical object through the second neural network model.

[0080] If the identification result indicates that the object is a symmetric object (S145-Y), the electronic device (100) converts the first feature points included in the 2-dimensional pose information into second feature points independent of symmetry (S153), and performs stereo matching based on the second feature points to obtain 3-dimensional pose information for the object (S155).

[0081] Here, 'first feature points' refers to feature points as described with reference to FIGS. 2 and FIGS. 3, and 'second feature points' refers to feature points obtained by converting the first feature points into feature points independent of symmetry.

[0082] Referring to FIG. 5, the electronic device (100) can convert first feature points, which are eight vertices (210, 220, 230, 240, 250, 260, 270, 280) constituting a three-dimensional bounding box corresponding to a 'wine glass', into second feature points, which are three points (510, 520, 530) on an axis passing through the center of the 'wine glass', and perform stereo matching based on the second feature points to obtain three-dimensional pose information for the 'wine glass'. Likewise, with reference to FIG. 6, the electronic device (100) can convert first feature points, which are eight vertices (310, 320, 330, 340, 350, 360, 370, 380) constituting a three-dimensional bounding box corresponding to a 'wine glass', into second feature points, which are three points (610, 620, 630) on an axis passing through the center of the 'wine glass', and perform stereo matching based on the second feature points to obtain three-dimensional pose information for the 'wine glass'.

[0083] Meanwhile, in the case of a symmetrical object, since the number of bounding boxes corresponding to the object can be infinite, it is necessary to specify a criterion for identifying the first feature point. For example, referring to FIGS. 5 and 6, if a bounding box has an axis passing through the center of the 'wine glass' as the z-axis, then all of them can be bounding boxes corresponding to the 'wine glass'.

[0084] Accordingly, when constructing training data for training the first neural network model, in the case of a symmetrical object, annotation can be performed based on a three-dimensional coordinate system in which the x-axis or y-axis is perpendicular to the camera. Accordingly, the electronic device (100) can identify a first feature point based on a three-dimensional coordinate system in which the x-axis or y-axis is perpendicular to the camera through the trained first neural network model. Specifically, if the object included in the image is a symmetrical object, the electronic device (100) can identify a bounding box corresponding to the object based on a three-dimensional coordinate system in which the x-axis or y-axis is perpendicular to the camera, and identify eight vertices constituting the identified bounding box as the first feature point. That is, accordingly, the electronic device (100) can identify the first feature points under consistent criteria for a symmetrical object.

[0085] If the identification result indicates that the object is not an object with symmetry (S145-N), the electronic device (100) can obtain three-dimensional pose information for the object by performing stereo matching based on the first feature points included in the two-dimensional pose information (S157). That is, in the case of an object not having symmetry, since the first feature points have directionality, the electronic device (100) can perform stereo matching based on the first feature points without converting the first feature points into second feature points.

[0086] According to the embodiment described above with reference to FIGS. 4 to 7, the electronic device (100) can acquire three-dimensional pose information for an object in an efficient manner depending on whether the object included in the image is an object having symmetry.

[0087] Meanwhile, the control method of the electronic device (100) according to the above-described embodiment may be implemented as a program and provided to the electronic device (100). In particular, the program including the control method of the electronic device (100) may be stored and provided on a non-transitory computer-readable medium.

[0088] Specifically, in a non-transient computer-readable recording medium comprising a program for executing a control method of an electronic device (100), the control method of the electronic device (100) comprises the steps of: acquiring a plurality of images through at least one camera; inputting RGB data for each of the plurality of images into a first neural network model to acquire two-dimensional pose information for an object included in the plurality of images; inputting RGB data for at least one of the plurality of images into a second neural network model to identify whether the object is transparent; if the object is a transparent object, performing stereo matching based on two-dimensional pose information for each of the plurality of images to acquire three-dimensional pose information for the object; and if the object is an opaque object, acquiring three-dimensional pose information for the object based on one of the plurality of images and depth information corresponding to the image.

[0089] Although a method for controlling an electronic device (100) and a computer-readable recording medium including a program for executing the method for controlling the electronic device (100) have been briefly described above, this is merely to avoid redundant descriptions, and it is obvious that various embodiments of the electronic device (100) can also be applied to a method for controlling the electronic device (100) and a computer-readable recording medium including a program for executing the method for controlling the electronic device (100).

[0091] FIG. 7 is a block diagram briefly illustrating the configuration of an electronic device according to one embodiment of the present disclosure, and FIG. 8 is a block diagram detailedly illustrating neural network models and modules according to one embodiment of the present disclosure.

[0092] As illustrated in FIG. 7, an electronic device (100) according to one embodiment of the present disclosure includes a camera (110), a memory (120), and a processor (130). Additionally, as illustrated in FIG. 8, a processor (130) according to one embodiment of the present disclosure may include a plurality of modules such as a 2D pose information acquisition module (131), an object feature identification module (132), and a 3D pose information acquisition module (133).

[0093] The camera (110) can acquire an image of at least one object. Specifically, the camera (110) includes an image sensor, and the image sensor can convert light entering through a lens into an electrical image signal.

[0094] Specifically, the electronic device (100) according to the present disclosure may include at least one camera (110), that is, one or two or more cameras (110). If the electronic device (100) includes two or more cameras (110), the positional relationship between the two or more cameras (110) may be fixed or changed. For example, the electronic device (100) may include a camera (110) positioned on the rear left side of the electronic device (100) and a camera (110) positioned on the rear right side of the electronic device (100). When the electronic device (100) is implemented as a robot, at least one camera (110) may include two cameras (110) placed on the head and hand of the robot, respectively, namely a head camera (110) and a hand camera (110), and in this case, the positional relationship between the two cameras (110) may change as the position of at least one camera (110) among the head camera (110) and the hand camera (110) changes.

[0095] Meanwhile, the camera (110) according to the present disclosure is a depth camera and may include a depth sensor together with an image sensor. Here, the depth sensor may be a Time-of-Flight (ToF) sensor or an IR depth sensor, provided that there are no special limitations on the type of depth sensor according to the present disclosure.

[0096] At least one instruction regarding an electronic device (100) may be stored in the memory (120). Additionally, an operating system (O / S) for operating the electronic device (100) may be stored in the memory (120). Furthermore, various software programs or applications for operating the electronic device (100) may be stored in the memory (120) according to various embodiments of the present disclosure. Additionally, the memory (120) may include semiconductor memory such as flash memory or magnetic storage media such as a hard disk.

[0097] Specifically, various software modules for operating an electronic device (100) according to various embodiments of the present disclosure may be stored in the memory (120), and the processor (130) may control the operation of the electronic device (100) by executing the various software modules stored in the memory (120). That is, the memory (120) is accessed by the processor (130), and reading / writing / modifying / deleting / updating of data by the processor (130) may be performed.

[0098] Meanwhile, in the present disclosure, the term memory (120) may be used to include memory (120), ROM (not shown), RAM (not shown) within the processor (130), or a memory card (not shown) (e.g., micro SD card, memory stick) mounted in the electronic device (100).

[0099] In particular, in various embodiments according to the present disclosure, the memory (120) may include RGB data for each image according to the present disclosure, depth information, data for a first neural network model (121) and a second neural network model (122), etc. In addition, various information necessary within the scope of achieving the purpose of the present disclosure may be stored in the memory (120), and the information stored in the memory (120) may be updated as it is received from an external device or input by a user.

[0100] The processor (130) controls the overall operation of the electronic device (100). Specifically, the processor (130) is connected to the configuration of the electronic device (100) including the camera (110) and memory (120) as described above, and can control the overall operation of the electronic device (100) by executing at least one instruction stored in the memory (120) as described above.

[0101] The processor (130) can be implemented in various ways. For example, the processor (130) can be implemented as at least one of an Application Specific Integrated Circuit (ASIC), an embedded processor, a microprocessor, hardware control logic, a hardware Finite State Machine (FSM), and a Digital Signal Processor (DSP). Meanwhile, in this disclosure, the term processor (130) may be used to include a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), and a MPU (Main Processing Unit), etc.

[0102] In particular, in various embodiments according to the present disclosure, the processor (130) can acquire a plurality of images through at least one camera (110) and can acquire three-dimensional pose information for an object included in the plurality of images through a plurality of modules.

[0103] The '2D pose information acquisition module (131)' refers to a module capable of acquiring two-dimensional pose information for an object included in an image based on RGB data for the image. In particular, the 2D pose information acquisition module (131) can acquire two-dimensional pose information for an object included in an image by using the first neural network model (121) according to the present disclosure. As described above, the first neural network model (121) refers to a neural network model trained to output two-dimensional pose information for an object included in an image based on input RGB data, and the processor (130) can use the first neural network model (121) by accessing data for the first neural network model (121) stored in memory (120). Specifically, the 2D pose information acquisition module (131) can detect a bounding box corresponding to each object included in the image through the first neural network model (121), and acquire 2D coordinate information of each of the preset feature points constituting the detected bounding box as 2D pose information.

[0104] The 'object feature identification module (132)' refers to a module capable of identifying the features of an object included in an image. In particular, the object feature identification module (132) can identify the features of an object included in an image by utilizing the second neural network model (122) according to the present disclosure. As described above, the second neural network model (122) refers to a neural network model trained to classify the features of an object included in an image and output the results, and the processor (130) can utilize the second neural network model (122) by accessing data regarding the second neural network model (122) stored in memory (120). Specifically, the object feature identification module (132) can obtain information regarding the probability that an object included in an image corresponds to each of a plurality of classes (or categories, domains) distinguished according to various features of the image through the second neural network model (122), and can identify the features of an object included in an input image based thereon. Here, the plurality of classes may be defined according to features such as whether the object is transparent or symmetrical.

[0105] The '3D pose information acquisition module (133)' refers to a module capable of acquiring 3D pose information for an object by reconstructing 2D pose information for the object. In particular, the 3D pose information acquisition module (133) can acquire 3D pose information in different ways depending on the characteristics of the object. Specifically, when 2D pose information is received through the 2D pose information acquisition module (131) and information that the object is a transparent object is received through the object feature identification module (132), the 3D pose information acquisition module (133) can acquire 3D pose information for the object by performing stereo matching based on 2D pose information for multiple images. When 2D pose information is received through the 2D pose information acquisition module (131) and information that the object is an opaque object is received through the object feature identification module (132), the 3D pose information acquisition module (133) can acquire 3D pose information for the object based on one of the multiple images and depth information corresponding to the image.

[0106] Various embodiments according to the present disclosure based on the control of the processor (130) have been described in detail with reference to FIGS. 1 to 6, so a redundant description is omitted.

[0108] FIG. 9 is a block diagram showing in detail the configuration of an electronic device according to one embodiment of the present disclosure.

[0109] As illustrated in FIG. 9, an electronic device (100) according to one embodiment of the present disclosure may further include a camera (110), a memory (120), and a processor (130), as well as a driving unit (140), a communication unit (150), an input unit (160), and an output unit (170). However, the configurations illustrated in FIG. 7 through 9 are merely exemplary, and it is understood that in carrying out the present disclosure, new configurations may be added or some configurations may be omitted in addition to the configurations illustrated in FIG. 7 through 9.

[0110] The driving unit (140) refers to a configuration capable of generating power to implement various operations of the electronic device (100) based on driving data. The power generated by the driving unit (140) is transmitted to a support member (not shown) physically connected to the driving unit (140) to move the support member (not shown), thereby enabling various operations of the electronic device (100) to be implemented.

[0111] Specifically, the driving unit (140) may include a motor that generates a rotational force of a predetermined torque based on electrical energy supplied to the electronic device (100), and may also include a piston or cylinder device that generates a rotational force based on hydraulic or pneumatic pressure supplied to the electronic device (100). In addition, the support unit (not shown) may include a plurality of segments and a plurality of joints connecting the segments, and may be implemented in various shapes to perform various operations of the electronic device (100). Meanwhile, the electronic device (100) according to the present disclosure may include a plurality of driving units (140) or a plurality of support units (not shown). Furthermore, the number of components included in each driving unit (140) and support unit (not shown) is not limited to a specific number.

[0112] In particular, in various embodiments according to the present disclosure, the processor (130) can control the driving unit (140) so that the position of at least one camera (110) among the first camera (110) and the second camera (110) is changed. Specifically, when the driving unit (140) is controlled by the processor (130), the position of at least one camera (110) connected to the support unit (not shown) can be changed accordingly. When the position of at least one camera (110) is changed, the processor (130) can obtain position information regarding the positional relationship between the first camera (110) and the second camera (110) based on the changed position of at least one camera (110), and can perform stereo matching based on the 2D pose information for a plurality of images and the obtained position information.

[0113] The communication unit (150) includes a circuit and can perform communication with an external device. Specifically, the processor (130) can receive various data or information from an external device connected through the communication unit (150) and can also transmit various data or information to the external device.

[0114] The communication unit (150) may include at least one of a WiFi module, a Bluetooth module, a wireless communication module, and an NFC module. Specifically, the WiFi module and the Bluetooth module can each perform communication using the WiFi method and the Bluetooth method. When using the WiFi module or the Bluetooth module, various connection information such as SSID is first transmitted and received, and then various information is transmitted and received after establishing a communication connection using this information.

[0115] In addition, the wireless communication module can perform communication according to various communication standards such as IEEE, Zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), and 5G (5th Generation). Also, the NFC module can perform communication using the NFC (Near Field Communication) method, which uses the 13.56 MHz band among various RF-ID frequency bands such as 135 kHz, 13.56 MHz, 433 MHz, 860~960 MHz, and 2.45 GHz.

[0116] In particular, in various embodiments according to the present disclosure, the processor (130) may receive a plurality of images from an external device (e.g., a user terminal) through a communication unit (150). That is, although various embodiments of the present disclosure have been described above on the premise that a plurality of images are acquired through at least one camera (110), a plurality of images according to the present disclosure may be acquired by a camera (110) included in an external device and then transmitted to an electronic device (100).

[0117] Meanwhile, although the above description is based on the premise that the first neural network model (121) and the second neural network model (122) are stored in the memory (120) of the electronic device (100) and operate as on-device, at least one of the first neural network model (121) and the second neural network model (122) may be implemented through an external device (e.g., a server). In this case, the processor (130) may control the communication unit (150) to transmit data for a plurality of images to the external device, and may receive information corresponding to the output of at least one of the first neural network model (121) and the second neural network model (122) from the external device through the communication unit (150).

[0118] Meanwhile, if the electronic device (100) according to the present disclosure includes a driving unit (140), the processor (130) may receive driving data for controlling the driving unit (140) from an external device (e.g., an edge computing device) through a communication unit (150), and may control the operation of the driving unit (140) based on the received driving data.

[0119] The input unit (160) includes a circuit, and the processor (130) can receive user commands to control the operation of the electronic device (100) through the input unit (160). Specifically, the input unit (160) may include a microphone (not shown) and a remote control signal receiving unit (not shown), etc. Additionally, the input unit (160) may be implemented in a form included in a display as a touch screen.

[0120] In particular, in various embodiments according to the present disclosure, the input unit (160) can receive various types of user input, such as user input for acquiring a plurality of images, user input for acquiring a depth map including three-dimensional pose information for an object, etc.

[0121] The output unit (170) includes a circuit, and the processor (130) can output various functions that the electronic device (100) can perform through the output unit (170). In addition, the output unit (170) may include at least one of a display, a speaker, and an indicator. The display may output image data under the control of the processor (130). Specifically, the display may output an image stored in the memory (120) under the control of the processor (130). A display according to one embodiment of the present disclosure may also display a user interface stored in the memory (120). The display may be implemented as an LCD (Liquid Crystal Display Panel), OLED (Organic Light Emitting Diodes), etc., and the display may also be implemented as a flexible display, a transparent display, etc. depending on the case. However, the display according to the present disclosure is not limited to a specific type. The speaker can output audio data under the control of the processor (130), and the indicator can be lit under the control of the processor (130).

[0122] In particular, in various embodiments according to the present disclosure, the output unit (170) can output a plurality of images obtained through at least one camera (110), two-dimensional pose information for an object included in the plurality of images, a depth map including three-dimensional pose information for an object, etc.

[0123] According to various embodiments of the present disclosure as described above, the electronic device (100) can acquire three-dimensional pose information for an object in an efficient manner according to the characteristics of the object included in the image.

[0124] Meanwhile, functions related to the first neural network model (121) and the second neural network model (122), etc., as described above, can be performed through memory (120) and a processor (130). The processor (130) may be composed of one or more processors (130). At this time, the one or more processors (130) may be a general-purpose processor (130) such as a CPU, AP, etc., a graphics-dedicated processor (130) such as a GPU, VPU, etc., or an artificial intelligence-dedicated processor (130) such as an NPU.

[0125] One or more processors (130) control input data to be processed according to predefined operation rules or artificial intelligence models stored in non-volatile memory (120) and volatile memory (120). The predefined operation rules or artificial intelligence models are characterized by being created through learning.

[0126] Here, being created through learning means that a predefined behavioral rule or artificial intelligence model of a desired characteristic is created by applying a learning algorithm to a number of learning data. Such learning may be performed on the device itself where the artificial intelligence according to the present disclosure is executed, or it may be performed through a separate server / system.

[0127] An artificial intelligence model may be composed of multiple neural network layers. Each layer has multiple weight values ​​and performs operations on the layer through the operation of the multiple weights and the operation of the previous layer. Examples of neural networks include CNN (Convolutional Neural Network), DNN (Deep Neural Network), RNN (Recurrent Neural Network), RBM (Restricted Boltzmann Machine), DBN (Deep Belief Network), BRDNN (Bidirectional Recurrent Deep Neural Network), GAN (Generative Adversarial Networks), and Deep Q-Networks; however, the neural networks in this disclosure are not limited to the aforementioned examples except where specified.

[0128] A learning algorithm is a method of training a specific target device (e.g., a robot) using a number of learning data to enable the target device to make decisions or predictions on its own. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, and the learning algorithms in this disclosure are not limited to the aforementioned examples except where specified.

[0129] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory storage medium' simply means that it is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily. For example, a 'non-transitory storage medium' may include a buffer in which data is stored temporarily.

[0130] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., downloadable app) may be temporarily stored or temporarily created in a device-readable storage medium such as the memory (120) of the manufacturer's server, the application store's server, or the relay server.

[0131] Each component (e.g., module or program) according to the various embodiments of the present disclosure as described above may be composed of a single or multiple entities, and some of the aforementioned sub-components may be omitted, or other sub-components may be further included in the various embodiments. Generally or additionally, some components (e.g., module or program) may be integrated into a single entity to perform the same or similar functions as those performed by each of the respective components prior to integration.

[0132] Operations performed by a module, program, or other component according to various embodiments may be executed sequentially, in parallel, iteratively, or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0133] Meanwhile, the terms “part” or “module” as used in this disclosure include a unit composed of hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A “part” or “module” may be a component formed integrally, or a minimum unit or part thereof that performs one or more functions. For example, a module may be composed of an application-specific integrated circuit (ASIC).

[0134] Various embodiments of the present disclosure may be implemented as software comprising instructions stored on a machine-readable storage medium (e.g., a computer). The machine may include an electronic device (e.g., an electronic device (100)) according to the disclosed embodiments, which is a device capable of calling instructions stored from the storage medium and operating according to the called instructions.

[0135] When the above instruction is executed by a processor, the processor may perform the function corresponding to the instruction directly or by using other components under the control of the processor. The instruction may include code generated or executed by a compiler or an interpreter.

[0136] Although preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above. It is understood that various modifications can be made by those skilled in the art without departing from the essence of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present disclosure. Explanation of the symbols

[0138] 100: Electronic device 110: Camera 120: Memory 130: Processor 140: Driving unit 150: Communication unit 160: Input section 170: Output section

Claims

Claim 1 An electronic device comprising: at least one camera; a memory; and a processor that acquires a plurality of images through the at least one camera, inputs RGB data for each of the plurality of images into a first neural network model to acquire two-dimensional pose information for an object included in the plurality of images, inputs RGB data for at least one of the plurality of images into a second neural network model to identify whether the transparency of the object is greater than or equal to a preset threshold value, and if it is identified that the transparency of the object is greater than or equal to the preset threshold value, performs stereo matching based on the two-dimensional pose information for each of the plurality of images to acquire three-dimensional pose information for the object, and if it is identified that the transparency of the object is less than the preset threshold value, acquires three-dimensional pose information for the object based on one of the plurality of images and depth information corresponding to the one image. Claim 2 In claim 1, the processor is an electronic device that obtains information regarding whether the object is transparent through the second neural network model and identifies whether the object is transparent based on the information regarding whether the object is transparent. Claim 3 An electronic device according to claim 1, wherein the processor obtains information regarding whether the object is symmetrical through the second neural network model, identifies whether the object is symmetrical based on the information regarding whether the object is symmetrical, and if the object is a symmetrical object, converts the first feature points included in the two-dimensional pose information into second feature points independent of symmetry, and performs stereo matching based on the second feature points to obtain three-dimensional pose information for the object. Claim 4 In claim 3, if the object is a symmetric object, the electronic device in which the first feature points are identified based on a three-dimensional coordinate system in which the x-axis or y-axis is perpendicular to the at least one camera. Claim 5 An electronic device according to claim 1, wherein the plurality of images are two images acquired at different times through a first camera among the at least one camera. Claim 6 In claim 5, the electronic device wherein the plurality of images are two images acquired at the same time through each of the first camera and the second camera among the at least one camera. Claim 7 In claim 6, the processor is an electronic device that acquires first position information regarding the positional relationship between the first camera and the second camera, and performs stereo matching based on the plurality of two-dimensional pose information for each image and the first position information. Claim 8 An electronic device according to claim 7, further comprising: a driving unit; wherein the processor controls the driving unit to change the position of at least one camera among the first camera and the second camera, obtains second position information regarding the positional relationship between the first camera and the second camera based on the changed position of at least one camera, and performs stereo matching based on the plurality of two-dimensional pose information per image and the second position information. Claim 9 An electronic device according to claim 1, wherein the first neural network model and the second neural network model are included in a single integrated neural network model. Claim 10 A control method performed by an electronic device comprising: a step of acquiring a plurality of images through at least one camera; a step of inputting RGB data for each of the plurality of images into a first neural network model to acquire two-dimensional pose information for an object included in the plurality of images; a step of inputting RGB data for at least one of the plurality of images into a second neural network model to identify whether the transparency of the object is greater than or equal to a preset threshold value; a step of, if identified as the transparency of the object being greater than or equal to a preset threshold value, performing stereo matching based on the two-dimensional pose information for each of the plurality of images to acquire three-dimensional pose information for the object; and a step of, if identified as the transparency of the object being less than a preset threshold value, acquiring three-dimensional pose information for the object based on one of the plurality of images and depth information corresponding to the one image. Claim 11 A method for controlling an electronic device according to claim 10, wherein the step of identifying whether the object is transparent comprises: a step of obtaining information regarding whether the object is transparent through the second neural network model; and a step of identifying whether the object is transparent based on the information regarding whether the object is transparent. Claim 12 In claim 10, the step of identifying whether the object is symmetrical comprises: a step of obtaining information regarding whether the object is symmetrical through the second neural network model; and a step of identifying whether the object is symmetrical based on the information regarding whether the object is symmetrical; and the control method performed by the electronic device further comprises, if the object is an object having symmetry, a step of converting first feature points included in the two-dimensional pose information into second feature points independent of symmetry; and a step of obtaining three-dimensional pose information for the object by performing stereo matching based on the second feature points. Claim 13 A method for controlling an electronic device according to claim 12, wherein if the object is a symmetric object, the first feature points are identified based on a three-dimensional coordinate system in which the x-axis or y-axis is perpendicular to the at least one camera. Claim 14 A method for controlling an electronic device according to claim 10, wherein the plurality of images are two images acquired at different times through a first camera among the at least one camera. Claim 15 A method for controlling an electronic device according to claim 14, wherein the plurality of images are two images acquired at the same time through each of the first camera and the second camera among the at least one camera. Claim 16 In claim 15, the control method performed by the electronic device further comprises: a step of obtaining first position information regarding the positional relationship between the first camera and the second camera; and a step of performing stereo matching based on the plurality of two-dimensional pose information for each image and the first position information. Claim 17 In claim 16, the control method performed by the electronic device further comprises: a step of controlling the position of at least one camera among the first camera and the second camera to be changed; a step of obtaining second position information regarding the positional relationship between the first camera and the second camera based on the changed position of at least one camera; and a step of performing stereo matching based on the plurality of two-dimensional pose information per image and the second position information. Claim 18 A method for controlling an electronic device according to claim 10, wherein the first neural network model and the second neural network model are included in a single integrated neural network model. Claim 19 A non-transient computer-readable recording medium comprising a program for executing a method for controlling an electronic device, wherein the method for controlling the electronic device comprises: a step of acquiring a plurality of images through at least one camera; a step of inputting RGB data for each of the plurality of images into a first neural network model to acquire two-dimensional pose information for an object included in the plurality of images; a step of inputting RGB data for at least one of the plurality of images into a second neural network model to identify whether the transparency of the object is greater than or equal to a preset threshold value; a step of, if identified as the transparency of the object being greater than or equal to a preset threshold value, performing stereo matching based on the two-dimensional pose information for each of the plurality of images to acquire three-dimensional pose information for the object; and, if identified as the transparency of the object being less than a preset threshold value, acquiring three-dimensional pose information for the object based on one of the plurality of images and depth information corresponding to the one image.

Images