Electronic device and method for controlling the same

Abstract

A method for controlling an electronic device includes: acquiring an image and depth information of the acquired image; inputting the acquired image into a neural network model, wherein the neural network model is trained to acquire information related to objects included in the acquired image; acquiring intermediate feature values ​​output by intermediate layers of the neural network model; identifying feature regions for at least one object among the objects included in the acquired image based on the intermediate feature values; and acquiring distance information between the electronic device and the at least one object based on the feature regions for the at least one object and the depth information.

Description

Technical Field

[0001] This disclosure relates to an electronic device and a method for controlling the electronic device, and more specifically, to obtaining distance information between the electronic device and an object by using the output values ​​of an intermediate layer of an object recognition neural network model. Background Technology

[0002] Artificial intelligence (AI) systems are being used across various fields. Unlike rule-based intelligent systems in related fields, AI systems allow machines to learn, determine, and become more intelligent on their own. As AI systems are used more extensively, they exhibit higher recognition rates and become more accurate in understanding user preferences. Consequently, rule-based intelligent systems in related fields are gradually being replaced by deep learning-based AI systems.

[0003] Artificial intelligence technology consists of machine learning (e.g., deep learning) and element technology that utilizes machine learning.

[0004] Machine learning refers to an algorithmic technique that classifies / learns features from input data. Meta-technology refers to a technique used to perform human brain functions (such as cognition and determination) by using machine learning algorithms (such as deep learning), and includes technical fields such as language understanding, visual understanding, reasoning / prediction, knowledge representation, and operational control. Specifically, language understanding is a technique for recognizing human language / characters and applying / processing said language / characters, and includes natural speech processing, machine translation, communication systems, question answering, speech recognition / synthesis, etc. Summary of the Invention

[0005] Technical issues

[0006] Recently, there has been active effort to obtain distance information about objects using artificial intelligence systems. However, a problem exists where the accuracy of the object's distance information decreases when there is an obstacle in the image other than the object to be identified, or when multiple objects overlap in an image.

[0007] Technical solution

[0008] The embodiment provides an electronic device for acquiring distance information of an object with improved accuracy.

[0009] Other aspects will be set forth in part in the description which follows, and in part will be apparent from the description or may be learned by practice of the presented embodiments.

[0010] According to one aspect of this disclosure, an electronic device is provided, comprising: a memory storing at least one instruction; and a processor executing the at least one instruction stored in the memory, wherein the processor is configured by executing the at least one instruction to: acquire an image and depth information of the image; input the acquired image into a neural network model trained to acquire information relating to objects included in the image; acquire intermediate feature values ​​output by intermediate layers of the neural network model; identify feature regions for at least one object included in the image based on the intermediate feature values; and acquire distance information between the electronic device and the at least one object based on the feature regions for the at least one object and the depth information of the image.

[0011] The processor can be configured to: input the acquired image into the neural network model and obtain the final feature values ​​output by the final layer of the neural network model; identify type information and object region information for each of the at least one object included in the image based on the obtained final feature values; and obtain a feature map including feature regions for the at least one object included in the image based on the intermediate feature values.

[0012] The processor can be configured to: map the acquired feature map to an object region for each of the at least one object, and identify a feature region corresponding to each of the at least one object.

[0013] The feature map can be obtained by summing multiple intermediate feature values ​​obtained from multiple predefined intermediate layers.

[0014] The processor can be configured to: obtain a representative value based on the feature pixels corresponding to the feature region in the depth information of the image, and map the representative value to the feature region to obtain the distance information.

[0015] The processor can be configured to generate an image in which the type information and the distance information are displayed around the object region for each of the at least one object.

[0016] The electronic device may include a display, and the processor may be configured to control the display such that the generated image is output.

[0017] The processor can be configured to send signals to an external device via a communication interface, so that the generated image is output through the external device.

[0018] The processor can be configured to acquire depth information of the image via a depth camera or a stereo camera, or to acquire depth information of the image by analyzing the acquired image.

[0019] The intermediate layer may include a first intermediate layer corresponding to a first type of object, and the processor may be configured to: identify a first feature region for the first type of object based on a first intermediate feature value output by the first intermediate layer, obtain a first representative value based on feature pixels in the depth information of the image corresponding to the first feature region, and map the first representative value to the first feature region and obtain first distance information.

[0020] According to one aspect of this disclosure, a method for controlling an electronic device is provided, wherein the method includes the following steps: acquiring an image and depth information of the image; inputting the acquired image into a neural network model trained to acquire information related to objects included in the image; acquiring intermediate feature values ​​output by intermediate layers of the neural network model; identifying feature regions for at least one object included in the image based on the intermediate feature values; and acquiring distance information between the electronic device and the at least one object based on the feature regions for the at least one object and the depth information of the image.

[0021] Beneficial effects

[0022] According to an embodiment, users can obtain distance information of objects with improved accuracy through electronic devices.

[0023] Therefore, it can improve user convenience and satisfaction. Attached Figure Description

[0024] The above and / or other aspects, features, and advantages of specific embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0025] Figure 1 This is a diagram illustrating the operation of an electronic device according to an embodiment;

[0026] Figure 2 This is a block diagram illustrating the configuration of an electronic device according to an embodiment;

[0027] Figure 3 This is a block diagram illustrating a neural network model according to an embodiment;

[0028] Figure 4 This is a diagram illustrating a neural network model according to an embodiment;

[0029] Figure 5 This is a diagram illustrating a method for obtaining the distance to an object according to an embodiment;

[0030] Figure 6 This is a diagram illustrating a method for obtaining distance information of an object according to an embodiment;

[0031] Figure 7 This is a diagram illustrating a method for obtaining distance information of an object according to an embodiment;

[0032] Figure 8 This is a diagram illustrating a method for obtaining a feature map according to an embodiment;

[0033] Figure 9 This is a diagram illustrating a method for obtaining distance information of an object according to an embodiment;

[0034] Figure 10 This is a diagram illustrating a method for obtaining distance information of an object according to an embodiment;

[0035] Figure 11 This is a diagram illustrating the output image according to an embodiment; and

[0036] Figure 12 This is a flowchart illustrating a control method according to an embodiment. Detailed Implementation

[0037] In light of the functions described herein, as many commonly used terms as possible are employed. However, commonly used terms can change over time based on the intent of those skilled in the art working in the relevant field, court rulings, or the emergence of new technologies. Furthermore, terms may be chosen by the applicant, and in such cases, the meaning of those terms is described in detail below. Therefore, the terms used herein should be defined based on their meaning and the overall content described, rather than merely on their names.

[0038] As used herein, terms such as “first” or “first”, “second” or “second” may modify the corresponding component regardless of importance or order, and are used to distinguish one component from another without limiting the component.

[0039] Singular expressions include plural expressions, provided they do not have a significantly different meaning in the context. Furthermore, terms such as “comprising” and “consisting of” should be interpreted as specifying the presence of such characteristics, numbers, steps, operations, elements, components, or combinations thereof as described in the specification, but do not preclude the presence of one or more other characteristics, numbers, steps, operations, elements, components, or combinations thereof, or the possibility of adding one or more other characteristics, numbers, steps, operations, elements, components, or combinations thereof.

[0040] Specific embodiments will be described below with reference to the accompanying drawings. However, it should be noted that embodiments may be implemented in various different forms, and this disclosure is not limited to the embodiments described herein. Similar components are designated by similar reference numerals throughout the drawings and the specification.

[0041] Figure 1 This is a diagram illustrating the operation of an electronic device according to an embodiment.

[0042] The electronic device can acquire a captured image 10. The electronic device can input the captured image 10 into a neural network model 310 and obtain information related to the target object 1 included in the captured image 10. Specifically, the electronic device can obtain the final feature values ​​25 of the neural network model 310. The electronic device 100 can obtain type information 26 for the target object 1 based on the final feature values ​​25. For example, the type information 26 could be a classification value for the target object 1.

[0043] The electronic device can obtain object region information 27 for the target object 1 based on the final feature value 25. The object region information 27 may include the position information of the target object 1 in the captured image 10. For example, the object region information 27 may include the coordinate values ​​of a bounding box that is set to include the target object 1.

[0044] The electronic device can identify the feature region 22 based on the intermediate feature value 21. Here, the feature region can represent the region that becomes a feature of the target object 1 when the neural network model 310 identifies the target object 1. For example, in the case that the target object 1 is a person, the feature region can be the area surrounding the person's face. However, this is just an example, and the feature region can be defined as various regions depending on the type of the target object 1. The electronic device can map the intermediate feature value 21 and the object region information 27, and identify the feature region 22 corresponding to the target object 1. The feature region 22 can exist in the bounding box corresponding to the object region information 27. The electronic device can extract the intermediate feature value 21 from one or more intermediate layers included in the neural network model 310. According to the embodiment, the feature region can refer to an activation region, an activation map, or a target region.

[0045] The electronic device can acquire depth information 30 from the captured image 10. The depth information 30 may include depth information for the target object 1. The electronic device can acquire the depth information 30 through various methods. For example, the electronic device can acquire the depth information 30 by analyzing the captured image 10. However, this is merely an example, and the electronic device can acquire the depth information 30 based on image data acquired by a depth camera.

[0046] The electronic device can acquire distance information for the target object 1 based on the intermediate feature value 21 and the depth information 30 and / or generate an output image 40 displaying the distance information. The electronic device can acquire the distance information based on the feature pixels 31 corresponding to the feature region 22 of the intermediate feature value 21 in the depth information 30. Specifically, the electronic device can map the representative value of the feature pixel 31 to the intermediate feature value 21 and acquire the distance information. The representative value of the feature pixel 31 can be at least one of the mode, median, average, or minimum value of the feature pixel 31. The distance information can be the distance information from the camera acquiring the captured image 10 to the target object 1. For example, if the electronic device has captured the target object 1, the distance information can be the distance information from the electronic device to the target object 1.

[0047] The electronic device can generate an image based on type information 26, object region information 27, and distance information for the target object 1. For example, the electronic device can generate an image that includes type information 26, object region information 27, and distance information for the target object 1. The electronic device can output the generated image to a display or send the image to another external device.

[0048] Figure 2 This is a block diagram illustrating the configuration of an electronic device according to an embodiment. (Refer to...) Figure 2 The electronic device 100 may include a camera 110, a memory 120, a communication interface 130, a processor 140, and a sensor 150. The electronic device 100 need not be implemented to include all of the above components, but may be implemented using only some of the components.

[0049] Camera 110 can acquire various images. For example, camera 110 can be mounted on electronic device 100 and capture images of the area around electronic device 100. Camera 110 can include various types of cameras. For example, camera 110 can include 2D-based RGB cameras and IR cameras. Camera 110 can also include 3D-based Time-of-Flight (ToF) cameras and stereo cameras.

[0050] The memory 120 may store the operating system (OS) used to control the overall operation of the components of the electronic device 100, as well as instructions or data related to the components of the electronic device 100. For this purpose, the memory 120 may be implemented as a non-volatile memory (e.g., hard disk, solid-state drive (SSD), flash memory), volatile memory, etc.

[0051] Memory 120 may include at least one instruction. For example, memory 120 may store instructions that, when executed, cause processor 140 to acquire information related to objects included in an image when acquiring an image from camera 110. Memory 120 may store a neural network model (or object recognition model, etc.) for recognizing objects. Specifically, the neural network model may be executed by a general-purpose processor (e.g., a central processing unit (CPU)) or a separate AI-specific processor (e.g., a graphics processing unit (GPU), a neural processing unit (NPU), etc.).

[0052] The communication interface 130 includes at least one circuit and is capable of communicating with various types of external servers or external devices according to various communication methods. For example, the communication interface 130 may include at least one of a Wi-Fi communication module, a cellular communication module, a fourth-generation (4G) mobile communication module, a fourth-generation (4G) Long Term Evolution (LTE) communication module, or a fifth-generation (5G) mobile communication module.

[0053] Electronic device 100 can acquire images of its surroundings taken by an external device via communication interface 130. Electronic device 100 can acquire information related to objects included in the images acquired from the external device. Electronic device 100 can transmit the acquired information related to the objects to the external device via communication interface 130. Electronic device 100 may be a server device.

[0054] The processor 140 can control the overall operation of the electronic device 100.

[0055] For example, processor 140 can acquire an image of the surroundings of the imaging electronic device 100 and the depth information of that image. Processor 140 can acquire an image and its depth information via camera 110. Specifically, processor 140 can acquire an image captured by an RGB camera. Processor 140 can acquire the depth information of the image via a depth camera or stereo camera surrounding the imaging electronic device 100. Optionally, processor 140 can analyze the pixel values ​​of the image captured by the RGB camera and acquire depth information of objects included in the image.

[0056] Processor 140 can input the acquired image into a neural network model trained to acquire information related to objects included in the image, and acquire intermediate feature values ​​output by intermediate layers of the neural network model. Processor 140 can identify feature regions for at least one object included in the image based on the intermediate feature values. Processor 140 can acquire a feature map including the feature regions for at least one object included in the image based on the intermediate feature values. The feature map can be acquired by summing multiple intermediate feature values ​​acquired by multiple predefined intermediate layers.

[0057] Processor 140 can input the acquired image into a neural network model and identify feature regions for objects based on the final feature values ​​output by the final layer of the neural network model. Specifically, processor 140 can identify type information and information about object regions for each of at least one object included in the image based on the acquired final feature values. Processor 140 can map feature maps to object regions for each of the at least one object and identify feature regions corresponding to each of the at least one object.

[0058] Processor 140 can acquire distance information between electronic device 100 and the at least one object based on feature regions of the at least one object and depth information of the image. For example, processor 140 can map pixel values ​​corresponding to feature regions of the object in the acquired depth information to the feature regions of the object, and acquire distance information between electronic device 100 and the object. Processor 140 can acquire a representative value of the depth information of the image, map the acquired representative value to the feature regions of the object, and acquire distance information. The representative value can be at least one of the mode, median, average, or minimum value of the depth values ​​of a plurality of pixels constituting the image.

[0059] Processor 140 can generate an image displaying object type information and distance information around an object region for each of the at least one object. Processor 140 can control a display included in electronic device 100 such that the generated image is displayed. Optionally, processor 140 can send signals to an external device via communication interface 130, causing the generated image to be displayed on another external device. Thus, electronic device 100 can provide a user with at least one of distance information between objects included in an image or type information of objects included in an image.

[0060] Sensor 150 can be a component used to acquire depth information about an object. For example, sensor 150 may include a ToF sensor, a LiDAR sensor, and a radar sensor. Electronic device 100 can acquire distance information about the object based on the depth information acquired by sensor 150. As described above, electronic device 100 can analyze images captured by camera 110 and acquire depth information of objects included in the images.

[0061] Figure 3 This is a block diagram illustrating a neural network model according to an embodiment. (Refer to...) Figure 3 The neural network model 310 may include an input layer 320, an intermediate layer 330, and a final layer 350. Here, each layer may include multiple neural network layers.

[0062] Input layer 320 can receive input image 301 and output input feature values ​​from input image 301. Input layer 320 can send the input feature values ​​to intermediate layer 330. Intermediate layer 330 can extract and output intermediate feature values ​​340 of input image 301 based on the input feature values ​​received from input layer 320.

[0063] Processor 140 can acquire information related to feature regions of objects included in input image 301 based on intermediate feature values ​​340. For example, processor 140 can analyze intermediate feature values ​​340 and extract feature regions of objects included in input image 301. The component of intermediate feature value 340 corresponding to the feature region of the object may be larger than the component corresponding to the region where no object exists.

[0064] The intermediate layer 330 can send intermediate feature values ​​340 to the final layer 350. The final layer 350 can extract and output final feature values ​​360 for the input image 301 based on the intermediate feature values ​​340 received from the intermediate layer 330. The final feature values ​​360 can be associated with various information about the object. For example, the final feature values ​​360 can be associated with at least one of location information of the region where the object is located or object type information. Specifically, when the neural network model 310 is trained to perform an operation of classifying objects, the final feature values ​​360 can be associated with object type information. Furthermore, when the neural network model 310 is trained to perform an operation of recognizing objects, the final feature values ​​360 can be associated with location information of the region where the object is located or object type information.

[0065] As described above, the neural network model 310 may consist of multiple neural network layers. Each of the multiple neural network layers has multiple weight values, and neural network operations are performed through operations between the results of operations of previous layers and the multiple weight values. The multiple weight values ​​of the multiple neural network layers can be optimized through the learning results of the artificial intelligence model. For example, the multiple weight values ​​can be updated such that the loss value or cost obtained from the artificial intelligence model during the learning process is reduced or minimized. Artificial neural networks may include deep neural networks (DNNs), and there exist, for example, convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), or deep Q-networks, etc., but this is not limiting.

[0066] Figure 4 This is a diagram illustrating a neural network model according to an embodiment.

[0067] Reference Figure 4The neural network model 410 can output final feature values ​​420 based on the input image 401. The processor 140 can acquire the final feature values ​​420 output from the neural network model 410. The processor 140 can identify type information 421 and object region information 422 for the target object ob based on the final feature values ​​420. The processor 140 can generate an image based on the identified type information 421 (i.e., human) and object region information 422.

[0068] The neural network model 410 may include multiple intermediate layers 411. Various operations can be performed within the multiple intermediate layers 411. For example, an operation can be performed to add the intermediate feature values ​​output by each of the multiple intermediate layers 411 to the product of the weights w1, w2, w3 corresponding to the intermediate feature values. Furthermore, global average pooling (GAP) can be performed within the intermediate layers 411. The neural network model 410 may include multiple convolutional layers (CONV). These multiple convolutional layers may be included within the intermediate layers 411.

[0069] Figure 5 This is a diagram illustrating a method for obtaining the distance to an object according to an embodiment.

[0070] Processor 140 can acquire input image 510 and object region information 511 for a first object ob1 extracted based on input image 510. Processor 140 can input input image 510 to... Figure 4 In the neural network model 410, object region information 511 is obtained. The processor 140 can obtain depth information 520 corresponding to the input image 510. The processor 140 can obtain distance information of a first object ob1 based on the object region information 511 and the depth information 520. Specifically, for example, the processor 140 can obtain the distance value of the first object ob1 by mapping the sub-depth information 521 corresponding to the object region information 511 in the depth information 520 to the object region information 511. Specifically, the processor 140 can define the representative value of the pixels of the sub-depth information 521 as the distance value of the first object ob1. The representative value can be at least one of the mode, median, average, or minimum value of the pixels corresponding to the sub-depth information 521. That is, the representative value can represent a representative distance value calculated based on multiple pixel values ​​that respectively include distance values.

[0071] When there is another obstacle in front of the target object whose distance information is to be acquired, the electronic device can acquire the distance information of the target object.

[0072] Figure 6 This is a diagram illustrating a method for obtaining distance information of an object according to an embodiment.

[0073] Processor 140 can acquire input image 610 and object region information 611 for a first object ob1 extracted based on input image 610. Processor 140 can input input image 610 to... Figure 4 In the neural network model 410, object region information 611 is acquired. Furthermore, the processor 140 can acquire depth information 620 corresponding to the input image 610. The depth information 620 can be a depth image corresponding to the input image 610. In this case, the processor 140 can define or acquire the distance value of the first object ob1 based on the depth information 620. For example, the processor 140 can define the distance value of the first object ob1 by mapping sub-depth information 621 corresponding to the object region information 611 in the depth information 620 to the object region information 611. Specifically, the processor 140 can define representative values ​​of the pixels in the sub-depth information 621 to calculate the distance value of the first object ob1.

[0074] The object region information 611 may include data for the second object ob2. That is, the object region information 611 for the first object ob1, which is the object whose distance information will be acquired, may include data for the second object ob2, which is an obstacle but not the object whose distance information will be acquired. In other words, as... Figure 6 As shown, the bounding box of the depicted object region information 611 may include not only the first object ob1, but also the second object ob2. Therefore, the sub-depth information 621 may include depth information corresponding not only to the first object ob1 but also to the second object ob2.

[0075] Therefore, when the processor 140 calculates the representative value of the sub-depth information 621, depth information for the second object ob2 that is unrelated to the first object ob1 may be reflected. Consequently, the accuracy of the distance information for the first object ob1 may be reduced. To prevent this, when calculating the representative value of the sub-depth information 621, the processor 140 may calculate the representative value based on the values ​​of the remaining pixels in the pixels corresponding to the sub-depth information 621, excluding the pixels corresponding to the second object ob2.

[0076] In this embodiment, when multiple objects exist within a bounding box, the problem of reduced accuracy in the distance to the target object due to the presence of an obstacle or irrelevant object other than the target object in the input image can be eliminated.

[0077] Figure 7 This is a diagram illustrating a method for obtaining distance information of an object according to an embodiment.

[0078] Processor 140 may acquire object region information 711 for the first object ob1 based on input image 710 including the first object ob1. Processor 140 may acquire depth information 730 corresponding to input image 710. Object region information 711 and depth information 730 may respectively correspond to Figure 6 The object region information 611 and depth information 620 are omitted, so repeated descriptions will be omitted.

[0079] The processor 140 can acquire a feature map 720 including a feature region 721 for a first object ob1 based on the input image 710. The feature map 720 may include location information related to the feature region 721 for the first object ob1. The feature region 721 may be located within a bounding box corresponding to the object region information 711.

[0080] Processor 140 can acquire or define distance information of the first object ob1 based on feature region 721 for the first object ob1. For example, processor 140 can acquire distance information of the first object ob1 based on sub-depth information 731 in depth information 730 corresponding to feature region 721 for the first object ob1. Specifically, processor 140 can map representative values ​​of pixels corresponding to sub-depth information 731 to feature map 720 and acquire distance information of the first object ob1.

[0081] As described above, processor 140 can obtain the distance information of the first object ob1 based on the sub-depth information 731 corresponding to the feature region 721, rather than the depth information corresponding to the entire bounding box. Therefore, processor 140 can prevent the problem of the depth information of the second object ob2 being reflected when obtaining the distance information of the first object ob1. Thus, according to the embodiment, in the case where the target object (i.e., the person as the first object) and the obstacle object (i.e., the table) overlap, processor 140 can obtain accurate distance information for the target object. That is, according to the embodiment, the accuracy of the distance information of the first object ob1 can be improved.

[0082] Feature maps 720 can be obtained based on intermediate feature values ​​of neural network models.

[0083] Figure 8 This is a diagram illustrating a method for obtaining a feature map according to an embodiment.

[0084] Reference Figure 8 The processor 140 can obtain the final feature value 820 by inputting the input image 801 into the neural network model 810. The final feature value 820 may include type information 821 and object region information 822 for the first object ob1. Figure 8 The neural network model 810 in the text can correspond to Figure 4The neural network model 410 in the text will be omitted due to repetitive descriptions.

[0085] The neural network model 810 may include multiple intermediate layers 830. The processor 140 may obtain at least one intermediate feature value from the multiple intermediate layers 830. For example, the processor 140 may obtain a first intermediate feature value 831, a second intermediate feature value 832, and a third intermediate feature value 833 output by all or some of the intermediate layers 830. The processor 140 may obtain a feature map 834 by summing the first intermediate feature value 831, the second intermediate feature value 832, and the third intermediate feature value 833. The feature map 834 may be calculated as the sum of the weights w1 and w1 multiplied by the first intermediate feature value 831, w2 and w2 multiplied by the second intermediate feature value 832, and w3 and w3 multiplied by the third intermediate feature value 833.

[0086] Each of the first intermediate feature value 831, the second intermediate feature value 832, and the third intermediate feature value 833 may include location information for a feature region of at least one object. For example, the first intermediate feature value 831 may include location information for a feature region of a first object ob1.

[0087] The above description is based on an example where there is one object in the input image from which distance information will be extracted. However, this is not limiting, and the processor 140 can acquire distance information for multiple objects included in the input image.

[0088] Figure 9 This is a diagram illustrating a method for obtaining distance information of an object according to an embodiment.

[0089] Processor 140 can acquire an input image 910 including a first object ob1 and a second object ob2, and depth information 930 corresponding to the input image 910. Processor 140 can acquire object region information for each of the first object ob1 and the second object ob2 based on the input image 910. Processor 140 can acquire object region information for each of the first object ob1 and the second object ob2 based on the final feature values ​​obtained by inputting the input image 910 into a neural network model.

[0090] Processor 140 may acquire feature map 920 based on input image 910. Feature map 920 may include information relating to a first feature region 921 for a first object ob1 and a second feature region 922 for a second object ob2. Processor 140 may identify the first feature region 921 and the second feature region 922, respectively. Processor 140 may acquire information for each of the first feature region 921 and the second feature region 922 based on intermediate feature values ​​obtained by inputting input image 910 into a neural network model.

[0091] Processor 140 can obtain distance information for each of the first object ob1 and the second object ob2 based on feature map 920 and depth information 930. Specifically, processor 140 can identify a first feature pixel 931 corresponding to the first feature region 921 and a second feature pixel 932 corresponding to the second feature region 922 in the depth information 930. Processor 140 can obtain a first representative value based on the first feature pixel 931. Processor 140 can obtain a second representative value based on the second feature pixel 932. Processor 140 can map the first representative value to the first feature region 921 and the second representative value to the second feature region 922, and obtain the distance information for the first object ob1 and the second object ob2.

[0092] As described above, the processor 140 can obtain the distance to the object based on the feature region. Therefore, the bounding box of each of the first object ob1 and the second object ob2 is as follows: Figure 9 In cases where the frames overlap as described above, the processor 140 can accurately obtain distance information for each of the first object ob1 and the second object ob2. However, this is not limiting, and the frames do not necessarily need to overlap and can simply touch or separate.

[0093] Figure 10 This is a diagram illustrating a method for obtaining distance information of an object according to an embodiment.

[0094] Processor 140 can input an input image 1001, including a first object ob1 and a second object ob2, into neural network model 1100. Neural network model 1100 may include at least one intermediate layer corresponding to an object of a specific type. For example, neural network model 1100 may include a first intermediate layer 1110 corresponding to the first object ob1 of a first type (i.e., a person). Neural network model 1100 may include a second intermediate layer 1120 corresponding to the second object ob2 of a second type (i.e., a dog). The first intermediate layer 1110 may output a first intermediate feature value 1210. Processor 140 may obtain information about a first feature region 1211 corresponding to the first object ob1 based on the first intermediate feature value 1210. The second intermediate layer 1120 may output a second intermediate feature value 1220. Processor 140 may obtain information about a second feature region 1221 corresponding to the second object ob2 based on the second intermediate feature value 1220.

[0095] As described above, the intermediate layers included in the neural network model 1100 may correspond to specific types of objects. The processor 140 can obtain type information for the first object ob1 and the second object ob2 without the final feature values ​​of the neural network model 1100. The processor 140 can obtain distance information for each of the first object ob1 and the second object ob2 based on the depth information 1300 corresponding to the input image 1001. Specifically, the processor 140 can obtain first sub-depth information 1310 corresponding to the first feature region 1211 based on information about the first feature region 1211. The processor 140 can map the first sub-depth information 1310 to a first intermediate feature value 1210 and obtain distance information for the first object ob1. Similarly, the processor 140 can obtain second sub-depth information 1320 corresponding to the second feature region 1221 based on information about the second feature region 1221. The processor 140 can map the second sub-depth information 1320 to a second intermediate feature value 1220 and obtain distance information for the second object ob2.

[0096] Although only two objects are described, the number of objects and object types can be greater than 2, for example, 3, 4, up to 10, etc.

[0097] For example, as described above, when at least one or some of the intermediate layers included in the neural network model correspond to a specific type of object, the processor 140 can obtain the object's distance information based on intermediate feature values ​​rather than feature maps.

[0098] The processor 140 can generate an image based on the acquired information about the object.

[0099] Figure 11 This is a diagram showing the output image according to an embodiment.

[0100] Processor 140 can generate output image 1400 based on object type information, distance information, and object region information. The output image may include a display of the object region. For example, the output image may include a bounding box displayed around the object region. Furthermore, the object type information (i.e., ...) may be displayed around the bounding box in the output image. Figure 11 (People and dogs in the image). Additionally, in the output image, object type information or object distance information can be displayed around the object area.

[0101] The processor 140 can control a display, causing the generated output messages to be displayed on a display included in the electronic device 100 as described above. Optionally, the processor 140 can send signals to an external device via the communication interface 130, causing the generated output image to be displayed on another external device. Through these operations, the electronic device 100 can provide the user with various information about the object.

[0102] Figure 12 This is a flowchart illustrating a control method according to an embodiment.

[0103] Reference Figure 12 The control method may include an operation S1210 of acquiring an image and its depth information, an operation S1220 of inputting the acquired image into a neural network model and acquiring intermediate feature values, an operation S1230 of identifying feature regions of objects included in the image, and an operation S1240 of acquiring distance information between the electronic device and the object based on the feature regions and depth information.

[0104] During operation S1210, processor 140 can acquire an image and its depth information. Processor 140 can acquire an image of the surroundings of electronic device 100 captured by camera 110 of electronic device 100, along with the image's depth information. The image can be an RGB image captured by an RGB camera included in camera 110, and the depth information can be a depth image captured by a depth camera included in camera 110. Processor 140 can also acquire an image captured by an external device and its depth information.

[0105] In operation S1220, processor 140 can input the acquired image into a neural network model and obtain intermediate feature values. The neural network model can be trained to acquire information about objects included in the image. Information about the objects may include object type information, distance information, and location information.

[0106] In operation S1230, processor 140 can identify feature regions for objects included in an image. Processor 140 can identify type information for the object and information about the object regions based on the final feature values ​​obtained from the neural network model. Processor 140 can map feature maps obtained based on intermediate feature values ​​to the object regions and identify feature regions for the objects. Processor 140 can sum multiple intermediate feature values ​​obtained from predefined intermediate layers of the neural network model and obtain a feature map.

[0107] Furthermore, during operation S1240, processor 140 can acquire distance information between the electronic device and the object based on the feature region of the object and the depth information of the image. Processor 140 can acquire representative values ​​based on feature pixels corresponding to the feature region in the depth information of the image. Processor 140 can map the acquired representative values ​​to the feature region and acquire the distance information between the electronic device and the object.

[0108] Electronic device 100 can be implemented as various types of electronic devices. For example, electronic device 100 can be implemented as a cleaning robot, an automated guided vehicle (AGV), etc. Processor 140 can generate drive control signals for electronic device 100 based on the acquired information about the object.

[0109] The embodiments described above can be implemented using a recording medium readable by a computer or a computer-like device, or by using software, hardware, or a combination thereof. In some cases, the embodiments described herein can be implemented by one or more processors. Depending on the software implementation, the processes and functions described herein can be implemented by separate software modules. Each of these software modules can perform one or more functions and operations described herein.

[0110] An embodiment may be implemented as software comprising one or more instructions stored in a machine-readable (e.g., computer-readable) storage medium (e.g., internal or external memory). A processor may invoke the instructions from the storage medium and may operate according to the invoked instructions. When the instructions are executed by the processor, the processor may perform the function corresponding to the instructions directly or under the processor's control using other components. The instructions may comprise code written by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium.

[0111] Non-transitory computer-readable media are media that store data semi-permanently and can be read by a machine, rather than media that store data temporarily (such as registers, caches, and memory). Specific examples of non-transitory computer-readable media include CDs, DVDs, hard drives, Blu-ray discs, USB drives, memory cards, ROMs, etc.

[0112] While specific embodiments have been shown and described with reference to the accompanying drawings, these embodiments are provided for illustrative purposes, and those skilled in the art will understand that various modifications and equivalent embodiments may be made from this disclosure. Therefore, the true technical scope of this disclosure is defined by the technical spirit of the claims.

Claims

1. A method for controlling an electronic device, the method comprising: Acquire the image and its depth information; The acquired image is input into a neural network model, wherein the neural network model is trained to acquire information related to one or more objects included in the acquired image; Obtain the intermediate feature values ​​output by the intermediate layers of the neural network model and the final feature values ​​output by the final layer of the neural network model; Based on the final feature value, object region information for at least one of the one or more objects is obtained, wherein the object region information includes coordinate values ​​set to include at least one bounding box of the at least one object; Based on the intermediate feature values ​​and object region information, feature regions for the at least one object are identified, wherein the feature regions represent regions that become features of the object when the neural network model identifies the object; and Distance information between the electronic device and the at least one object is obtained based on the feature regions of the at least one object and the depth information. The method further includes: The intermediate feature values ​​are then input into the final layer of the neural network model. Obtain the final feature values ​​output by the final layer; and Based on the final feature values, type information for one or more objects included in the acquired image and information about one or more object regions in the acquired image are identified, wherein the one or more objects are respectively present in the one or more object regions. The step of identifying the feature region further includes: Based on the intermediate feature values, feature maps are obtained, including feature regions for each of the one or more objects. Map the feature map to the object region for each of the one or more objects; and Based on the mapping, feature regions corresponding to each of the one or more objects are identified. The intermediate layer includes a first intermediate layer corresponding to a first object of a first type among the one or more objects, and a second intermediate layer corresponding to a second object of a second type among the one or more objects. The identification step further includes: identifying a first feature region for a first object of a first type among the one or more objects based on a first intermediate feature value output by a first intermediate layer, and identifying a second feature region for a second object of a second type among the one or more objects based on a second intermediate feature value output by a second intermediate layer. The steps for obtaining the distance information further include: The first representative value, calculated based on the first feature pixel corresponding to the first feature region in the depth information, is mapped to the first feature region; and the second representative value, calculated based on the second feature pixel corresponding to the second feature region in the depth information, is mapped to the second feature region. Based on the mapping, the distance information between the electronic device and the first object and the distance information between the electronic device and the second object are obtained respectively. The at least one object is one of the first object and the second object.

2. The method of claim 1, wherein, The step of obtaining the feature map further includes: The feature map is obtained by summing multiple intermediate feature values ​​obtained from some of the intermediate layers. The intermediate layer includes some or all of the plurality of intermediate layers.

3. The method of claim 1, wherein, The step of obtaining the distance information further includes: obtaining distance information between the electronic device and the one or more objects based on the feature region and the depth information, respectively. The method further includes: The output image is generated based at least on the final feature values; and The type information and the distance information are displayed on the output image around the one or more object regions, respectively, for the one or more objects.

4. The method of claim 1, wherein, The steps for obtaining the distance information also include: A representative value is obtained based on the feature pixels corresponding to the feature region present in the depth information; Map the representative value to the feature region; and The distance information is obtained based on the mapping.

5. The method of claim 1, wherein, The steps for obtaining the depth information further include: The depth information can be obtained by using a depth camera or a stereo camera, or by analyzing the acquired images.

6. The method of claim 1, further comprising: The drive control signal for the electronic device is generated based on the type information, the information about the object region, and the distance information.

7. An electronic device, comprising: The memory is configured to store at least one instruction; as well as The processor is configured to execute the at least one instruction. The processor is further configured to execute the at least one instruction as follows: Acquire the image and its depth information. The acquired image is input into a neural network model, which is trained to acquire information related to one or more objects included in the acquired image. Obtain the intermediate feature values ​​output by the intermediate layers of the neural network model and the final feature values ​​output by the final layer of the neural network model. Based on the final feature value, object region information for at least one of the one or more objects is obtained, wherein the object region information includes coordinate values ​​set to include at least one bounding box of the at least one object; Based on the intermediate feature values ​​and object region information, feature regions for the at least one object are identified, wherein the feature region represents the region that becomes a feature of the object when the neural network model identifies the object, and Distance information between the electronic device and the at least one object is obtained based on the feature regions of the at least one object and the depth information. The processor is also configured as follows: The intermediate feature values ​​are then input into the final layer of the neural network model. Obtain the final feature values ​​output by the final layer. Based on the final feature values, type information for one or more objects included in the acquired image and information about one or more object regions in the acquired image are identified, wherein the one or more objects are respectively present in the one or more object regions. The processor is also configured as follows: Based on the intermediate feature values, a feature map is obtained, comprising feature regions for the one or more objects included in the obtained image. Map the feature map to the object region for each of the one or more objects; and Based on the mapping, feature regions corresponding to each of the one or more objects are identified. The intermediate layer includes a first intermediate layer corresponding to a first object of a first type among the one or more objects, and a second intermediate layer corresponding to a second object of a second type among the one or more objects. The processor is further configured to: identify a first feature region for a first object of a first type among the one or more objects based on a first intermediate feature value output by a first intermediate layer, and identify a second feature region for a second object of a second type among the one or more objects based on a second intermediate feature value output by a second intermediate layer. The processor is also configured as follows: The first representative value, calculated based on the first feature pixel corresponding to the first feature region in the depth information, is mapped to the first feature region; and the second representative value, calculated based on the second feature pixel corresponding to the second feature region in the depth information, is mapped to the second feature region. Based on the mapping, the distance information between the electronic device and the first object and the distance information between the electronic device and the second object are obtained respectively. The at least one object is one of the first object and the second object.

8. The electronic device as claimed in claim 7, wherein, The processor is also configured as follows: Map the feature map to the object region for each of the one or more objects, and Based on the mapping, feature regions corresponding to each of the one or more objects are identified.

9. The electronic device of claim 7, wherein, The feature map is obtained by summing multiple intermediate feature values ​​obtained from some of the intermediate layers. The intermediate layer includes some or all of the plurality of intermediate layers.

10. The electronic device of claim 7, wherein, The processor is also configured as follows: Representative values ​​are obtained based on the feature pixels corresponding to the feature regions present in the depth information. Map the representative value to the feature region, and The distance information is obtained based on the mapping.

11. The electronic device of claim 7, wherein, The processor is also configured as follows: Based on the feature region and the depth information, distance information between the electronic device and the one or more objects is obtained respectively, and An output image is generated based at least on the final feature values, such that the generated output image includes the type information and the distance information for the one or more objects, respectively.

12. The electronic device of claim 11, further comprising: monitor, The processor is also configured as follows: The control display shows the output image such that the type information and distance information for the one or more objects are displayed around the one or more object regions, respectively.

Images