Electronic device for recognizing an object by distorting an image and control method thereof

Abstract

An electronic device is provided. The electronic device includes a sensor, a camera, a memory, and a processor configured to connect to the sensor, the camera, and the memory. The memory includes an artificial intelligence model trained to recognize at least one object. The processor is further configured to: detect an object based on sensing data received from the sensor; warp an object region including the detected object in an image acquired via the camera based on distance information of the object region based on the detected object being recognized as having a height less than a predetermined threshold; and recognize the detected object by inputting the warped object region into the artificial intelligence model.

Description

TECHNICAL FIELD

[0001] The disclosure relates to object recognition, and more particularly, to an electronic device and a control method thereof, in which an object region including a flat object in an image acquired via a camera is corrected according to distance information, and object recognition is performed by using the corrected object region. BACKGROUND

[0002] An electronic device that performs object recognition generally recognizes an object in an image by inputting an image photographed via a camera into an artificial intelligence model.

[0003] However, there is a problem in the related art that an object recognition rate or accuracy is not constant according to an angle at which a surrounding object is photographed by a camera provided in an electronic device, a distance between the camera and the surrounding object, and the like.

[0004] In particular, for an object photographed at a relatively long distance from the camera, the object recognition rate or accuracy is relatively low. In addition, when a position of the camera is relatively low, for example, a camera included in a robot cleaner, for an object placed on the ground and a flat surface, the object recognition rate or accuracy is relatively low.

[0005] In order to solve such a problem, methods such as using a camera capable of photographing a high-resolution image, or dividing a photographed image into a plurality of regions and sequentially recognizing an object for each of the plurality of regions have been used, but these methods have a disadvantage that a calculation amount increases and a calculation rate decreases, while an effect of object recognition is not significantly improved. SUMMARY

[0006] TECHNICAL PROBLEM

[0007] Various embodiments provide an electronic device and a control method thereof capable of performing object recognition at a high object recognition rate regardless of a viewing angle or a photographing angle of an object photographed via a camera.

[0008] More particularly, embodiments provide an electronic device and a control method thereof capable of performing object recognition at a high object recognition rate even for a flat object (or an object having a relatively low height) placed on the ground.

[0009] TECHNICAL SOLUTION

[0010] According to aspects of the disclosure, an electronic device is provided, including a sensor, a camera, a memory, and a processor, wherein the memory includes an artificial intelligence model trained to recognize an object, and the processor is configured to: detect an object by sensing sensing data received from the sensor, recognize whether the detected object is a ground object that the electronic device is capable of crossing or climbing over, based on the detected object being recognized as the ground object, warp an object region including the detected object in an image acquired via the camera based on distance information of the object region, and recognize the detected object by inputting the warped object region into the artificial intelligence model.

[0011] According to aspects of the embodiment, an electronic device is provided, including a sensor, a camera, a memory, and a processor, wherein the memory includes an artificial intelligence model trained to recognize an object, and the processor is configured to: detect an object by sensing data received from the sensor, recognize whether the detected object is a ground object that the electronic device is capable of crossing or climbing over, based on the detected object being recognized as the ground object, warp an image acquired by the camera based on distance information of a plurality of regions in the image, and recognize the detected object by inputting the warped image into the artificial intelligence model.

[0012] According to aspects of the disclosure, an electronic device is provided, including a sensor, a camera, a memory, and a processor configured to be connected to the sensor, the camera, and the memory to control the electronic device, wherein the memory includes a plurality of artificial intelligence models trained to recognize a ground object that the electronic device is capable of crossing or climbing over, the plurality of artificial intelligence models being trained based on images of the ground object taken at different distances, and the processor is configured to: detect an object by sensing data received from the sensor, recognize whether the detected object is a ground object, based on the detected object being recognized as the ground object, warp a region including the detected object in an image acquired by the camera based on distance information of the object region, identify an artificial intelligence model trained based on an image corresponding to the distance information of the object region among the plurality of artificial intelligence models, and recognize the detected object by inputting the warped object region into the identified artificial intelligence model.

[0013] According to aspects of the disclosure, a control method of an electronic device including a memory in which an artificial intelligence model trained to recognize an object is stored is provided, the control method including: detecting an object by sensing data received from a sensor; recognizing whether the detected object is a ground object that the electronic device is capable of crossing or climbing over; based on the detected object being recognized as the ground object, warping an object region including the detected object in an image acquired via a camera based on distance information of the object region; and recognizing the detected object by inputting the warped object region into the artificial intelligence model.

[0014] According to an aspect of the disclosure, there is provided an electronic device including a sensor, a camera, a memory, and a processor configured to be connected to the sensor, the camera, and the memory, wherein the memory includes an artificial intelligence model trained to recognize at least one object, and wherein the processor is further configured to: detect an object based on sensing data received from the sensor; warp an object region including the detected object based on distance information of the object region in an image acquired via the camera based on the detected object being recognized as having a height less than a predetermined threshold; and recognize the detected object by inputting the warped object region into the artificial intelligence model.

[0015] According to an aspect of the disclosure, there is provided an electronic device including a sensor, a camera, a memory, and a processor configured to be connected to the sensor, the camera, and the memory, wherein the memory includes a plurality of artificial intelligence models trained to recognize at least one object, wherein the plurality of artificial intelligence models are trained based on images of the at least one object taken at different distances, and wherein the processor is further configured to: detect an object based on sensing data received from the sensor; warp an object region including the detected object based on distance information of the object region in an image acquired via the camera based on the detected object being recognized as having a height less than a predetermined threshold; identify an artificial model trained based on an image corresponding to the distance information of the object region among the plurality of artificial intelligence models; and recognize the detected object by inputting the warped object region into the identified artificial intelligence model.

[0016] According to an aspect of the disclosure, there is provided a method of controlling an electronic device including a memory in which an artificial intelligence model trained to recognize an object is stored, the method including: detecting an object based on sensing data received from a sensor; identifying that the detected object is passable or climbable by the electronic device when the electronic device is driven to move; warping an object region including the detected object based on distance information of the object region in an image acquired via a camera based on the detected object being identified as passable or climbable by the electronic device; and recognizing the detected object by inputting the warped object region into the artificial intelligence model.

[0017] Advantageous Effects

[0018] The electronic device and the control method according to the disclosure have an effect that an object around the electronic device can be accurately recognized regardless of a height of the object and an angle of a camera provided in the electronic device.

[0019] In particular, the electronic device according to the embodiment has an advantage that a flat object photographed even through a low-positioned camera can be recognized with high precision. BRIEF DESCRIPTION OF DRAWINGS

[0020] The above and other aspects, features and advantages of certain embodiments of the disclosure will be more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0021] Figure 1 is a diagram illustrating an example of correcting an object region including an object to recognize the object in an electronic device according to an embodiment;

[0022] Figure 2a is a block diagram illustrating a configuration of an electronic device according to an embodiment;

[0023] Figure 2b is a block diagram illustrating a functional configuration of an electronic device according to an embodiment;

[0024] Figure 2c is a block diagram illustrating a functional configuration of an electronic device according to an embodiment;

[0025] Figure 2d is a block diagram illustrating a functional configuration of an electronic device according to an embodiment;

[0026] Figure 3a is a diagram illustrating an example of detecting an object by using a sensor in an electronic device according to an embodiment;

[0027] Figure 3b is a diagram illustrating an example of recognizing a height of a detected object with respect to a ground in an electronic device according to an embodiment;

[0028] Figure 3c is a diagram illustrating an example of recognizing an object region from an image acquired via a camera in an electronic device according to an embodiment;

[0029] Figure 3d is a diagram illustrating an example of recognizing an object by selectively distorting an object region according to whether the object is a ground object in an electronic device according to an embodiment;

[0030] Figure 4 is a diagram illustrating a method of distorting an image according to a reference distance in an electronic device according to an embodiment;

[0031] Figure 5a and Figure 5b is a diagram illustrating an example according to an embodiment in which a degree of distortion changes according to a difference between a distance of a nearest pixel and a distance of a farthest pixel in an object region;

[0032] Figure 6 is a diagram illustrating an example according to an embodiment in which a position of a pixel is transformed according to a difference between a reference distance and a distance of each pixel;

[0033] Figure 7ais a block diagram illustrating a configuration of an electronic device using an artificial intelligence model trained to recognize a ground object according to an embodiment;

[0034] Figure 7b is a block diagram illustrating an example of a functional configuration of an electronic device; Figure 7a

[0035] Figure 8 is a diagram illustrating an example of using an artificial intelligence model according to whether a detected object is a ground object in an electronic device according to an embodiment;

[0036] Figure 9a is a block diagram illustrating an example of using a plurality of artificial intelligence models trained to recognize a ground object for each distance at which a ground object is photographed in an electronic device according to an embodiment;

[0037] Figure 9b is a block diagram illustrating an example of a functional configuration of an electronic device; Figure 9a

[0038] Figure 10 is a diagram illustrating a training process of a plurality of artificial intelligence models stored in an electronic device; Figure 9a

[0039] Figure 11 is a diagram illustrating an example of using an artificial intelligence model according to whether an object is a ground object and distance information of a ground object in an electronic device according to an embodiment;

[0040] Figure 12 is a block diagram illustrating a detailed configuration of an electronic device according to various embodiments;

[0041] Figure 13a is a block diagram illustrating a configuration of an electronic device including a plurality of processors according to an embodiment;

[0042] Figure 13b is a block diagram illustrating an example of a more detailed configuration of an electronic device; Figure 13a

[0043] Figure 14 is a flowchart illustrating a method of controlling an electronic device according to an embodiment;

[0044] Figure 15 is a flowchart illustrating an example of warping an object region and recognizing an object according to whether a ground object is detected in a method of controlling an electronic device according to an embodiment;

[0045] Figure 16 is a flowchart illustrating an example of using an artificial intelligence model according to whether a ground object is detected in a method of controlling an electronic device according to an embodiment; and

[0046] ​​​​Figure 17 is a flowchart illustrating an example of a method of controlling an electronic device based on whether a ground object is detected and an object recognition result according to an embodiment. DETAILED DESCRIPTION

[0047] The terms used in the specification and claims have been selected with consideration of functions in various embodiments. However, the terms can be changed according to the intention of a person skilled in the art, the legal or technical interpretation, and the appearance of new technologies, and in addition, some terms are arbitrarily selected by the applicant. The terms can be interpreted based on the meaning defined in the specification and also based on the general meaning of the terms in the related art and common technical knowledge of those skilled in the art, not the specific definition of the terms.

[0048] In addition, in the accompanying drawings, which form a part hereof, like reference numerals or symbols are used to designate parts or components that perform the same or similar functions. For the ease of explanation and understanding, the same reference numerals or symbols will be used to describe different embodiments. That is, although all components are shown with the same reference numerals in the drawings, the drawings do not imply one embodiment.

[0049] In addition, in the specification and claims, terms including ordinal numbers such as "first" and "second" can be used to distinguish components. The ordinal numbers are used to distinguish components from one another, and the meaning of the terms should not be interpreted as being limited by the use of the ordinal numbers. As an example, components coupled to the ordinal numbers should not be interpreted as being limited to the order, layout order, or the like by the ordinal numbers. The corresponding ordinal numbers can be used interchangeably if necessary.

[0050] In the present specification, the singular expression includes the plural expression, unless the context clearly dictates otherwise. It will also be understood that the term "comprise" or "consist of", as used in this application, lists the existence of the features, numbers, steps, operations, components, parts, or combinations thereof mentioned in the specification, but does not exclude the existence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0051] In the embodiments, the terms "module", "unit", "part", and the like are terms used to refer to components that perform at least one function or operation, and such components can be implemented in hardware or software or a combination of hardware and software. In addition, a plurality of "modules", "units", "components", and the like can be integrated into at least one module or chip, and can be implemented in at least one processor, except for the case where each of them needs to be implemented in a specific hardware.

[0052] Also, in an embodiment, it will be understood that when an element is referred to as being "connected to" another element, it can be directly connected to the other element or be indirectly connected via another element. Also, a "comprising" of any list of components is to be construed as meaning including but not excluding any other components.

[0053] The expression, for example, "at least one of a, b or c", when preceding a list of two or more items, modifies the entire list of items and does not modify the list of items as a whole. For example, the expression "at least one of a, b or c" indicates only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

[0054] According to an embodiment, a function related to artificial intelligence (AI) can be operated via a processor and a memory. The processor can include one or more processors. The one or more processors can include a general-purpose processor, such as a central processing unit (CPU), an application processor (AP), a digital signal processor (DSP), a dedicated graphic processor such as a graphic processing unit (GPU) or a visual processing unit (VPU), a dedicated AI processor such as a neural processing unit (NPU), etc., but are not limited thereto. The one or more processors can control input data to be processed according to a predetermined operation rule or an AI model stored in the memory. When the one or more processors are a dedicated AI processor, the dedicated AI processor can be designed to have a hardware structure specialized for processing a specific AI model.

[0055] The predefined operation rule or AI model can be created through a training process. The predefined operation rule or AI model can be set, for example, to perform a desired characteristic (or purpose) created by training a base AI model through a learning algorithm using a large amount of training data. The training process can be performed by a device for performing AI or a separate server and / or system. Examples of the learning algorithm can include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning, but embodiments of the disclosure are not limited thereto.

[0056] The AI model can include a plurality of neural network layers. Each of the neural network layers can have a plurality of weight values, and can perform various neural network calculations via arithmetic operations on a result of a calculation in a previous layer and the plurality of weight values in the current layer. The plurality of weights in each of the neural network layers can be optimized by a result of training the AI model. For example, the plurality of weights can be updated to reduce or minimize a loss or cost value acquired by the AI model during a training process. The artificial neural network can include, for example, and without limitation, a deep neural network (DNN), and can include, for example, and without limitation, a convolutional neural network (CNN), a DNN, a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent DNN (BRDNN), a deep Q-network (dQn), etc., but is not limited thereto.

[0057] Figure 1 FIG. 1 is a diagram illustrating an example of correcting (or scaling) an object region including an object to recognize the object in an electronic device according to an embodiment.

[0058] Referring to Figure 1 , the electronic device 10 can detect a surrounding object through a sensor in a process of moving and photographing a region (e.g., a front region) around the electronic device by using, for example, a camera included in the electronic device 10. For example, the electronic device 10 can be implemented as a robot cleaner. Figure 1 In the electronic device 10, for the purpose of illustration, it is assumed that an object 1 in a front region is detected in a process of being driven.

[0059] The electronic device 10 can recognize the object 1 as a defecation pad 1' by inputting at least a part of an image 11 photographed via the camera into an artificial intelligence model for object recognition.

[0060] However, referring to Figure 1 , the object 1 is photographed at a very low angle according to a height of the electronic device 10 implemented as a robot cleaner, and thus the object 1 in the image 11 can appear on the image 11 in a distorted form having a very narrow vertical length.

[0061] That is, when the electronic device 10 inputs the image 11 into the artificial intelligence model for object recognition, it is highly likely that the "defecation pad" cannot be recognized due to the distorted form of the object 1 in the image 11.

[0062] To solve this problem, Figure 1The electronic device 10 can identify the height of the object 1 detected through the sensor, and based on the identified height of the object 1, the electronic device 10 can identify that the object 1 is an object having a relatively low height, i.e., the object 1 corresponds to an object placed on a ground (hereinafter, referred to as a "ground object") that the electronic device 10 can pass through or climb over. Although the term "ground object" includes "ground", it should be understood that the term is used only to describe the present disclosure and simplify the description, and does not indicate or imply that the mentioned element must have a specific positional relationship (e.g., contact with the ground) with the "ground", and thus the term should not be interpreted as limiting the present disclosure. The term "ground object" is intended to include any object in an image of an object that has a distorted form or shape due to the object being photographed at a low angle.

[0063] In this case, the electronic device 10 can identify an object region 21 of the image 11 acquired via the camera, which corresponds to the detected object 1, and then warp the object region 21 so that the object region 21 is generally enlarged and so that a warped object region 21' as shown in Figure 1 is obtained. As a result, with reference to the warped object region 21' that is warped in Figure 1 , the shape of the diaper pad 1' in the image is restored to be similar to the actual shape of the diaper pad.

[0064] In addition, the electronic device 10 can identify the object 1 as the diaper pad 1' by inputting the warped object region 21' into an artificial intelligence model for object recognition.

[0065] As such, because the electronic device 10 according to the embodiment corrects an object region including a ground object and then identifies the ground object based on the corrected object region, the accuracy of object recognition can be improved even if the ground object is photographed at a relatively low angle.

[0066] Hereinafter, the configuration and operation of the electronic device according to various embodiments will be described in greater detail with reference to the accompanying drawings.

[0067] Figure 2a is a block diagram illustrating a configuration of an electronic device according to an embodiment.

[0068] Referring to Figure 2a , the electronic device 100 can include a sensor 110, a camera 120, a storage 130, and a processor 140. The electronic device 100 can be implemented as one or more various terminal devices, such as a robot cleaner, a health care robot, a smart phone, a tablet personal computer (PC), and a wearable device, but is not limited thereto.

[0069] The processor 140 according to an embodiment can detect an object around the electronic device 100 through the sensor 110. The processor 140 can identify a height of the detected object using sensing data of the sensor 110.

[0070] In this case, the processor 140 can determine whether the detected object is a specific type of object (e.g., a floor object). In an embodiment, the electronic device 100 can be a terminal device located on a specific location (e.g., a floor), as in the example of a robot cleaner. In this embodiment, the processor 140 can identify a height from a floor on which the electronic device 100 is located to the detected object, and can identify the detected object as a floor object when the identified height is less than a predetermined threshold.

[0071] The floor object includes an object placed on a floor, and can refer to an object having a height from a floor less than a threshold. The floor can refer to a surface of a floor on which the electronic device 100 is placed. For example, the floor can be a flat surface. The floor does not need to be completely flat, and can be identified as a floor by the electronic device 100 if a degree of curvature in the floor is less than a threshold value.

[0072] The degree of curvature in the floor can refer to a degree of deformation or distortion of a plane including two or more points based on a floor on which the electronic device 100 is placed, and for example, a portion of the floor that is concave or partially protruding can be identified as not being a floor depending on the degree of curvature of the portion of the floor.

[0073] Specifically, the degree of curvature of a specific portion of the floor can correspond to a distance of the specific portion from a plane including two or more points of a floor on which the electronic device 100 is placed, and when the distance between the specific portion of the floor and the plane including two or more points of the floor on which the electronic device 100 is placed is a threshold value or more, the corresponding portion of the floor can be identified as not being a floor.

[0074] The floor object according to an embodiment is not any object that is simply placed on a floor, but refers to an object that is not only placed on a floor but also has a height from the floor less than a predetermined threshold. For example, a potty mat, a carpet, a mat, a threshold, etc. having a height less than a certain value can correspond to a floor object.

[0075] The threshold described above, which is a reference for determining whether a detected object is a floor object, can be pre-set based on a maximum height of an object that the electronic device 100 can pass over or climb over by using a moving means provided in the electronic device 100. For example, the threshold can be determined based on a length of a diameter of a wheel constituting the moving means of the electronic device 100. In addition, the threshold can be determined based on a threshold value related to the degree of curvature in the floor described above.

[0076] The processor 140 can acquire an image including the detected object through the camera 120. Here, the processor 140 can control the camera 120 to photograph an image of the surrounding environment at a predetermined time interval, and / or can control the camera 120 to photograph an image in a direction in which the detected object is located only when the sensor 110 detects the object.

[0077] The processor 140 can identify an object region including the detected object in the image by using sensing data, an angle / angle of view of the camera 120 photographing the image, a time point at which the image is photographed, etc. In an embodiment, the processor 140 can use at least one artificial intelligence model trained to detect an object region from an image.

[0078] If the detected object is identified as a ground object, the processor 140 can warp the object region in the image based on distance information of the object region.

[0079] The distance information refers to a distance between a specific portion of the object represented by each pixel in the image (or the object region) and the electronic device 100 and / or the camera 120. The processor 140 can acquire the distance information through the sensor 110. In an embodiment, the sensor 110 can be implemented as a three-dimensional (3d) sensor, a light detection and ranging (LiDAR) sensor, an ultrasonic sensor, etc.

[0080] Warping can refer to an operation of correcting an image by rotating the image along an X-axis and / or a Y-axis or using scaling, etc. When warping is performed, a position of each pixel in the image can be transformed by using a predetermined position transformation function, etc.

[0081] Specifically, the processor 140 can acquire distance information of each of a plurality of pixels constituting the object region including the ground object, and can transform positions of the plurality of pixels based on the acquired distance information.

[0082] As a result, among a plurality of regions included in the object region, a region farther from the electronic device 100 and / or the camera 120 than a reference distance can be enlarged, and a region closer to the electronic device 100 and / or the camera 120 than the reference distance can be reduced. A more detailed description will be described later with reference to Figure 4 etc.

[0083] The processor 140 can identify an object included in the warped object region. In an embodiment, the processor 140 can identify the object by inputting the warped object region into an artificial intelligence model 135 stored in the memory 130. The artificial intelligence model 135 can be a model trained to identify an object included in an image when the image (e.g., an object region in the image) is input.

[0084] Specifically, the processor 140 can acquire information about a ground object included in the object region by loading the artificial intelligence model 135 stored in the memory 130 into a volatile memory included in or connected to the processor 140 and inputting the distorted object region into the loaded artificial intelligence model 135.

[0085] The sensor 110 can detect a surrounding object of the electronic device 100. The processor 140 can detect a surrounding object of the electronic device 100 based on sensing data generated by the sensor 110.

[0086] The camera 120 can acquire one or more images around the electronic device 100. The camera 120 can be implemented as a red, green, and blue (RGB) camera, a depth camera, or the like.

[0087] Various information related to a function of the electronic device 100 can be stored in the memory 130. The memory 130 can include read-only memory (ROM), random access memory (RAM), a hard disk, a solid state drive (SSD), a flash memory, or the like.

[0088] An artificial intelligence model 135 trained to recognize an object can be stored in the memory 130. When an image is input to the artificial intelligence model 135, the artificial intelligence model 135 can acquire information (e.g., a name, a type, a product name, a person's name, or the like) about an object from the input image.

[0089] Specifically, when an image corresponding to an object region including an object is input to the artificial intelligence model 135, the artificial intelligence model 135 can operate as a classifier that selectively outputs information corresponding to an object included in the object region among information output by the artificial intelligence model 135.

[0090] The processor 140 can be connected to the sensor 110, the camera 120, and the memory 130 to control the electronic device 100.

[0091] Figure 2b is a block diagram illustrating a functional configuration of an electronic device according to an embodiment.

[0092] Referring to Figure 2bThe electronic device 100 can include an object detection module 210, an object region detection module 220, a warping module 230, and an object recognition module 240. One or more of these modules 210-240 can be implemented in the form of software stored on the memory 130 and executed by the processor 140, or in the form of hardware including circuitry controlled by the processor 140. Furthermore, the modules 210-240 can be implemented in a combination of software and hardware executed and / or controlled by the processor 140. The configuration of the electronic device 100 is not limited to the above-described components, and some components can be added, omitted, or merged as needed. For example, referring to Figure 2d The electronic device 100 can not include the object region detection module 220 among the above-described modules 210-240.

[0093] Hereinafter, the operation of each module and the processor 140 will be described.

[0094] The processor 140 can detect an object around the electronic device 100 through the object detection module 210. The object detection module 210 can detect an object by using sensing data received from the sensor 110.

[0095] As an example, the object detection module 210 can identify a ground by using sensing data from the sensor 110 such as a LiDAR sensor, an ultrasonic sensor, a 3d sensor, an infrared sensor, etc., and can detect an object having a certain distance or depth from the ground.

[0096] Here, the object detection module 210 can identify a direction and a position of the detected object with respect to the electronic device 100. Furthermore, the object detection module 210 can also identify a size or a height of the detected object by using the sensing data.

[0097] The object detection module 210 can use an algorithm or one or more artificial intelligence models to respectively detect a ground, an obstacle (e.g., a wall, an object, etc.) based on the sensing data of the sensor 110.

[0098] The object region detection module 220 can detect an object region including a detected object among images acquired through the camera 120.

[0099] Specifically, the object region detection module 220 can identify an object region (e.g., a pixel) matching a direction, a position, and a size of a detected object based on sensing data from the sensor 110 within an image.

[0100] To this end, the object region detection module 220 can use a relative position of the camera 120 and the sensor 110, an angle / angle of view of the camera 120 at a time point of photographing an image, etc.

[0101] The warping module 230 is a component for warping at least one object region or image according to distance information of the object region or image.

[0102] The warping module 230 can transform a position of each pixel in the object region based on distance information of each pixel included in the object region or image. Specifically, the warping module 230 can transform the position of each pixel in the object region so that the distance information of each pixel in the object region matches a reference distance.

[0103] As a result, each of a plurality of regions included in the object region or included in the image can be individually scaled up or scaled down according to distance information of each of the plurality of regions.

[0104] The object recognition module 240 is a component for recognizing an object included in the object region. Specifically, when the artificial intelligence model 135 stored in the memory 130 is loaded onto the volatile memory of the processor 140, the object region can be input to the loaded artificial intelligence model 135 to recognize the object.

[0105] The processor 140 can warp the object region including the object recognized as a ground object according to distance information of the object region and input the warped object region into the artificial intelligence model 135, and can input the object region including the object recognized as not being a ground object into the artificial intelligence model 135 without performing warping according to distance information thereof.

[0106] However, although the processor 140 can not perform "warping" on the object region including the object recognized as not being a ground object, the processor 140 can perform minimal preprocessing (e.g., full-size scaling) for inputting the object region including the object recognized as not being a ground object into the artificial intelligence model 135.

[0107] On the other hand, referring to Figure 2c Operations of the object detection module 210 and the object region detection module 220 can be independently performed from each other.

[0108] In this case, the object detection module 210 can detect an object using sensing data received through the sensor 110 and identify whether the detected object is a ground object. In an embodiment, the object detection module 210 can detect an object using a depth image acquired through a 3D sensor or the like included in the sensor 110. In this case, the object detection module 210 can use an artificial intelligence model trained to detect an object from a depth image.

[0109] In an embodiment, the object detection module 210 can identify a height of an object detected through an ultrasonic sensor included in the sensor 110 or the like, and identify whether the object is a floor object based on the identified height.

[0110] The object region detection module 220 can identify an object region including an object from an image acquired through the camera 120. In this case, the camera 120 can be implemented as an RGB camera, and the object region detection module 220 can use an artificial intelligence model trained to identify an object region from an RGB image.

[0111] The warping module 230 can identify an object that matches an object region identified by the object region detection module 220 among objects detected by the object detection module 210.

[0112] As an example, the warping module 230 can identify an object that matches the identified object region by comparing a depth image including a detected object among depth images acquired through the sensor 110 (e.g., a 3d sensor) with an RGB image acquired through the camera 120, but is not limited thereto.

[0113] Further, if an object matching the object region is identified as a floor object, the warping module 230 can warp the object region according to distance information of the object region.

[0114] The object recognition module 240 can identify an object (e.g., a type of object) included in the warped object region.

[0115] Referring to Figure 2d , the electronic device 100 can not include the object region detection module 220.

[0116] Here, if a detected object is identified as a floor object, the processor 140 can warp the entire image including the floor object according to distance information using the warping module 230. Further, the processor 140 can input the warped image into the object recognition module 240 to identify a floor object in the image.

[0117] On the other hand, when a detected object is identified as not being a floor object, the processor 140 can input an unwarped image into the object recognition module 240 to identify an object in the image.

[0118] Hereinafter, embodiments of an electronic device using the modules described above Figure 3a , Figure 3b , Figure 3c and Figure 3d will be described in detail. Figure 2a

[0119] Figure 3a ​is a diagram illustrating an example in which an electronic device detects an object through a sensor according to an embodiment.

[0120] Referring to Figure 3a , the electronic device 100 implemented as a robot cleaner can be driven in a specific space 300. Referring to Figure 3a , a toilet mat 301, a book 302, and a flower pot 303 are located on the space 300.

[0121] During the driving, the electronic device 100 can detect a floor or an obstacle around the electronic device 100 through a sensor 110 implemented as, for example, an ultrasonic sensor, a LiDAR sensor, or the like. Figure 3a The image 300' illustrates an image of the surroundings of the electronic device 100 detected by the electronic device 100 according to the sensing data of the sensor 110.

[0122] Referring to Figure 3a , the electronic device 100 can detect a floor 305 by recognizing points located on a plane such as a floor on which the electronic device 100 is placed based on the sensing data.

[0123] In addition, the electronic device 100 can detect the objects 301, 302, and 303 at different distances or depths from the floor 305, respectively. The electronic device 100 can also detect points corresponding to a wall.

[0124] The object detection module 210 can use the sensing data received from the sensor 110 to recognize the position, size, height, and / or approximate shape of the detected object. In particular, the object detection module 210 can detect the height of each of the objects 301, 302, and 303 from the floor 305.

[0125] To this end, the sensor 110 can include a plurality of LiDAR sensors and / or a plurality of ultrasonic sensors arranged at predetermined intervals on the electronic device 100.

[0126] Referring to Figure 3b , the object detection module 210 can recognize the height of each of the objects 301, 302, and 303 from the floor based on the sensing data.

[0127] Here, the processor 140 can recognize the object 301, which is among the objects 301, 302, and 303, whose height is less than a threshold value, as a floor object, and can recognize the objects 302 and 303, whose heights are equal to or greater than the threshold value, as not being floor objects.

[0128] The threshold value can be a height that the electronic device 100 can pass over or climb over by using a moving means. For example, in an embodiment in which the electronic device 100 is a robot cleaner and a diameter of a wheel that is a moving means of the electronic device 100 is 4 cm, the threshold value can be set to 4 cm, or, for example, considering a possible error in the recognized height of the object, can be set to a value slightly larger than the diameter of the wheel (e.g., 6 cm).

[0129] Referring to Figure 3c , the object region detection module 220 can extract object regions 351, 352, and 353 from the image 350 acquired through the camera 120, respectively, according to a position, a size, and an approximate shape of each of the detected objects 301, 302, and 303.

[0130] Further, referring to Figure 3d , the warping module 230 can warp the object region 351 including the object 301, which is identified as a floor object, among the objects 301, 302, and 303, according to distance information of the object region 351.

[0131] On the other hand, the warping module 230 can not warp the object regions 352, 353 of the objects 302 and 303.

[0132] Referring to Figure 3d , the shape of the disposable pad in the warped object region 351' is generally enlarged compared to before warping, and a visible angle at which the disposable pad is visible is corrected, and thus, a high object recognition rate and / or a high object recognition accuracy can be achieved. The warping method according to an embodiment will be described later with reference to Figure 4 , Figure 5a , Figure 5b and Figure 6 .

[0133] Referring to Figure 3d , the object recognition module 240 can recognize the objects included in the warped object region 351' and the object regions 352 and 353, respectively.

[0134] Here, the object recognition module 240 can input the object regions 351', 352, and 353 into the artificial intelligence model 135, respectively, to recognize the disposable pad 301, the book 302, and the flower pot 303 included in each of the object regions 351', 352, 353.

[0135] When the electronic device 100 does not include the object region detection module 220 as shown in Figure 2d , the processor 140 can not perform the extraction of the object region 351 as shown in Figure 3c and Figure 3d .

[0136] In this case, the processor 140 can warp the image 350 itself including the ground object according to the distance information of the image 350 using the warping module 230. Further, the processor 140 can input the warped image into the object recognition module 240 to recognize the ground object in the image 350.

[0137] Hereinafter, an exemplary method in which the warping module 230 warps the object region of the ground object according to the distance information will be described with reference to Figure 4 、 Figure 5a 、 Figure 5b and Figure 6

[0138] In warping the object region, the processor 140 can transform the positions of the pixels included in the object region according to the reference distance.

[0139] Specifically, the processor 140 can acquire the distance information of each of the plurality of pixels constituting the object region, and can transform the positions of the plurality of pixels based on the acquired distance information.

[0140] As a result, in the plurality of regions included in the object region, a region farther than the reference distance from the electronic device 100 and / or the camera 120 can be enlarged, and a region closer than the reference distance from the electronic device 100 and / or the camera 120 can be reduced.

[0141] The reference distance can be determined in advance by the manufacturer or the user of the electronic device 100. In this case, the reference distance can be determined in advance within the distance range in which the artificial intelligence model 135 included in the storage 130 can recognize the object, and can be determined through experiments. For example, if the object recognition rate of the artificial intelligence model 135 is relatively high when the image of the object photographed at a distance between about 1 m and 3 m is input, the reference distance can be determined in advance as 2 m, which is obtained by averaging the minimum and maximum values of the distance range of 1 m and 3 m.

[0142] Alternatively, the reference distance can correspond to the distance of the region where the focus point of the camera 120 is located in the image including the object region. That is, the processor 140 can set the distance of the region where the focus point of the camera 120 is located in the image including the object region as the reference distance. In an embodiment, at least a part of the processor 140 and the camera 120 can be implemented in the same manner as a general auto-focus camera.

[0143] Alternatively, the reference distance can be set as the distance of the pixel having the closest distance to the electronic device 100 and / or the camera 120 among the plurality of pixels constituting the object region. In this case, the corresponding pixel having the closest distance is set as the reference pixel, as a result, the object region can be enlarged as a whole. ​

[0144] Figure 4 is a diagram illustrating a method of warping an image (e.g., an object region of a ground object) according to a reference distance in an electronic device according to an embodiment.

[0145] The processor 140 can identify a reference pixel corresponding to the reference distance among the plurality of pixels constituting the object region, and can identify a first pixel having a distance farther than the reference distance and a second pixel having a distance closer than the reference distance among the plurality of pixels.

[0146] Further, the processor 140 can transform a position of the first pixel such that a distance between the first pixel and the reference pixel increases, and can transform a position of the second pixel such that a distance between the second pixel and the reference pixel decreases.

[0147] Referring to Figure 4 , as a result of the processor 140 warping the object region 410, a warped object region 420 is obtained. Before warping, the object region 410 includes an upper portion and a lower portion with respect to a reference line 410-1 including a reference pixel corresponding to the reference distance. Pixels in the upper portion of the object region 410 can be farther away from the reference line 410-1, and pixels in the lower portion of the object region 410 can be closer to the reference line 410-1. As a result of warping, the upper portion and the lower portion of the object region 410 can be scaled up or down, and thus, an angle at which an object included in the object region 410 is visible (e.g., an angle of view of the object) can be improved.

[0148] The reference line 410-1 can be a straight line that bisects the image while including at least one pixel corresponding to the reference distance, or a line consisting only of pixels corresponding to the reference distance.

[0149] For ease of understanding, it can be conceptually explained that, in the warped object region 420, an upper portion above the reference line 410-1 is rotated to appear closer, and a lower portion below the reference line 410-1 is rotated to appear farther away.

[0150] Among the pixels included in the upper portion of the object region 410, a pixel having a large distance difference from the reference line 410-1 is rotated to appear closer to a greater extent, and among the pixels included in the lower portion of the object region, a pixel having a large distance difference from the reference line 410-1 is rotated to further away to a greater extent.

[0151] To this end, the processor 140 can transform a position of the first pixel such that an extent to which an interval between the reference pixel and the first pixel increases after the transformation becomes larger as a difference between the reference distance and a distance of the first pixel increases.

[0152] Further, the processor 140 can transform the position of the second pixel such that the degree of reduction in the interval between the reference pixel and the second pixel after the transformation becomes greater as the difference between the reference distance and the distance of the second pixel increases.

[0153] As a result, the degree to which each of the plurality of regions in the object region 410 is enlarged or reduced can be proportional to the distance between each of the plurality of regions and the reference line 410-1.

[0154] As such, as a result of adjusting the perspective of the plurality of regions in the object region 410, the plurality of regions in the distorted object region 420 can be provided as flat and appear to be located at the same distance (or almost similar distances to each other).

[0155] In Figure 4 In an embodiment, for convenience of understanding, the reference line 410-1 is illustrated as a straight line, but the reference line 410-1 can also have a different form, such as a curve. When the reference line 410-1 is a curve, preferably, the curve of the reference line has one extreme point (e.g., a concave curve or a convex curve). In this case, even if the distorted object region is inversely transformed back to the object region before the distortion, the distorted object region can be restored as it is to the existing object region.

[0156] In particular, in an embodiment in which the object recognition result based on the distorted object region is provided together with the image including the object region that was not distorted previously, the object region needs to be restored, and thus, the reference line 410-1 needs to be a straight line or a curve having one extreme point.

[0157] In an embodiment, the degree to which the plurality of regions in the image are enlarged or reduced as a result of the distortion can vary according to the difference between the distance of the nearest pixel and the distance of the farthest pixel in the object region.

[0158] Figure 5a and Figure 5b are graphs illustrating that the degree to which the plurality of regions in the object region are enlarged or reduced by the distortion can vary according to the difference between the distance of the nearest pixel and the distance of the farthest pixel in the object region. Figure 5a illustrates a case in which the difference between the distance of the nearest pixel and the distance of the farthest pixel in the object region is relatively large, while Figure 5b illustrates a case in which the difference between the distance of the nearest pixel and the distance of the farthest pixel in the object region is relatively small.

[0159] In Figure 5aIn this case, because the electronic device 100 corrects the existing object region 510 with a large viewing angle to a planar image 520, the degree of increase or decrease in the spacing between pixels in the object region 510 as a result of the distortion change (e.g., the degree to which multiple regions in the object region 510 are magnified or reduced, proportional to the length of H1) is relatively large.

[0160] On the other hand, Figure 5b In this case, since the electronic device 100 corrects the existing object region 530 with a small viewing angle to a planar image 540, the degree to which the spacing between pixels in the object region 530 increases or decreases as a result of the distortion change (e.g., the degree to which multiple regions in the object region 530 are magnified or reduced, proportional to the length of H2) is relatively small.

[0161] Figure 6 It shows that Figure 3d The diagram shows an example of a distorted object region 351, including object 301 identified as a ground object. (See also...) Figure 6 Compared to the reference line 610 which includes pixels corresponding to reference distances, the pixels of the display object 301 correspond to pixels with relatively long distances. Therefore, the size of the object 301 in the object region 351 is usually enlarged by distortion.

[0162] Reference Figure 6 Before distortion, the first interval between pixels 601 and 602 in object region 351 is equal to the second interval between pixels 602 and 603. Pixels 601, 602, and 603 in object region 351 correspond to pixels 601', 602', and 603' in the distorted object region 351', respectively.

[0163] exist Figure 6 In the middle, referring to the distorted object region 351', the first interval and the second interval are increased respectively by distortion.

[0164] Furthermore, due to the large difference in distance between pixels and the reference line, the increase in the spacing between pixels and the reference line increases the degree of increase; therefore, the reference... Figure 6 In the distorted object region 351', the second interval between pixels 602' and 603' is greater than the first interval between pixels 601' and 602'.

[0165] When the ground object is relatively far from the electronic device 100 (e.g. Figure 3a In the case of a toilet pad 301, there is a problem in the related art where the recognition rate of ground objects is low because the ground objects are low in height. On the other hand, in the electronic device 100 according to the embodiment, the recognition rate of ground objects can be improved by performing object recognition using the above-described twisting method.

[0166] The artificial intelligence model 135 can be trained to output information about an object included in the object region (e.g., a name or a type of the object such as a potty pad, a book, a flower pot, etc.). The artificial intelligence model 135 can also output a reliability value of the output information (e.g., a probability that a potty pad exists on the floor).

[0167] In this case, the processor 140 can input the warped object region into the artificial intelligence model 135 to acquire information about an object included in the warped object region and a reliability value of the information about the object.

[0168] If the acquired reliability value is less than a threshold value, the processor 140 can input the unwarped object region into the artificial intelligence model 135 to identify the detected object.

[0169] That is, the processor 140 can identify the floor object using a result obtained by inputting, into the artificial intelligence model 135, an object region having higher information reliability among the warped object region and the unwarped object region, which is output from the artificial intelligence model 135.

[0170] In an embodiment, when an object detected based on the sensing data is identified as a floor object, the processor 140 can also perform object identification using a separate artificial intelligence model.

[0171] Figure 7a is a block diagram illustrating a configuration of an electronic device using an artificial intelligence model trained to identify a floor object according to an embodiment.

[0172] Referring to Figure 7a , a plurality of artificial intelligence models 135-1 and 135-2 trained to identify an object can be stored in the memory 130.

[0173] Specifically, the first artificial intelligence model 135-1 can be a model trained to identify an object based on an image including the object other than a floor object, and the second artificial intelligence model 135-2 can be an artificial intelligence model trained to identify a floor object based on a plurality of images including a floor object.

[0174] The second artificial intelligence model 135-2 can be an artificial intelligence model trained to identify a floor object based on a plurality of images obtained by photographing a floor object at a relatively low angle.

[0175] Figure 7b is a block diagram illustrating an example of a functional configuration of the electronic device of Figure 7a Referring to Figure 7bThe object detection module 210, the object region detection module 220, and the object recognition module 240 can perform operations under the control of the processor 140.

[0176] First, the object detection module 210 can detect an object based on the sensing data.

[0177] Further, the object region detection module 220 can identify an object region including the detected object from an image acquired through the camera 120.

[0178] The object recognition module 240 can include a general object recognition module 710 for recognizing an object that is not a ground object and a ground object recognition module 720 for recognizing a ground object.

[0179] If it is identified that the detected object is not a ground object, the object recognition module 240 can recognize the object in the object region by using the general object recognition module 710. The processor 140 can load only the first artificial intelligence model 135-1 among the first and second artificial intelligence models 135-1 and 135-2 onto the volatile memory of the processor 140 and input the object region into the loaded first artificial intelligence model 135-1.

[0180] On the other hand, if it is identified that the detected object is recognized as a ground object, the object recognition module 240 can recognize the object in the object region by using the ground object recognition module 720. The processor 140 can load only the second artificial intelligence model 135-2 among the first and second artificial intelligence models 135-1 and 135-2 onto the volatile memory of the processor 140 and input the object region into the loaded second artificial intelligence model 135-2.

[0181] Figure 8 FIG. 3 is a diagram illustrating an example in which an electronic device according to an embodiment uses artificial intelligence models according to whether an object is identified as a ground object.

[0182] Referring to FIG. 3, Figure 8 The processor 140 can differently process the object regions 351, 352, and 353 according to whether a ground object is included therein. Figure 3c

[0183] Specifically, the processor 140 can perform object recognition on the object region 351 including the ground object 301 by using the ground object recognition module 720, and can perform object recognition on the object regions 352 and 353 including the objects 302 and 303, respectively, that are not ground objects by using the general object recognition module 710.

[0184] ​The electronic device 100 can also identify the ground object using a plurality of artificial intelligence models trained to recognize a ground object for each distance at which the ground object is photographed.

[0185] Figure 9a is a block diagram illustrating an example in which the electronic device according to an embodiment uses a plurality of artificial intelligence models trained to recognize a ground object for each distance at which the ground object is photographed.

[0186] Referring to Figure 9a , the second artificial intelligence model 135-2 stored in the memory 130 can include a plurality of artificial intelligence models, such as model a 910, model b 920, and model c 930. The models 910, 920, 930 can be models trained based on ground objects photographed at different distances.

[0187] Figure 9b is a block diagram illustrating an example of a functional configuration of the electronic device of Figure 9a .

[0188] Referring to Figure 9b , the object detection module 210, the object region detection module 220, the warping module 230, and the object recognition module 240 including the general object recognition module 710 and the ground object recognition module 720 can perform operations under the control of the processor 140.

[0189] Specifically, the processor 140 can detect an object by using the object detection module 210. Furthermore, the processor 140 can extract an object region including the detected object from an image acquired through the camera 120 by using the object region detection module 220.

[0190] If the detected object is recognized as not a ground object, the processor 140 can recognize an object in the object region by using the general object recognition module 710. The general object recognition module 710 can recognize the object using the first artificial intelligence model 135-1.

[0191] On the other hand, if the detected object is recognized as a ground object, the processor 140 can warp the object region by using the warping module 230. Furthermore, the processor 140 can input the warped object region into the ground object recognition module 720.

[0192] The ground object recognition module 720 can include modules for recognizing ground objects at different distances, such as a first distance module 721 and a second distance module 722. Specifically, the first distance module 721 can be a module for recognizing a ground object photographed at a first distance, and the second distance module 722 can be a module for recognizing a ground object photographed at a second distance different from the first distance.

[0193] The first distance module 721 can use the model a 910 to recognize the object, and the second distance module 722 can use the model b 920 to recognize the object.

[0194] The processor 140 can recognize the ground object in the object region by selectively using a module for recognizing a ground object corresponding to distance information of the object region before being warped among modules for recognizing ground objects at different distances included in the ground object recognition module 720.

[0195] The distance information of the object region before being warped can be an average of distances (e.g., distances between objects represented by each of a plurality of pixels and the electronic device and / or the camera) of a plurality of pixels included in the object region, but is not limited thereto.

[0196] Figure 10 A training process of a plurality of artificial intelligence models stored in the electronic device of FIG. 1 is illustrated. Figure 9a

[0197] Referring to FIG. 10, Figure 10 The model 910 can be a model trained to recognize a ground object based on an image 1010 of a ground object (e.g., a potty mat) photographed at a relatively far first distance.

[0198] As a result, the model a 910 can more effectively recognize a ground object at the first distance.

[0199] The model b 920 can be a model trained to recognize a ground object based on an image 1020 of a ground object (e.g., a potty mat) photographed at a second distance closer than the first distance.

[0200] In addition, the model c 930 can be a model trained to recognize a ground object based on an image 1030 of a ground object (e.g., a potty mat) photographed at a third distance closer than the second distance.

[0201] The images 1010, 1020, and 1030 can be images obtained by photographing a ground object at different angles. The angles can be determined according to a ground reference height of the camera 120 in the electronic device 100.

[0202] For example, when the electronic device 100 is a robot cleaner and the height of the camera 120 from the ground is 5 cm, the image 1010 can be an image obtained by photographing a ground object at a first distance at a height of 5 cm, the image 1020 can be an image obtained by photographing a ground object at a second distance at a height of 5 cm, and the image 1010 can be an image obtained by photographing a ground object at a third distance at a height of 5 cm. ​

[0203] Figure 11 is a diagram illustrating an example in which the electronic device according to an embodiment selectively uses a model trained as Figure 10 indicated.

[0204] Referring to Figure 11 , the processor 140 can warp the object region 351 including the ground object using the warping module 230, and can identify the object in the warped object region 351 by using the ground object identification module 720.

[0205] In this case, if the object region 351 before warping corresponds to the first distance, the processor 140 can identify the object in the object region 351 by using the first distance module 721 of the ground object identification module 720.

[0206] The processor 140 can selectively load a model 910 stored in the memory 130 among a plurality of artificial intelligence models onto the volatile memory of the processor 140, the model 910 can be a model trained to identify a ground object based on an image of a ground object photographed at a first distance.

[0207] In the case where the object regions 352 and 353 do not include the ground object, object identification can be performed by using the general object identification module 710 without performing warping.

[0208] Figure 12 is a block diagram illustrating a detailed configuration of the electronic device according to an embodiment.

[0209] Referring to Figure 12 , the electronic device 100 can include a communicator 150, a user input interface 160, a driver 170, etc., in addition to the sensor 110, the camera 120, the memory 130, and the processor 140.

[0210] The sensor 110 can include a LiDAR sensor 111, an ultrasonic sensor 112, a 3d sensor 113, an acceleration sensor 114, a geomagnetic sensor 115, etc.

[0211] The 3d sensor 133 can be implemented as a depth camera. The depth camera can be implemented as a time-of-flight (TOF) camera including a TOF sensor and infrared light. The depth camera can measure depth using parallax of images acquired with a plurality of cameras, and in this case, can include an IR stereo sensor. Alternatively, the depth camera can be implemented as a structured light method for measuring depth by photographing a light pattern projected by a projector having a camera.

[0212] The processor 140 can detect surrounding objects using a LiDAR sensor 111, an ultrasonic sensor 112, and a 3D sensor 113. Furthermore, the processor 140 can detect the direction, speed, and position of movement of the electronic device 100 using an accelerometer 114 and a geomagnetic sensor 115.

[0213] In addition to the sensor shown in Figure 9, sensor 110 may also include various other sensors.

[0214] Camera 120 may include an RGB camera.

[0215] Camera 120 may include, but is not limited to, a sensor, such as a charge-coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS). When camera 120 includes a CCD, the CCD may be implemented as a red / green / blue (RGB) CCD, an infrared (IR) CCD, etc.

[0216] For reference Figure 2a and Figure 7a As described, one or more artificial intelligence models (e.g., artificial intelligence models 135, 135-1, and 135-2) can be stored in memory 130. Specifically, one or more artificial intelligence models can be stored in storage space such as a hard disk or SSD in memory 130.

[0217] The functions of the stored artificial intelligence model can be executed based on the operation of the processor 140 and the memory 130.

[0218] For this purpose, processor 140 may include one or more processors. Here, one or more processors may be general-purpose processors such as CPUs and APs, graphics-specific processors such as GPUs and VPUs, or artificial intelligence-specific processors such as NPUs.

[0219] One or more processors execute control to process input data according to predetermined operating rules or artificial intelligence models stored in memory 130. The predefined operating rules or artificial intelligence models are characterized by being created through training.

[0220] Here, the predetermined operating rules or artificial intelligence model created through training refers to a predetermined operating rule or artificial intelligence model that has desired characteristics created by applying a training algorithm to a large amount of training data. This training can be performed within the apparatus itself, in which artificial intelligence according to the embodiment is executed, or it can be performed via a separate server / system.

[0221] An artificial intelligence model can include a plurality of neural network layers. Each layer has a plurality of weight values, and performs layer computation by calculating a result of computation of a previous layer and the plurality of weight values. Examples of a neural network include a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), and a deep Q-network, and the neural network in the present disclosure is not limited to the above examples unless otherwise specified.

[0222] A learning algorithm can be used to train a predetermined target device (e.g., a robot) using a large amount of learning data, so that the predetermined target device can make a decision or predict itself. Examples of a learning algorithm include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, and the learning algorithm in the present disclosure is not limited to the above examples unless otherwise specified.

[0223] The processor 140 can perform various operations by running / controlling the object detection module 210, the object region detection module 220, the warping module 230, the object recognition module 240, and the driving control module 250. Each module can be stored on a ROM of the memory 130 and / or implemented in a circuit form.

[0224] The communicator 150 is a component of the electronic device 100 for communicating with at least one external device to exchange signals / data. To this end, the communicator 150 can include a circuit.

[0225] The communicator 150 can include a wireless communication module, a wired communication module, etc.

[0226] In order to receive data (e.g., content) from an external server or an external device, the wireless communication module can include at least one of a Wi-Fi communication module, a Bluetooth module, an infrared data association (IrDA) module, a third generation (3G) mobile communication module, a fourth generation (4G) mobile communication module, or a fourth generation long term evolution (LTE) communication module.

[0227] The wired communication module can be implemented as a wired port such as a thunderbolt port or a universal serial bus (USB) port.

[0228] The processor 140 can transmit an object recognition result obtained by using the object recognition module 240 to an external device such as a server device via the communicator 150.

[0229] In addition, at least some of the artificial intelligence models stored in the memory 130 can be received by the electronic device 100 from an external device (e.g., a server device) through the communicator 150.

[0230] According to a user input received through the user input interface 160, the processor 140 can control the driver 170 to move the electronic device 100, and can perform object recognition while the electronic device 100 moves.

[0231] The user input interface 160 can include one or more buttons, a keyboard, a mouse, etc. Furthermore, the user input interface 160 can include a touch panel implemented with a display (not shown) or a separate touch pad (not shown).

[0232] The user input interface 160 can further include a microphone to receive a user's command or information by voice, and can be implemented together with the camera 120 to recognize a user's command or information in the form of motion.

[0233] The driver 170 is a component for moving the electronic device 100. The driver 170 can include a moving means implemented with, for example, one or more wheels, an actuator for driving the moving means, etc.

[0234] The processor 140 can control the driver 170 through a driving control module 250. The driving control module 250 can identify a moving speed, a moving direction, a position of the electronic device 100 based on sensing data of the acceleration sensor 114 and the geomagnetic sensor 115 included in the sensor 110, and can control the driver 170 based on the identified moving speed, moving direction, and position of the electronic device 100.

[0235] The driving control module 250 can control the driver 170 according to whether the detected object is recognized as a floor object and according to the object recognition result.

[0236] If the detected object is recognized as not a floor object that the electronic device 100 can go over or climb over by using the driver 170, the driving control module 250 can control the driver 170 to move while avoiding the detected object.

[0237] If the detected object is recognized as a floor object, the driving control module 250 can differently control the driver 170 according to the object recognition result.

[0238] Specifically, if the detected floor object is a carpet or a threshold, the driving control module 250 can control the driver 170 to move while climbing over or going over the detected object. On the other hand, if the detected floor object is a potty mat or a wire, the driving control module 250 can control the driver 170 to move while avoiding the detected object.

[0239] Figure 13a is a block diagram illustrating a configuration of an electronic device including a plurality of processors according to an embodiment.

[0240] Referring toFigure 13a The electronic device 100 can include a first processor 140-1 and a second processor 140-2.

[0241] The first processor 140-1 can be implemented as a main processor for controlling overall operations of the electronic device 100 by being connected to the sensor 110, the camera 120, etc. The first processor 140-1 can be implemented as a general-purpose processor such as a CPU or an AP or a graphics-dedicated processor such as a GUP or a VPU.

[0242] The second processor 140-2 can be implemented as a separate processor for performing object recognition by an artificial intelligence model stored in the memory 130. The second processor 140-2 can be implemented as an NPU that facilitates learning and / or computing of an artificial intelligence model.

[0243] The second processor 140-2 can load at least one artificial intelligence model onto the volatile memory 145-2 when recognizing an object, and input an object region into the loaded artificial intelligence model.

[0244] When the first artificial intelligence model 135-1 and the second artificial intelligence model 135-2 described above are included in the memory 130, the second processor 140-2 can receive information about whether a detected object is a floor object from the first processor 140-1. Also, when the detected object is recognized as not being a floor object, the second processor 140-2 can load the first artificial intelligence model 135-1 onto the volatile memory 145-2, and when the detected object is recognized as a floor object, can load the second artificial intelligence model 135-2 onto the volatile memory 145-2.

[0245] Figure 13b is a block diagram illustrating an example of a more detailed configuration of the electronic device of Figure 13a

[0246] Referring to Figure 13b , the object detection module 210, the object region detection module 220, the warping module 230, and the driving control module 250 are executed / controlled by the first processor 140-1, and the object recognition module 240 can be executed by the second processor 140-2.

[0247] In an embodiment, as a result that the object recognition module 240 can be executed by the first processor 140-1, the second processor 140-2 can perform object recognition under the control of the first processor 140-1, and thus, the configuration of the electronic device 100 is not limited to Figure 13b

[0248] Also, unlike Figure 13b ​​Differently, when the object detection module 210 uses at least one artificial intelligence model trained to detect an object based on the sensing data, the object detection module 210 can be operated by the second processor 140-2 or by the first and second processors 140-1 and 140-2.

[0249] Hereinafter, a control method of an electronic device according to an embodiment will be described with reference to Figure 14 , Figure 15 , Figure 16 and Figure 17 a control method of an electronic device according to an embodiment.

[0250] Figure 14 is a flowchart illustrating a control method of an electronic device including a memory storing an artificial intelligence model trained to recognize an object according to an embodiment.

[0251] Referring to Figure 14 , in the control method of the electronic device according to an embodiment, an object can be detected based on sensing data received from a sensor (S1410).

[0252] Further, it can be identified whether the detected object is a ground object (S1420). The ground object can be an object placed on a ground on which the electronic device is located, and can refer to an object having a height from the ground less than a predetermined threshold.

[0253] Specifically, a height of the object detected from the ground can be identified, and when the identified height is less than the predetermined threshold, the detected object can be identified as the ground object.

[0254] Further, if the detected object is identified as the ground object, at least a portion of an image acquired via a camera can be warped (S1430). Specifically, an object region including the detected object in the image can be warped based on distance information of the object region.

[0255] By warping, a position of a pixel can be transformed using a position transformation function such that distance information of a plurality of pixels in the object region corresponds to a reference distance. As a result, in a plurality of regions included in the object region, a region farther than the reference distance can be enlarged, and a region closer than the reference distance can be reduced.

[0256] The distance information refers to a distance between a specific portion of the object represented by each of a plurality of pixels in the object region and the electronic device or the camera.

[0257] Specifically, in the control method of the electronic device according to an embodiment, distance information of each of a plurality of pixels constituting an object region can be acquired, and a position of the plurality of pixels can be transformed based on the acquired distance information.

[0258] A reference pixel corresponding to the reference distance can be identified among the plurality of pixels included in the object region based on the acquired distance information, and a first pixel having a distance farther than the reference distance and a second pixel having a distance closer than the reference distance among the plurality of pixels can be identified.

[0259] Further, the position of the first pixel can be transformed such that the distance between the first pixel and the reference pixel increases, and the position of the second pixel can be transformed such that the distance between the second pixel and the reference pixel decreases.

[0260] In this case, the position of the first pixel can be transformed such that the degree of increase in the interval between the reference pixel and the first pixel after the transformation becomes larger as the difference between the reference distance and the distance of the first pixel increases. Further, the position of the second pixel can be transformed such that the degree of decrease in the interval between the reference pixel and the second pixel after the transformation becomes larger as the difference between the reference distance and the distance of the second pixel increases.

[0261] The distance of the region in the acquired image at which the focus of the camera is located can be set as the reference distance. Alternatively, the reference distance can be set within a distance range in which the artificial intelligence model can identify the object.

[0262] Alternatively, at least one pixel corresponding to the closest distance among the plurality of pixels in the object region can be identified as the reference pixel. That is, the distance of the corresponding pixel can be set as the reference distance.

[0263] Here, the first region can be scaled up such that the degree to which the first region is scaled up becomes larger as the difference between the distance of the first region and the reference distance increases, and the second region can be scaled down such that the degree to which the second region is scaled down becomes larger as the difference between the distance of the second region and the reference distance increases.

[0264] If the detected object is identified as not being a ground object, the warping in S1430 can not be performed.

[0265] Further, in the control method of the electronic device according to the embodiment, the detected object can be identified by inputting the object region into the artificial intelligence model (S1440). Specifically, if the detected object is identified as a ground object, the object region warped according to S1430 is input into the artificial intelligence model, and if the detected object is identified as not being a ground object, the object region which has not undergone S1430 can be input into the artificial intelligence model.

[0266] In the control method of the electronic device according to the embodiment, the distorted object region can be input into the artificial intelligence model to acquire information about an object included in the distorted object region and a reliability value of the information about the object.

[0267] In the embodiment, if the acquired reliability value is less than a threshold value, the detected object can be recognized by inputting an object region of the image including the detected object into the artificial intelligence model. That is, the object can be recognized by inputting the object region without distortion into the artificial intelligence model.

[0268] Figure 15 is an algorithm illustrating an example of distorting an object region and recognizing an object according to whether a ground object is detected in the control method according to the embodiment.

[0269] Referring to Figure 15 In the control method of the electronic device according to the embodiment, an object can be detected based on the sensing data (S1510). Further, if the object is detected, an image acquired by photographing a direction in which the detected object is located can be acquired (S1520).

[0270] In the control method of the electronic device according to the embodiment, a height of the object detected based on the sensing data can be recognized (S1530). If the recognized height is less than a threshold value (Yes in S1540), the corresponding object can be recognized as a ground object.

[0271] Further, an object region of the previously acquired image including the object can be recognized (S1550), and the recognized object region can be distorted according to distance information (S1560).

[0272] Further, the object can be recognized by inputting the distorted object region into the artificial intelligence model (S1570).

[0273] On the other hand, if the recognized height is the threshold value or more (No in S1540), the corresponding object can be recognized as not a ground object. In this case, an object region can be recognized in the image (S1580), and the object can be recognized by inputting the recognized object region into the artificial intelligence model (S1570). In this case, preprocessing required before inputting the object region into the artificial intelligence model, such as scaling the entire size of the object region, can be performed, but recognizing the object region in S1560 and distorting the object region according to the distance information in S1560 can be skipped.

[0274] The memory can further store a separate artificial intelligence model trained to recognize a ground object based on a plurality of images including the ground object. In this case, in the control method of the electronic device according to the embodiment, if the detected object is recognized as a ground object, the detected object can be recognized by inputting an object region of the image including the detected object into the above-described separate artificial intelligence model.

[0275] Figure 16 is an algorithm illustrating an example of using an artificial intelligence model according to whether a ground object is detected in the control method according to the embodiment.

[0276] Referring to Figure 16 In the control method of the electronic device according to the embodiment, an object can be detected based on the sensing data (S1610). Further, if the object is detected, an image of the detected object can be acquired (S1620).

[0277] In the control method of the electronic device according to the embodiment, a height of the object detected based on the sensing data can be recognized (S1630). If the recognized height is recognized as a threshold value or more (No in S1640), an object region in the image can be recognized (S1650), and the object can be recognized using the recognized object region and a general object recognition module (S1660). The general object recognition module can input the object region into an artificial intelligence model trained to recognize an object that is not a ground object.

[0278] On the other hand, if the recognized height is less than the threshold value (Yes in S1640), an object region can be recognized (S1670), and the object can be recognized using the recognized object region and a ground object recognition module (S1680). The ground object recognition module can input the object region into an artificial intelligence model trained to recognize a ground object.

[0279] In this case, the artificial intelligence model trained to recognize a ground object can be classified into a plurality of artificial intelligence models. The plurality of artificial intelligence models can be artificial intelligence models trained based on images of a ground object taken at different distances.

[0280] In the control method of the electronic device according to the embodiment, the object region can be input into an artificial intelligence model trained based on an image corresponding to distance information of the object region among the plurality of artificial intelligence models.

[0281] In the case where the electronic device includes a driver, in the control method of the electronic device according to the embodiment, the driver of the electronic device can be controlled according to whether the detected object is a ground object and according to the object recognition result.

[0282] Figure 17 is a flowchart illustrating an example of reflecting whether a floor object is detected and an object recognition result to drive in a control method according to an embodiment.

[0283] Referring to Figure 17 In the control method of the electronic device according to an embodiment, the electronic device can be controlled to start driving (S1710). Also, an object around the electronic device can be detected using sensing data received through a sensor at the time of driving (S1720). Specifically, various obstacles located in a driving direction of the electronic device can be detected.

[0284] If the object is detected, it can be identified whether the detected object is a floor object (S1730). Specifically, it can be determined whether the detected object is a floor object according to whether a height of the detected object is less than a threshold value.

[0285] Also, if the detected object is determined to be a floor object (Yes in S1730), an object region of an image acquired via a camera including the object can be identified (S1740), and the identified object region can be warped according to distance information (S1750).

[0286] Also, the object can be identified through the warped object region (S1760).

[0287] If the identified object is a predetermined object, such as a carpet or a threshold (Yes in S1770), the electronic device can be controlled to drive on or climb over the corresponding object (S1780). On the other hand, if the identified object is not a predetermined object (No in S1770), the electronic device can be controlled to drive while avoiding the corresponding object (S1790).

[0288] If the detected object is identified as not being a floor object (No in S1730), the electronic device can be controlled to drive while avoiding the detected object (S1790).

[0289] Referring to Figures 14 to 17 The described control method can be implemented by referring to Figure 2a , Figure 7a , Figure 9a , Figure 12 and Figure 13a the electronic device 100 illustrated and described.

[0290] Alternatively, the control method described can also be implemented by a system including the electronic device 100 and one or more external devices. Figures 14 to 17

[0291] The above-described embodiments can be implemented in a computer readable medium, in software, hardware, and / or firmware.​

[0292] The above-described embodiments can be implemented in a recording medium readable by a computer or a similar device using software, hardware, and / or a combination thereof.

[0293] According to a hardware implementation, the embodiments described in the present disclosure can be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, or electronic units for performing other functions.

[0294] In some cases, the embodiments described in the specification can be implemented by a processor itself. According to a software implementation, the embodiments described in the present disclosure, such as processes and functions, can be implemented as separate software modules. Each of the above-described software modules can perform one or more functions and operations described in the specification.

[0295] Computer instructions for performing the processing operations of the electronic device 100 according to the various embodiments described above can be stored in a non-transitory computer readable medium. The computer instructions stored in the non-transitory computer readable medium, when executed by a processor of a specific device, allow the specific device to perform the processing operations of the electronic device 100 according to the various embodiments described above.

[0296] The non-transitory computer readable medium refers to a medium that stores data semi-permanently and is read by a device, rather than a medium that stores data for a short time, such as a register, a cache, a memory, etc. Examples of the non-transitory computer readable medium can include a compact disc (CD), a digital versatile disc (DVD), a hard disk, a Blu-ray disc, a universal serial bus (USB), a memory card, a read-only memory (ROM), etc.

[0297] Although several embodiments have been shown and described, it will be understood by those skilled in the art that changes can be made in the embodiments without departing from the principles and spirit of the present disclosure, and the scope of the present disclosure is defined in the claims and their equivalents.

Claims

1.An electronic device comprising: a sensor; a camera; a memory; and a processor configured to be connected to the sensor, the camera, and the memory, wherein the memory includes an artificial intelligence model trained to recognize at least one object, and wherein the processor is further configured to: detect an object based on sensing data received from the sensor; based on recognizing that the detected object has a height less than a predetermined threshold, warp an object region including the detected object in an image acquired via the camera based on distance information of the object region; and recognize the detected object by inputting the warped object region into the artificial intelligence model, wherein the processor is further configured to, based on recognizing that the detected object has a height equal to or greater than the predetermined threshold, recognize the detected object by inputting the object region without warping into the artificial intelligence model. 2.The electronic device of claim 1, wherein the processor is further configured to perform warping of the object region based on also determining that the detected object is located on a plane in which the electronic device is located, and wherein wherein the height of the detected object is recognized with respect to the plane in which the electronic device is located. The predetermined threshold is predetermined based on a maximum height that the electronic device is able to go over or climb when the electronic device is driven to move. 3.The electronic device of claim 2, wherein, In warping the object region, the processor is further configured to acquire distance information of each of a plurality of pixels constituting the object region, and transform positions of the plurality of pixels based on the acquired distance information. 4.The electronic device of claim 1, wherein, In transforming the positions of the plurality of pixels, the processor is further configured to: 5.The electronic device of claim 4, wherein, identify a reference pixel corresponding to a reference distance among the plurality of pixels based on the acquired distance information; transform a position of a first pixel whose distance is greater than the reference distance such that a gap between the first pixel and the reference pixel increases; and transform a position of a second pixel whose distance is less than the reference distance such that a gap between the second pixel and the reference pixel decreases. In transforming the positions of the plurality of pixels, the processor is further configured to: 6.The electronic device of claim 5, wherein, transform the position of the first pixel such that the degree of increase in the gap between the first pixel and the reference pixel becomes greater as a difference between the reference distance and the distance of the first pixel increases, and transform the position of the second pixel such that the degree of decrease in the gap between the second pixel and the reference pixel becomes greater as a difference between the reference distance and the distance of the second pixel increases. In transforming the positions of the plurality of pixels, the processor is further configured to identify at least one pixel corresponding to a nearest distance among the plurality of pixels as the reference pixel. 7.The electronic device of claim 5, wherein, The processor is further configured to set a distance of a region in which a focus of the camera is located in the acquired image as the reference distance. 8.The electronic device of claim 5, wherein, The reference distance is predetermined within a distance range in which the artificial intelligence model recognizes the object. 9.The electronic device of claim 5, wherein, The processor is further configured to: 10.The electronic device of claim 1, wherein, ​ acquiring information about an object included in the distorted object region and a reliability value of the information about the object by inputting the distorted object region into the artificial intelligence model, and identifying the detected object by inputting the object region without distortion into the artificial intelligence model based on the acquired reliability value being less than a threshold value. 11.An electronic device comprising: a sensor; a camera; a memory; and a processor configured to be connected to the sensor, the camera, and the memory, wherein the memory includes a plurality of artificial intelligence models trained to recognize at least one object, wherein the plurality of artificial intelligence models are trained based on images taken at different distances of the at least one object, and wherein the processor is further configured to: detect an object based on sensing data received from the sensor; distort an object region including the detected object in an image acquired via the camera based on distance information of the object region based on the detected object having a height less than a predetermined threshold value; identify an artificial intelligence model trained based on an image corresponding to the distance information of the object region among the plurality of artificial intelligence models; and identify the detected object by inputting the distorted object region into the identified artificial intelligence model, wherein the processor is further configured to identify the detected object by inputting the object region without distortion into the artificial intelligence model based on a determination that the detected object has a height equal to or greater than the predetermined threshold value. 12.A method of controlling an electronic device including a memory in which an artificial intelligence model trained to recognize an object is stored, the method comprising: detecting an object based on sensing data received from a sensor; identifying whether the detected object can be stepped over or climbed over by the electronic device when the electronic device is driven to move; distorting an object region including the detected object in an image acquired via a camera based on distance information of the object region based on an identification that the detected object can be stepped over or climbed over by the electronic device; identifying the detected object by inputting the distorted object region into the artificial intelligence model; and identifying the detected object by inputting the object region without distortion into the artificial intelligence model based on an identification that the detected object has a height equal to or greater than a predetermined threshold value. 13.The method of claim 12, wherein distorting the object region is further performed based on a determination that the detected object is located on a plane, and wherein the detected object is identified as being able to be stepped over or climbed over based on a height of the detected object from the plane in which the electronic device is located. wherein, The distorting includes acquiring distance information of each of a plurality of pixels constituting the object region, and transforming positions of the plurality of pixels based on the acquired distance information. ​ 14. The method of claim 12, wherein, ​

Images