Electronic device and control method therefor

Abstract

Disclosed is an electronic device. The electronic device comprises: a LiDAR sensor; and at least one processor that (a) controls the LiDAR sensor such that the LiDAR sensor acquires first sensing data by sensing a space in a sensing direction of the LiDAR sensor corresponding to a preset angle on the basis of the occurrence of a preset event, (b) controls the LiDAR sensor to sense the space in the sensing direction of the LiDAR sensor corresponding to the preset angle, and then controls the direction of the LiDAR sensor so that the sensing direction of the LiDAR sensor is sequentially changed such that the LiDAR sensor acquires a plurality of pieces of second sensing data respectively corresponding to the sequentially changed sensing directions, and (c) acquires a depth map including distance information of the space on the basis of the first sensing data and the plurality of pieces of second sensing data.

Description

Electronic device and method of controlling the same

[0001] The present disclosure relates to an electronic device and a control method thereof, and more particularly, to an electronic device that provides a depth map including distance information of a space and a control method thereof.

[0002] Advances in electronic technology have led to the development and proliferation of various types of electronic devices. In particular, projectors, used in a variety of settings, including homes, offices, and public spaces, have seen continuous advancements in recent years.

[0003] Recently, portable projectors have become readily available for use in a variety of locations. These projectors utilize sensors to identify the surrounding environment and obstacles in the user's desired projection space, enabling them to find an appropriate projection surface.

[0004] The present disclosure is designed to improve the above-described problem, and an object of the present disclosure is to provide an electronic device and a control method thereof for obtaining a depth map based on a plurality of sensing data sensed by a lidar sensor at a plurality of angles.

[0005] According to an embodiment of the present disclosure, an electronic device includes a lidar sensor and at least one processor, wherein the at least one processor controls the lidar sensor to sense a space in a sensing direction of the lidar sensor corresponding to a preset angle based on the occurrence of a preset event so that the lidar sensor acquires first sensing data, controls the lidar sensor to sense the space in the sensing direction of the lidar sensor corresponding to the preset angle, and then controls the direction of the lidar sensor so that the sensing direction of the lidar sensor is sequentially changed so that the lidar sensor acquires a plurality of second sensing data each corresponding to the sequentially changed sensing direction, and acquires a depth map including distance information of the space based on the first sensing data and the plurality of second sensing data.

[0006] A method for controlling an electronic device including a lidar sensor according to an embodiment of the present disclosure includes a step of controlling the lidar sensor to sense a space in a sensing direction of the lidar sensor corresponding to a preset angle based on the occurrence of a preset event so that the lidar sensor acquires first sensing data; a step of controlling the lidar sensor to sense the space in the sensing direction of the lidar sensor corresponding to the preset angle, and then controlling the direction of the lidar sensor so that the sensing direction of the lidar sensor is sequentially changed so that the lidar sensor acquires a plurality of second sensing data each corresponding to the sequentially changed sensing direction; and a step of acquiring a depth map including distance information of the space based on the first sensing data and the plurality of second sensing data.

[0007] In one embodiment of the present disclosure, a non-transitory computer-readable recording medium storing computer instructions that, when executed by a processor of an electronic device including a lidar sensor, cause the electronic device to perform an operation, the operation includes: a step of controlling the lidar sensor to sense a space in a sensing direction of the lidar sensor corresponding to a preset angle based on the occurrence of a preset event so that the lidar sensor acquires first sensing data; a step of controlling the lidar sensor to sense the space in the sensing direction of the lidar sensor corresponding to the preset angle, and then a step of controlling the direction of the lidar sensor so that the sensing direction of the lidar sensor is sequentially changed so that the lidar sensor acquires a plurality of second sensing data each corresponding to the sequentially changed sensing direction; and a step of acquiring a depth map including distance information of the space based on the first sensing data and the plurality of second sensing data.

[0008] The above and other aspects, features and advantages of specific embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings.

[0009] FIG. 1 is a block diagram showing the configuration of an electronic device according to at least one embodiment of the present disclosure.

[0010] FIG. 2 is a diagram for explaining an operation of obtaining a depth map based on a plurality of sensing data according to at least one embodiment of the present disclosure.

[0011] FIG. 3 is a diagram for explaining an operation of obtaining a depth map based on a plurality of sensing data according to at least one embodiment of the present disclosure.

[0012] FIG. 4 is a drawing for explaining the operation of an electronic device according to at least one embodiment of the present disclosure.

[0013] FIG. 5A is a diagram illustrating an operation of obtaining a depth map according to at least one embodiment of the present disclosure.

[0014] FIG. 5b is a diagram illustrating an operation of obtaining a depth map according to at least one embodiment of the present disclosure.

[0015] FIG. 6A is a diagram for explaining an operation of obtaining vanishing point information according to at least one embodiment of the present disclosure.

[0016] FIG. 6b is a diagram for explaining an operation of obtaining vanishing point information according to at least one embodiment of the present disclosure.

[0017] FIG. 7 is a diagram illustrating an operation of updating distance information according to at least one embodiment of the present disclosure.

[0018] FIG. 8A is a diagram illustrating an operation of updating distance information according to at least one embodiment of the present disclosure.

[0019] FIG. 8b is a diagram illustrating an operation of updating distance information according to at least one embodiment of the present disclosure.

[0020] FIG. 9 is a diagram illustrating an operation of updating distance information according to at least one embodiment of the present disclosure.

[0021] FIG. 10 is a diagram illustrating an operation of updating distance information according to at least one embodiment of the present disclosure.

[0022] FIG. 11 is a detailed block diagram illustrating a detailed configuration of an electronic device according to at least one embodiment of the present disclosure.

[0023] FIG. 12 is a drawing for explaining in-home map information in at least one embodiment of the present disclosure.

[0024] FIG. 13 is a drawing for explaining in-home map information in at least one embodiment of the present disclosure.

[0025] FIG. 14 is a flowchart illustrating a method for controlling an electronic device according to at least one embodiment of the present disclosure.

[0026] Hereinafter, the present disclosure will be described in detail with reference to the attached drawings.

[0027] The terms used in the embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of this disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant disclosure. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on the meanings of the terms and the overall content of this disclosure.

[0028] In this specification, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a feature (e.g., a number, function, operation, or component such as a part), and do not exclude the presence of additional features.

[0029] The expression "at least one of A and / or B" should be understood to mean either "A" or "B" or "A and B".

[0030] As used herein, the expressions “first,” “second,” “first,” or “second,” etc., may describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0031] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that the component may be directly coupled to the other component, or may be connected through another component (e.g., a third component).

[0032] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "consist of" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0033] In the present disclosure, a "module" or "part" performs at least one function or operation and may be implemented in hardware or software, or a combination of hardware and software. Furthermore, multiple "modules" or multiple "parts" may be integrated into at least one module and implemented as at least one processor, excluding any "modules" or "parts" that need to be implemented as specific hardware.

[0034] In this specification, the term user may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

[0035] An embodiment of the present disclosure will be described in more detail with reference to the attached drawings below.

[0036] FIG. 1 is a block diagram showing the configuration of an electronic device according to at least one embodiment of the present disclosure.

[0037] According to FIG. 1, an electronic device (100) may include a lidar sensor (110), a memory (120), and at least one processor (130).

[0038] An electronic device (100) according to various embodiments of the present disclosure may include, for example, at least one of a smartphone, a tablet PC, a desktop PC, a laptop PC, or a wearable device. The wearable device may include at least one of an accessory type (e.g., a watch, a ring, a bracelet, an anklet, a necklace, glasses, contact lenses, or a head-mounted device (HMD)), a fabric or clothing-integrated type (e.g., an electronic garment), a body-attached type (e.g., a skin pad or tattoo), or a bio-implantable circuit.

[0039] In some embodiments, the electronic device may be, for example, a television, a digital video disk (DVD) player, an audio, a refrigerator, an air conditioner, a vacuum cleaner, an oven, a microwave oven, a washing machine, an air purifier, a set-top box, a home automation control panel, a security control panel, a media box (e.g., Samsung HomeSync). TM , Apple TV TM , or Google TV TM ), game consoles (e.g. Xbox TM , PlayStation TM ), may include at least one of an electronic dictionary, an electronic key, a camcorder, or an electronic picture frame. Meanwhile, among the electronic devices described above, a device having a display may be referred to as a display device. Meanwhile, the electronic device of the present disclosure may be a set-top box or a PC that provides images to a display device, even if it does not have a display.

[0040] According to one embodiment of the present disclosure, the electronic device (100) can be implemented as a projector that projects images onto a wall, screen, or screen, or various types of devices with image projection capabilities. Hereinafter, the operation of the electronic device (100) will be described assuming that the electronic device (100) is implemented as a device with image projection capabilities.

[0041] When the electronic device (100) is implemented as a device having an image projection function, the electronic device (100) can sense the surrounding space for projecting the image to identify a projection surface for projecting the image and obtain a depth map (or may be referred to as a '3D ToF (Time-of-Flight) depth map', '3D ToF image') containing distance information.

[0042] According to one embodiment, the electronic device (100) can sense the surrounding space through a LiDAR sensor (110) located on one side of the electronic device (100) to obtain a depth map. According to one example, the LiDAR sensor (110) can be located at various locations of the electronic device (100). For example, the LiDAR sensor (110) can be located at the top of the electronic device (100). For example, when the electronic device (100) is implemented as a spherical (ball-shaped) mobile projector (or mobile robot), the LiDAR sensor can be located at the top of the head of the mobile projector.

[0043] The lidar sensor (110) can identify the distance between the lidar sensor and an object based on the phase difference between the light output from the light emitting unit of the lidar sensor (110) and the light received from the light receiving unit. For example, the light receiving unit can be implemented as a ToF (Time-of-Flight) sensor, and the following description will assume that the light receiving unit is implemented as a ToF sensor.

[0044] According to one embodiment of the present disclosure, a lidar sensor (110) may include a light-emitting unit, a ToF sensor, and a driving unit. Meanwhile, the lidar sensor (110) does not necessarily have to be implemented by including all of the above-described components, and some components may be omitted or added.

[0045] The light emitting unit outputs modulated light toward an object around the lidar sensor (110). At this time, the modulated light (hereinafter, output light) output from the light emitting unit may have a square wave waveform or a sinusoidal wave waveform.

[0046] According to one embodiment of the present disclosure, the light emitting unit can output light of various frequency bands. Specifically, the light emitting unit can sequentially output light of different frequency bands. Here, sequentially outputting light means that the light emitting unit can sequentially output light of 5 MHz at preset intervals, for example, the light emitting unit can sequentially output light of 100 MHz after outputting light of 5 MHz. Alternatively, the light emitting unit can output light of different frequency bands depending on the operation of the lidar sensor.

[0047] The ToF sensor acquires light reflected by an object (hereinafter, referred to as reflected light). Specifically, the ToF sensor receives reflected light that is emitted from a light-emitting unit, reflected by an object, and then returned toward the lidar sensor.

[0048] The ToF sensor may be implemented as an iToF (indirect ToF) sensor. The iToF sensor can obtain the phase difference by identifying the phase difference between the received reflected light and the output light output from the light emitter. The ToF sensor may also be implemented as another ToF sensor (e.g., a dToF (direct ToF) sensor). However, the present invention is not limited thereto, and may be implemented as various types of ToF sensors.

[0049] For example, when a ToF sensor is implemented as an iToF sensor, the ToF sensor is connected to a light emitting unit and can acquire phase information of the output light emitted from the light emitting unit. Based on the acquired phase information, the ToF sensor can identify the difference between the phase of the output light emitted from the light emitting unit and the phase of the received reflected light.

[0050] For example, the lidar sensor (110) may include a driving unit. Specifically, the driving unit is configured to rotate the lidar sensor (110). The driving unit may rotate the lidar sensor 360 degrees at a preset rotation speed. To this end, the driving unit may be implemented as a motor.

[0051] As the lidar sensor (110) rotates 360 degrees, the light emitting unit outputs light at preset time intervals to scan the surrounding environment 360 degrees, and the lidar sensor (110) can create a 3D point cloud based on the scan results.

[0052] As an example, an encoder connected to a driving unit can record a rotation angle at each point in time when light is output. Here, the encoder may be a device for detecting information such as the rotation speed, angle, and direction of the motor. The rotation angle recorded by the encoder may refer to an angle between the direction in which the light emitting unit outputs light and a reference direction of the lidar sensor (110). For example, the reference direction may be determined based on an angle corresponding to a direction when the encoder is turned on, an angle corresponding to a direction preset in the lidar sensor (110), etc.

[0053] As an example, a ToF sensor can sequentially receive reflected light and sequentially identify the phase difference between the reflected light and the output light.

[0054] According to one example, the electronic device (100) can obtain a plurality of rotation angles identified by the encoder and obtain a phase difference corresponding to each of the plurality of rotation angles from the ToF sensor.

[0055] The memory (120) may store various sensing data acquired from the lidar sensor (110) and at least one command regarding the electronic device (100). The memory (120) may store various intermediate data used in the process of the electronic device (100) acquiring a depth map including distance information.

[0056] The memory (120) may be implemented as an internal memory such as a ROM (e.g., an electrically erasable programmable read-only memory (EEPROM)) or RAM included in at least one processor (130), or may be implemented as a separate memory from at least one processor (130). In this case, the memory (120) may be implemented as a memory embedded in the electronic device (100) or as a memory detachable from the electronic device (100) depending on the purpose of data storage. For example, data for driving the electronic device (100) may be stored in a memory embedded in the electronic device (100), and data for expanding functions of the electronic device (100) may be stored in a memory detachable from the electronic device (100).

[0057] In the case of memory embedded in the electronic device (100), it may be implemented as at least one of volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (e.g., one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), etc.), hard drive, or solid state drive (SSD), and in the case of memory that can be detachably attached to the electronic device (100), it may be implemented in the form of a memory card (e.g., compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (Mini-SD), extreme digital (xD), multi-media card (MMC), etc.), external memory that can be connected to a USB port (e.g., USB memory), etc.

[0058] The memory (120) may store an O / S (Operating System) for driving the electronic device (100). The memory (120) may also store various software programs or applications for operating the electronic device (100) according to various embodiments of the present disclosure. In addition, the memory (120) may include a semiconductor memory such as a flash memory or a magnetic storage medium such as a hard disk.

[0059] Specifically, various software modules for operating the electronic device (100) according to various embodiments of the present disclosure may be stored in the memory (120), and at least one processor (130) may control the operation of the electronic device (100) by executing various software modules stored in the memory (120). That is, the memory (120) is accessed by at least one processor (130), and data reading / recording / modifying / deleting / updating, etc. may be performed by at least one processor (130).

[0060] In the present disclosure, the term memory (120) may be used to mean a storage unit, a ROM, a RAM within at least one processor (130), or a memory card (e.g., a micro SD card, a memory stick) mounted on an electronic device (100).

[0061] At least one processor (130) is connected to the memory (120) and the lidar sensor (110) to perform various functions or commands of the electronic device (100).

[0062] At least one processor (130) can perform operations of the electronic device (100) according to various embodiments by executing at least one instruction stored in the memory (120).

[0063] At least one processor (130) may include one or more of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), an Accelerated Processing Unit (APU), a Many Integrated Core (MIC), a Digital Signal Processor (DSP), a Neural Processing Unit (NPU), a hardware accelerator, or a machine learning accelerator. The at least one processor (130) may control one or any combination of other components of the electronic device, and may perform operations related to communication or data processing. The at least one processor (130) may execute one or more programs or instructions stored in a memory. For example, the at least one processor (130) may perform a method according to one or more embodiments of the present disclosure by executing one or more instructions stored in a memory.

[0064] When a method according to one or more embodiments of the present disclosure includes multiple operations, the multiple operations may be performed by one processor or by multiple processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor) and the third operation may be performed by the second processor (e.g., an artificial intelligence-specific processor).

[0065] At least one processor (130) may be implemented as a single core processor including one core, or may be implemented as one or more multicore processors including multiple cores (e.g., homogeneous multicores or heterogeneous multicores). When at least one processor (130) is implemented as a multicore processor, each of the multiple cores included in the multicore processor may include internal processor memory such as cache memory or on-chip memory, and a common cache shared by the multiple cores may be included in the multicore processor.

[0066] In embodiments of the present disclosure, a processor may mean a system on a chip (SoC) in which at least one processor (130) and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, a GPU, an APU, a MIC, a DSP, an NPU, a hardware accelerator, or a machine learning accelerator, but embodiments of the present disclosure are not limited thereto.

[0067] Meanwhile, the processor (130) may also perform various operations of the present disclosure using an artificial intelligence model. An artificial intelligence model is a computer system or software module that implements human-level intelligence. The artificial intelligence model has the characteristics of allowing the machine to learn and make judgments on its own, and its recognition rate improves with use.

[0068] Artificial intelligence models are composed of machine learning (deep learning) technology that uses algorithms that classify / learn the characteristics of input data on their own, and element technologies that use machine learning algorithms to simulate the cognitive and judgment functions of the human brain.

[0069] The element technologies may include, for example, at least one of a linguistic understanding technology that recognizes human language / characters, a visual understanding technology that recognizes objects as if they were human vision, an inference / prediction technology that judges information and logically infers and predicts, and a knowledge representation technology that processes human experience information into knowledge data.

[0070] The artificial intelligence model in the present disclosure can perform an operation of logically inferring and predicting a depth map including distance information of a space according to sensing data based on previously stored data by executing at least one processor (130), an operation of analyzing sensing data, etc. and identifying a depth map through probability-based inference based on the analysis results and usage history thereof, etc.

[0071] These various operations of the artificial intelligence model according to the present disclosure can be performed by the processor (130) and the memory (120).

[0072] The processor (130) may be comprised of one or more processors. As described above, the processor (130) may be implemented in various forms. In particular, one or more processors may be implemented as a dedicated artificial intelligence processor. The dedicated artificial intelligence processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0073] Meanwhile, the artificial intelligence model can be stored in memory (120). The artificial intelligence model can be created through learning.

[0074] Creating through learning means that a basic artificial intelligence model is learned by a learning algorithm using a plurality of learning data, thereby creating a predefined operation rule or artificial intelligence model set to perform a desired characteristic (or purpose). This learning may be performed in the electronic device (100) itself on which the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and / or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

[0075] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and performs neural network operations by calculating the results of previous layers and the multiple weights. The multiple weights of the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated during the learning process to reduce or minimize the loss or cost values ​​obtained by the artificial intelligence model.

[0076] Artificial neural networks may include deep neural networks (DNNs), such as, but not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), or deep Q-networks.

[0077] In various embodiments of the present disclosure, a depth map obtained based on spatial distance information may be obtained as a result of an inference and prediction process utilizing an artificial intelligence model. Inference and prediction are technologies for logically inferring and predicting by judging information, and include knowledge and probability-based inference, optimized prediction, preference-based planning, and recommendations. According to various embodiments of the present disclosure, at least one processor (130) (hereinafter, "processor") may obtain a depth map using an artificial intelligence model.

[0078] According to one embodiment, the processor (130) can obtain a depth map including distance information of a space by using a plurality of sensing data stored in the memory (120).

[0079] According to one embodiment, the processor (130) may control the lidar sensor (110) to sense space in a sensing direction of the lidar sensor (110) corresponding to a preset angle according to a preset event. Here, the angle may correspond to an angle formed with the ground. For example, the preset event may include an event in which the electronic device (100) is turned on.

[0080] For example, the preset event may correspond to an event that is activated to sense a space through the lidar sensor (110) after the electronic device (100) is turned on. That is, when the electronic device (100) is set to a power-saving state after being turned on, the processor (130) may not sense a space in the sensing direction of the lidar sensor (110) corresponding to the preset angle. On the other hand, when the electronic device (100) is not set to a power-saving state after being turned on but is activated to sense a space, the electronic device (100) may sense a space in the sensing direction of the lidar sensor (110) corresponding to the preset angle. The subsequent operations after the electronic device (100) senses a space in the sensing direction of the lidar sensor (110) corresponding to the preset angle according to the event that is activated to sense a space will be described in detail in the following part.

[0081] For example, when the electronic device (100) is activated to sense space, the lidar sensor (110) may sense space in a sensing direction of the lidar sensor (110) corresponding to an angle parallel to the ground. For example, when the lidar sensor (110) is positioned at the top of the electronic device (100), the lidar sensor (110) may emit light in a direction parallel to the ground. For example, when the electronic device (100) is activated to sense space, the processor (130) may control the lidar sensor (110) to start sensing at the same time as it is activated to sense space, instead of starting sensing after adjusting the sensing direction of the lidar sensor (110). Accordingly, the lidar sensor (110) may start sensing space from the sensing direction corresponding to an angle parallel to the ground.

[0082] For example, the processor (130) may acquire first sensing data acquired through the lidar sensor (110) according to a preset event and store the data in the memory (120). For example, the processor (130) may acquire first sensing data acquired by sensing space in a sensing direction of the lidar sensor (110) corresponding to an angle parallel to the ground, and store the first sensing data in the memory (120).

[0083] According to one embodiment, the processor (130) can store a plurality of second sensing data acquired by sequentially changing the sensing direction of the lidar sensor (110) after the space is sensed in a sensing direction corresponding to an angle parallel to the ground, in the memory (120).

[0084] For example, the processor (130) may control the lidar sensor (110) to sequentially change the sensing direction in an upward direction while the lidar sensor is facing parallel to the ground, thereby obtaining a plurality of second sensing data through the lidar sensor (110). However, the present invention is not limited thereto, and the lidar sensor (110) may also be controlled to sequentially change the sensing direction in a downward direction while the lidar sensor is facing parallel to the ground, thereby obtaining a plurality of second sensing data through the lidar sensor (110). For example, when the electronic device (100) is implemented as a mobile device, the electronic device (100) may be positioned on an object (e.g., furniture) that is a preset distance from the ground, rather than on the ground. In this case, the electronic device (100) may identify the distance from the ground using various sensors, and control the lidar sensor (110) to sequentially change the sensing direction in a downward direction while the lidar sensor is facing parallel to the ground according to the identified distance.

[0085] According to one embodiment, the processor (130) may obtain a depth map including distance information of a space based on the first sensing data and a plurality of second sensing data. The depth map may be information including information indicating the distance of each pixel to the lidar sensor (110). For example, the depth map may be in a form in which each pixel has a distance value. For example, the distance value may be expressed in grayscale, and the brightness of a specific pixel may indicate the distance between the pixel and the lidar sensor (110). However, the present invention is not limited thereto. The distance value may be expressed as at least one of a floating point, a digital distance code, and a number of bits. The depth map may be implemented in the form of a two-dimensional point cloud. Here, the point cloud is a set of points representing each point in a 3D space, and may be implemented in a form in which a plurality of points are arranged in rows. FIGS. 2 and 3 are diagrams for explaining an operation of obtaining a depth map based on a plurality of sensing data according to at least one embodiment of the present disclosure.

[0086] According to FIG. 2, a process (1000) is illustrated in which a processor (130) obtains a depth map (10) including distance information of a space based on first sensing data and second sensing data.

[0087] For example, the space may be at least a portion of the surrounding space where the electronic device (100) is located. The surrounding space may be a 360-degree surrounding space with the electronic device (100) as the center and the sensing distance as the radius. For example, the sensing distance refers to a distance that the electronic device (100) can sense through the lidar sensor (110) and may be a preset distance in the lidar sensor (110). However, the sensing distance may be implemented in various ways depending on the performance of each of the light-emitting unit and the ToF sensor of the lidar sensor (110).

[0088] For example, the electronic device (100) can identify whether a specific space in the surrounding space is a projectible area. For example, the projectible area may refer to an area suitable for the electronic device (100) to project an image.

[0089] For example, the processor (130) can obtain a depth map corresponding to a space and identify whether the space is a projectible area based on the obtained depth map.

[0090] For example, if the processor (130) determines that the set space is not a projectible area, it may reset an adjacent space or another space (e.g., a space spaced apart from the set space by a preset distance) as a space and determine whether the reset space is a projectible area. According to one embodiment, the processor (130) may acquire first sensing data and a plurality of second sensing data through the lidar sensor (110).

[0091] For example, the first sensing data and the plurality of second sensing data may be data acquired by sensing space in different directions as described above. For example, each of the first sensing data and the second sensing data may include a rotation angle and a phase difference between the received reflected light and the output light.

[0092] The processor (130) can identify a distance value for space based on this phase difference. Here, the distance value may be a distance value between the lidar sensor and the object identified by the ToF sensor. The distance value may also be referred to as distance, depth, depth value, depth, or depth value.

[0093] However, it is not limited thereto, and if the lidar sensor (110) is equipped with a dToF sensor, it may be the time required for the light output from the lidar sensor (110) to be received by the dToF sensor. In this case, the processor (130) can identify a distance value for space based on the required time.

[0094] The processor (130) can obtain a depth map (10) including distance information of the space based on the identified distance value.

[0095] Meanwhile, the processor (130) can obtain a depth map (10) based on other types of sensing data in addition to the first sensing data and the second sensing data.

[0096] According to FIG. 3, a process (2000) is illustrated in which a processor (130) obtains a depth map (10) including distance information of a space based on first sensing data, second sensing data, and third sensing data.

[0097] The processor (130) can obtain third sensing data through the RGB sensor (140). The RGB sensor (140) may be a sensor that detects three basic colors, red, green, and blue, to generate a color image.

[0098] For example, the RGB sensor (140) may be implemented as a camera. Here, the camera is a device capable of capturing still images and moving images, and may include one or more image sensors (e.g., a front sensor or a rear sensor), a lens, an image signal processor (ISP), and a flash (e.g., an LED, a Xenon lamp, etc.).

[0099] According to one embodiment, the RGB sensor (140) can capture an arbitrary object under the control of the processor (130) and transmit the captured data to the processor (130). Of course, the captured data may be stored in the memory (120) under the control of the processor (130). Here, the captured data may be called various things such as pictures, images, still images, and moving images, but for the convenience of explanation, they will be collectively referred to as images hereinafter. Meanwhile, it should be understood that the images according to various embodiments of the present disclosure may also mean, in addition to live view images captured by a camera, images received from an external device or an external server, images stored in the memory (120), etc.

[0100] The processor (130) can obtain a color image of the environment surrounding the electronic device (100) through the RGB sensor (140). That is, the processor (130) can obtain the obtained color image as third sensing data.

[0101] Here, the third sensing data may include multiple object areas. Here, the multiple objects may refer to multiple objects recognized by the electronic device (100) as existing in the space surrounding the electronic device. Here, the multiple object segmentation images may refer to the areas occupied by each object. Here, the segmentation image is an image obtained by performing RGB segmentation (hereinafter, “segmentation”) on an RGB image obtained by an RGB sensor. For example, segmentation may be a process of analyzing pixels included in an RGB image to identify and separate multiple objects through a lidar sensor (110).

[0102] The processor (130) can obtain a depth map (10) using both the lidar sensor (110) and the RGB sensor (140). That is, the processor (130) can obtain a depth map (10) including distance information in space based on first sensing data obtained through the lidar sensor (110), a plurality of second sensing data, and third sensing data obtained through the RGB sensor (140).

[0103] The processor (130) can also obtain a depth map (10) based on other sensing data. Here, the other sensing data may correspond to sensing data obtained by other sensors (e.g., a light sensor, a distance sensor, a biometric information sensor, an acceleration sensor, etc.) mounted on the electronic device (100) in addition to the lidar sensor (110) and the RGB sensor (140). Hereinafter, an operation of the electronic device (100) to obtain the depth map (10) will be described in detail.

[0104] FIG. 4 is a drawing for explaining the operation of an electronic device according to at least one embodiment of the present disclosure.

[0105] According to FIG. 4, the electronic device (100) includes a lidar sensor (110), and the electronic device (100) can sense space (20) through the lidar sensor (110).

[0106] The electronic device (100) may be an electronic device (100) having an external appearance as illustrated in FIG. 4, but is not limited thereto.

[0107] The lidar sensor (110) may be a sensor having an external appearance as shown in FIG. 4, but is not limited thereto.

[0108] The electronic device (100) can sense a space (20) along multiple lines. Here, the multiple lines may correspond to multiple sensing directions. The lowest line among the multiple lines may correspond to a line along which the electronic device (100) senses the space in the sensing direction of the lidar sensor (110) corresponding to a preset angle.

[0109] Specifically, the electronic device (100) controls the lidar sensor (110) to sense space in a sensing direction of the lidar sensor (110) corresponding to a preset angle based on the occurrence of a preset event so that the lidar sensor acquires first sensing data, and the electronic device can acquire the first sensing data through the lidar sensor (110) and store it in the memory (120).

[0110] The electronic device (100) can identify whether a preset event has occurred.

[0111] The event set here is an event for the electronic device (100) to sense space.

[0112] For example, the preset event may be the activation of the electronic device. Specifically, based on the occurrence of the preset event in which the electronic device is activated to sense a space, the electronic device may control the lidar sensor (110) to sense the space in a direction of the lidar sensor (110) corresponding to a preset angle so that the lidar sensor (110) acquires first sensing data, wherein the preset angle may correspond to an angle parallel to the ground based on the direction in which the lidar sensor is facing.

[0113] According to one embodiment, when the electronic device (100) receives a signal from an external device, an event may occur that activates the electronic device (100) to sense space. For example, when the electronic device (100) receives a signal requesting transmission of sensing data from an external device, the electronic device (100) may be activated to sense space. Alternatively, when the electronic device (100) receives a signal requesting transmission of distance information acquired by the electronic device (100) from an external device, the electronic device (100) may be activated to sense space.

[0114] According to one embodiment, when the electronic device (100) receives a user input that causes the electronic device (100) to acquire sensing data, the electronic device (100) may be activated to sense space, but is not limited thereto.

[0115] When the electronic device (100) identifies that a preset event has occurred, it can control the lidar sensor to sense space in the sensing direction of the lidar sensor (110) corresponding to the preset angle.

[0116] The preset angle here is the angle that the lidar sensor (110) first senses in space after a preset event occurs. The angle here may be an angle relative to the ground.

[0117] For example, the preset angle may be an angle parallel to the ground. In this case, when the electronic device (100) identifies that a preset event has occurred, it may control the lidar sensor (110) to sense space in a sensing direction corresponding to the angle parallel to the ground.

[0118] For example, the preset angle may be set to various angles depending on the terrain where the electronic device (100) is located. For example, if the electronic device (100) is located on an inclined surface at a specific angle, the preset angle may be an angle parallel to a flat surface located below the inclined surface. In this case, the electronic device (100) may control the lidar sensor (110) to sense space in a sensing direction corresponding to an angle parallel to a flat surface rather than an angle parallel to the inclined surface.

[0119] However, the preset angle is not limited to the example above, and the preset angle can be set to various angles depending on the user settings.

[0120] The electronic device (100) can control the lidar sensor (110) to obtain first sensing data through the lidar sensor (110) and store it in the memory (120).

[0121] The first sensing data may be data acquired by the lidar sensor (110) by sensing the surroundings of the electronic device (100) in a 360-degree omnidirectional manner. In this case, the electronic device (100) may acquire sensing data for the space surrounding the electronic device (100) in a 360-degree direction around the electronic device (100).

[0122] The electronic device (100) can acquire first sensing data and store it in the memory (120), and then control the lidar sensor (110) to sequentially change the sensing direction of the lidar sensor (110).

[0123] Here, the sensing direction of the lidar sensor (110) may mean the up-down direction in which the lidar sensor (110) senses space.

[0124] If the electronic device (100) cannot change the direction of the lidar sensor (110) separately from the electronic device (100), the vertical sensing direction may match the direction in which the front of the electronic device (100) is facing.

[0125] The electronic device (100) can change the sensing direction of the lidar sensor (110) in the vertical direction. Here, the vertical direction can vary depending on the tilting angle of the lidar sensor (110). At this time, as described above, the electronic device (100) can control the driving unit of the lidar sensor (110) to sense the surroundings of the electronic device (100) in a 360-degree omnidirectional manner.

[0126] At this time, the electronic device (100) may include an actuator for controlling the sensing direction of the lidar sensor (110). The actuator may be implemented as a motor. The electronic device (100) may control the lidar sensor (110) to change the sensing direction. At this time, the actuator may include an encoder. Here, the encoder may record a tilting angle according to the sensing direction.

[0127] Here, sequentially changing may mean that the electronic device (100) changes the sensing direction of the lidar sensor (110) in one direction multiple times. Alternatively, the electronic device (100) may continuously change the sensing direction of the lidar sensor (110). This will be described in detail in FIG. 5b.

[0128] For example, after acquiring the first sensing data, the electronic device (100) can change the sensing angle of the lidar sensor (110) vertically upward or downward several times.

[0129] For example, the electronic device (100) can change the sensing direction of the lidar sensor (110) several times to a final angle, and then sequentially change the sensing direction again from the final angle.

[0130] Here, the electronic device (100) can sequentially change the sensing direction according to a preset angular interval. For example, the preset angular interval may be a constant angular interval. However, this is not limited thereto, and the preset angular interval may be a non-constant interval as the sensing direction of the lidar sensor (110) is sequentially changed.

[0131] Meanwhile, the electronic device (100) can control the lidar sensor (110) to sequentially change its direction in the up-and-down direction while the left-right sensing direction range is within a preset angular range. Here, the left-and-right sensing direction can be referred to as a horizontal sensing direction.

[0132] Here, the range of the left and right sensing directions may be the range of the direction in which the lidar sensor (110) senses the space horizontally. That is, the electronic device (100) may limit the range to a preset angle when the lidar sensor (110) senses the space around the electronic device (100) in all directions. However, this is not limited thereto, and the preset angle range may correspond to various angle ranges set by the driving direction, location, and user settings of the electronic device (100).

[0133] Accordingly, when the electronic device (100) does not require distance information for a space other than a space with a limited angular range described above among the spaces around the electronic device (100), the electronic device (100) can limit the sensing range to an angular range as needed, thereby omitting unnecessary data processing and reducing power consumption.

[0134] The electronic device (100) can sequentially change the sensing direction of the lidar sensor (110) and acquire second sensing data in each sensing direction. Then, the electronic device (100) can store the acquired second sensing data in the memory (120), and accumulate and store a plurality of second sensing data according to each sensing direction in the memory (120).

[0135] FIG. 5A and FIG. 5B are diagrams illustrating an operation of obtaining a depth map according to at least one embodiment of the present disclosure.

[0136] According to FIG. 5a, the electronic device (100) is located at a distance D from the wall surface, which is space (20), and the lidar sensor (110) is located at multiple angles ( ) can sense the space (20) in the sensing direction according to the final angle ( ) is assumed to be spaced apart by a distance d corresponding to the direction according to .

[0137] The electronic device (100) is set at a preset angle The space (20) can be sensed in the sensing direction of the lidar sensor (110) corresponding to the lidar sensor (110) to obtain the first sensing data through the lidar sensor (110). And the electronic device (100) controls to sequentially change the sensing direction of the lidar sensor (110). ) can acquire multiple second sensing data.

[0138] Meanwhile, as described above, the electronic device (100) can obtain second sensing data by changing the lidar sensor in the up-down direction while the left-right sensing direction range of the lidar sensor (110) is within a preset angular range.

[0139] At this time, the electronic device (100) can sense the space opposite to the direction the electronic device is facing (rear). That is, when the electronic device (100) or the lidar sensor (110) is tilted and the sensing direction of the lidar sensor (110) is changed, the opposite space can be sensed.

[0140] Specifically, the electronic device (100) controls the lidar sensor to sense space in a sensing direction corresponding to a preset angle according to a preset event. At this time, the electronic device (100) can control the lidar sensor (110) to sense an opposite space, thereby acquiring first sensing data for the opposite space.

[0141] Thereafter, the electronic device (100) senses space by controlling the sensing direction of the lidar sensor (110). At this time, the electronic device (100) can sense objects in the floor area existing in the opposite space and control the lidar sensor (110) to obtain a plurality of second sensing data through the lidar sensor (110).

[0142] Accordingly, the lidar sensor (110) senses an obstacle located behind the electronic device (100) or a charging station device for charging the electronic device (100), so that the electronic device (100) can obtain distance information from the electronic device (100) to the obstacle or charging station.

[0143] Meanwhile, according to FIGS. 5a and 5b, D represents the distance between the electronic device (100) or the lidar sensor (110). represents the rotation angle (rad) of the actuator (tilting motor). r represents the radius of the electronic device (100). d represents the distance between the tilted lidar sensor (110) and the wall.

[0144] h may refer to the vertical offset of the sensing points when the lidar sensor (110) senses a limited left-right angular range, then tilts and rotates 360 degrees to sense the limited angular range. For example, since the lidar sensor (110) rotates and senses the limited angular range in a row in the left-right direction, the sensing points are arranged in a row to form the first row. Afterwards, when the lidar sensor (110) is tilted upwards and rotates 360 degrees again to sense space, the sensing points are arranged in a second row above the first row. In this case, h may refer to the interval between the first row and the second row.

[0145] However, as an example, if the lidar sensor (110) senses a limited angular range left and right while rotating, and at the same time, the lidar sensor (110) is continuously tilted upward, the first row and the second row, etc. described above may not appear parallel to the ground. For example, if the rotation direction of the lidar sensor (110) is from left to right while facing space, the first row and the second row, etc. may show a right-upward shape. In this case as well, the h value may mean the gap between the first row and the second row.

[0146] According to the mathematical formula (12a) of Fig. 5b, the distance d is the distance that the lidar sensor (110) moves while tilting and the tilting angle from the original position of the lidar sensor (110). When tilted, it may correspond to the sum of the distances from the sensing point. Referring to the figure of Fig. 5a, the distance d is the distance D, the radius r, and the angle It can be expressed using the following mathematical expressions 1 and 2.

[0147]

[0148]

[0149] Meanwhile, the vertical interval h is the distance d, D and the angle It can be expressed as mathematical formula 3 as follows.

[0150]

[0151] Based on mathematical expressions 1 to 3, the upper and lower interval h is determined according to the following mathematical expression 4. can be expressed as a function of .

[0152]

[0153] That is, the tilting angle As increases, the value of h increases. Graph (12b) shows the case where r = 0.1 m (10 cm) and D = 3 m. Indicates the h value according to the change in value. As the value increases, the slope also increases, causing the value of h to increase.

[0154] That is, when the lidar sensor (110) senses space by gradually increasing the tilting angle, the density of the sensing points may decrease as the upper and lower interval h of the sensing points increases. Accordingly, when a depth map is formed based on the sensing data acquired by the lidar sensor (110), the interval of distance values ​​may become wider as the distance from the sensing point near the lidar sensor (110) increases.

[0155] Returning to FIG. 5A, the electronic device (100) can acquire first sensing data and a plurality of second sensing data according to the various methods described above. Thereafter, the electronic device (100) can acquire a depth map (10) including distance information of a space (20) based on the first sensing data and the plurality of second sensing data.

[0156] Specifically, as described above, each of the first sensing data and the plurality of second sensing data may include a phase difference between the phase of the received reflected light and the phase difference between the output light. Based on the phase difference included in each sensing data, the electronic device (100) may identify a plurality of distance values ​​for a space in the direction in which each sensing data was acquired.

[0157] An electronic device (100) can obtain a depth map (10) including distance information of a space (20). The distance information can include a plurality of coordinates and a distance value corresponding to each coordinate.

[0158] Here, each of the plurality of coordinates is a coordinate in a virtual three-dimensional space and may correspond to a point where the lidar sensor (110) senses the space (20). Here, the sensing point may mean a point where the light output by the lidar sensor (110) is reflected in the space (20).

[0159] The electronic device (100) can obtain a rotation angle from the encoder of the aforementioned lidar sensor (110) and a tilting angle from the encoder of the aforementioned actuator. The electronic device (100) can identify a sensing point by matching the obtained rotation angle and tilting angle.

[0160] The electronic device (100) can identify multiple coordinates through the sensing point and the identified distance value. That is, the electronic device (100) can identify multiple coordinates in the three-dimensional space by mapping the sensing point and the distance value of the point to a virtual three-dimensional space. Here, information about the sensing point can be obtained by an encoder of a motor for controlling the rotation of the lidar sensor (110). The encoder may refer to a device that measures the rotation angle or speed of the motor. The lidar sensor (110) can obtain information about the sensing point by identifying the rotation angle recorded by the encoder at the time of outputting light.

[0161] Meanwhile, the electronic device (100) can obtain a depth map by inputting first sensing data and a plurality of second sensing data into an artificial intelligence model.

[0162] Here, the artificial intelligence model may be a model trained to obtain a depth map including distance information between an object included in space and the lidar sensor based on first training sensing data received from the lidar sensor and a plurality of second training sensing data received by controlling the lidar sensor (110) to sequentially change the sensing direction.

[0163] Compared to the case where the lidar sensor (110) of the electronic device (100) senses space while being fixed toward the center of the space, when the electronic device (100) acquires distance information based on the first sensing data and the plurality of second sensing data according to a preset event as described above, a depth map including richer distance information can be acquired.

[0164] Meanwhile, the electronic device (100) can acquire information other than distance information of the space (20) based on the first sensing data and the plurality of second sensing data. The electronic device (100) can update the distance information included in the depth map (10) acquired first based on the other acquired information.

[0165] For example, the electronic device (100) can obtain information about the structure of space based on the first sensing data and a plurality of second sensing data.

[0166] FIGS. 6A and 6B are diagrams illustrating an operation of obtaining vanishing point information according to at least one embodiment of the present disclosure.

[0167] According to FIG. 6, the electronic device (100) can obtain vanishing point information (11) based on first sensing data and a plurality of second sensing data.

[0168] Specifically, the electronic device (100) can detect a vanishing point. Here, the vanishing point may refer to a point where different parallel straight lines detected from space appear to converge.

[0169] Specifically, the electronic device (100) can obtain distance information of a space (20) based on first sensing data and a plurality of second sensing data.

[0170] The electronic device (100) can identify an edge based on the acquired distance information. Here, the edge can be detected by an edge detection algorithm (e.g., a Canny edge detection algorithm). Specifically, the electronic device (100) can identify two coordinates with the greatest change in distance values ​​and acquire the edge coordinates of a point located in the center between the two coordinates. The electronic device (100) can identify one edge including the acquired multiple edge coordinates. The electronic device (100) can identify multiple edges through the lidar sensor (110) by repeating the above process for the distance information.

[0171] The electronic device (100) can identify a plurality of straight lines among the identified plurality of edges. Here, the straight lines can be detected by a straight line detection algorithm (e.g., a Hough transform algorithm). Specifically, the electronic device (100) can detect straight lines as points of intersection in a new coordinate space expressed by Hough transforming edge coordinates. Here, the Hough transform refers to transforming a straight line equation into a parameter (e.g., slope and intercept) space. A straight line can be detected as a point of intersection of a plurality of straight lines obtained in the parameter space. Here, the straight line can be detected by substituting the coordinates of the intersection points into the straight line equation. However, the algorithm for detecting straight lines in this way is only one example, and the straight line detection algorithm can be implemented in various ways. Here, the detected straight line can be expressed in the form of a three-dimensional equation in a virtual three-dimensional space.

[0172] The electronic device (100) can identify one or more coordinates at which the identified plurality of straight lines converge or intersect. The electronic device (100) can identify the one or more identified coordinates as coordinates of a vanishing point. Furthermore, the electronic device (100) can identify information about the plurality of straight lines converging or intersecting with the identified vanishing point as vanishing line information. Here, the vanishing line information can include a plurality of coordinates included in the vanishing line. Alternatively, the vanishing line information can be expressed using the three-dimensional equation described above.

[0173] The electronic device (100) can obtain the acquired vanishing point coordinates and vanishing line information as vanishing point information.

[0174] The electronic device (100) can update distance information based on the acquired vanishing point information. This will be described in detail below.

[0175] The electronic device (100) can interpolate distance information based on the coordinates of the vanishing point and information on the vanishing line. Here, interpolation may refer to a technique for estimating unknown data by obtaining an interpolation polynomial from known data.

[0176] Here, the base data may be distance information acquired by the electronic device (100) based on the first sensing data and the second sensing data. The unknown data is described below.

[0177] As described above, the lidar sensor (110) rotates, outputs light, does not output light for a preset time interval, and then outputs light again after a preset time has elapsed. While the lidar sensor (110) does not output light, the lidar sensor (110) does not receive light and therefore cannot identify a phase difference. Accordingly, the electronic device (100) cannot obtain distance values ​​only for coordinates at a certain interval. In other words, a distance value may not exist for a specific area. The unknown data described above may mean a distance value corresponding to each of a plurality of coordinates occupied by the specific area.

[0178] The electronic device (100) can identify a plurality of coordinates included in the coordinates of the vanishing point and the vanishing line information as update coordinates at preset intervals. The preset interval refers to the interval between the update coordinates and may be a value that can be set according to the desired resolution. The electronic device (100) can obtain an update distance value from each of the plurality of identified update coordinates. Specifically, the electronic device (100) can obtain the update distance value from the update coordinates through the reverse process of the process of mapping the aforementioned sensing points and distance values ​​to a three-dimensional space. Accordingly, the electronic device (100) can obtain update distance information including the update coordinates and the update distance value.

[0179] Meanwhile, the electronic device (100) can extend a vanishing line based on the coordinates of the vanishing point and identify a plurality of coordinates included in the extended vanishing line as update coordinates. The electronic device (100) can obtain an update distance value from each of the identified plurality of update coordinates. Accordingly, the electronic device (100) can obtain update distance information including the update coordinates and the update distance value. That is, the electronic device (100) can linearly extend the distance information obtained based on the first sensing data and the second sensing data.

[0180] According to FIG. 6b, the electronic device (100) can obtain a new depth map (10') including updated distance information.

[0181] That is, the electronic device (100) can obtain a spatial depth map including more distance information by interpolating the distance information (or 3D ToF information) acquired based on the first sensing data and the second sensing data based on the vanishing point information.

[0182] Accordingly, the electronic device (100) can obtain a first depth map (10) based on the first sensing data and the second sensing data, and update the first depth map based on the vanishing point information to obtain a second depth map (or new depth map) (10').

[0183] Meanwhile, the electronic device (100) may also update the distance information of the depth map (10) based on third sensing data acquired by an RGB sensor (140) in addition to the lidar sensor (110).

[0184] FIGS. 7 to 10 are drawings for explaining an operation of updating distance information according to at least one embodiment of the present disclosure.

[0185] According to FIG. 7, the electronic device (100) can obtain an extended depth map (10') from a sparse depth map (10). Here, the extended depth map (10') may also be referred to as an updated depth map or a dense depth map.

[0186] Here, among the multiple coordinates included in the distance information, if the average of the distance values ​​between two adjacent coordinates is greater than or equal to a threshold value, the depth map including such distance information may be referred to as a sparse depth map (10). On the other hand, if the average of the distance values ​​between two adjacent coordinates is less than or equal to a threshold value, the depth map including such distance information may be referred to as a dense depth map (10).

[0187] The extended depth map (10') may include updated distance information included in the sparse depth map (10). Accordingly, the extended depth map (10') may include distance information for a space (20) with a wider field of view (FoV) than the sparse depth map (10).

[0188] The electronic device (100) can utilize a color image of space (20) to obtain an extended depth map (10').

[0189] A space (20) may include multiple objects (e.g., multiple chairs, multiple desks, a blackboard, etc.). The electronic device (100) may perform RGB segmentation on a color image in which multiple objects are captured to obtain a segmentation image (30).

[0190] Here, the segmentation image (30) may include multiple object areas (31) distinguished by different colors for each object type (e.g., chair, desk, blackboard). The multiple object areas (31) may also be displayed by distinguishing them by different patterns (e.g., diagonal lines, check marks, horizontal lines, etc.) for each object.

[0191] The electronic device (100) can obtain an extended depth map (10') based on distance information of a space (20) included in the depth map (10) and the plurality of object areas (31) above. This will be described in detail below with reference to FIG. 8.

[0192] According to FIG. 8, the electronic device (100) can obtain a depth map (10) based on the first sensing data and the second sensing data obtained by the lidar sensor (110).

[0193] The electronic device (100) can identify multiple object areas (31) based on third sensing data acquired through the RGB sensor (140).

[0194] The electronic device (100) can obtain a color image of space (20) as third sensing data.

[0195] The electronic device (100) can identify multiple object areas (31) by segmenting an RGB image.

[0196] Specifically, the electronic device (100) can obtain a segmentation image (30) including segmentation information for an area (31) (hereinafter, referred to as an object area) corresponding to each of a plurality of objects included in a color image. That is, the electronic device can use a segmentation model to identify a plurality of objects included in a color image for a space (20), and obtain a segmentation image (30) that represents segmentation information for a plurality of object areas (31) with a corresponding color (or pattern). Here, the plurality of object areas (31) may correspond to an area occupied by a specific object in the segmentation image (30).

[0197] The electronic device (100) can identify distance information of a plurality of object areas identified in the third sensing data based on the distance information included in the depth map (10) described above.

[0198] Specifically, the electronic device (100) can identify distance information of at least one object by aligning the depth map with an image acquired by an RGB sensor. The electronic device (100) can obtain a distance value of the object by identifying at least one point overlapping with an object area (31) included in a segmentation image (30). Here, the point may be a point included in the depth map (10) and having different sizes depending on the distance value corresponding to each coordinate.

[0199] For example, the electronic device (100) can identify a point having the smallest size among at least one point, and identify a distance value corresponding to that size as the distance in the object area. That is, the electronic device (100) can identify a point at the farthest position among points overlapping an object, and identify a distance corresponding to that point as the distance in the object area.

[0200] For example, the electronic device (100) may identify the point with the largest size among the points overlapping the object area (31) and identify the point at the closest location as the distance value of the object. Alternatively, the electronic device may identify a plurality of overlapping points and identify the average of the distance values ​​corresponding to the size of each point as the distance value of the object.

[0201] The electronic device (100) can identify distance information of an object area. The distance information of the object area can include an identified distance value.

[0202] Meanwhile, the lidar sensor (110) and the RGB sensor (140) can be mounted at different locations of the electronic device (100) to sense space. The lidar sensor (110) and the RGB sensor (140) can sense space at different angles. In the case where there is a difference in the location and angle of the lidar sensor (110) and the RGB sensor, errors may occur in the distance information of multiple object areas of the electronic device (100).

[0203] According to FIG. 8b, to compensate for this difference, the electronic device (100) can match the viewpoints of the sensing data.

[0204] Specifically, the electronic device (100) can match the view points of the first sensing data acquired through the lidar sensor (110), the second sensing data, and the third sensing data acquired through the RGB sensor.

[0205] For example, matching the view points of the first sensing data, the second sensing data, and the third sensing data by the electronic device (100) here may mean matching the view points by performing coordinate transformation on the depth map acquired with the first sensing data and the second sensing data.

[0206] First, the electronic device (100) can identify the difference in sensing angles. Specifically, the electronic device (100) can perform calibration to determine the relative positions and angles of the RGB sensor and the lidar sensor (110). For example, by using a calibration pattern (e.g., a checkerboard, etc.), the sensing angles of each sensor can be identified and the difference in sensing angles can be identified. Here, the calibration pattern can be a pattern drawn indoors or outdoors, an image output by an external electronic device, or an image projected by the electronic device (100) onto an indoor or outdoor wall using a projection device.

[0207] The electronic device (100) may perform coordinate transformation for each of the plurality of coordinates in order to compensate for the sensing angle difference as described above. Here, the coordinate transformation may be based on the difference in the sensing angles of the lidar sensor (110) and the RGB sensor (140). For example, the coordinate transformation may be an affine transformation. The affine transformation maintains parallelism even after transformation while maintaining the characteristics of a linear transformation. That is, the electronic device (100) may transform each coordinate into a new coordinate while maintaining the distance value corresponding to each coordinate for each of the plurality of coordinates.

[0208] The electronic device (100) can obtain new transformed distance information through the above-described coordinate transformation. The transformed distance information can include a plurality of transformed coordinates and a distance value corresponding to each coordinate. The electronic device (100) can obtain a transformed depth map including the transformed distance information. The transformed depth map (10) can be a depth map whose viewpoint is matched with an RGB image.

[0209] For example, a transformed depth map (10) can be implemented as a two-dimensional image by mapping a plurality of transformed coordinates to their respective corresponding locations. The location of each point can correspond to the transformed coordinates.

[0210] The electronic device (100) can identify distance information of each of at least one object region by aligning (or fusing) the transformed depth map with the image acquired by the RGB sensor. The electronic device (100) can identify distance information of the object region by identifying at least one point overlapping with the object region (31) existing in the segmentation image. Since the operation of the electronic device (100) identifying distance information of the object region has been described in detail in the above-described section, a redundant description will be omitted.

[0211] Thereafter, the electronic device (100) can update the distance information included in the depth map (10) based on the distance information of the identified object area, and may be referred to as an updated depth map (or dense depth map) (10') including the updated distance information. This will be described in detail with reference to Drawing 8a.

[0212] Returning to FIG. 8A again, the electronic device (100) can identify distance information of a plurality of object areas identified in the third sensing data based on distance information included in the depth map, and update the distance information included in the depth map (10) based on the distance information of the plurality of identified object areas.

[0213] For example, the electronic device (100) can identify an area corresponding to each of the plurality of object areas (31) in the depth map (10) based on the plurality of object areas (31) included in the third sensing data, and the electronic device (100) can update the area corresponding to each of the plurality of object areas (31) in the depth map (10) with the same distance information.

[0214] Here, the corresponding area may mean an area that overlaps with the object area (31) in the depth map (10) when the depth map (10) or the transformed depth map is aligned with the segmentation image (30).

[0215] The electronic device (100) can update distance information by mapping distance values ​​to coordinates occupied by corresponding areas in the depth map (10).

[0216] Here, the same distance information may be one of the minimum distance value, maximum distance value, and average distance value in the object area, according to the above-described embodiment.

[0217] Taking Fig. 7 as an example, the electronic device (100) can update the area corresponding to the area of ​​the closest desk in the depth map (10) with the same distance information.

[0218] According to the example described above, when the electronic device (100) matches the viewpoints of the first sensing data to the third sensing data, the distance information included in the depth map can be updated based on the first sensing data, the second sensing data, and the third sensing data with the matched viewpoints. That is, the electronic device (100) can update the distance information included in the depth map by aligning the change depth map acquired by converting the depth map with the segmentation image (30).

[0219] The electronic device (100) can obtain an updated depth map (10') based on the updated distance information.

[0220] Meanwhile, the electronic device (100) can also identify structural information of a space (20) using the above-described segmentation image (30).

[0221] According to FIG. 9, the electronic device (100) obtains a segmentation image (30) for a space (20), and through this obtains information on a vanishing point (32) of the space (20).

[0222] Here, the vanishing point (32) may correspond to a point where multiple parallel lines included in the segmentation image (30) appear to converge.

[0223] The electronic device (100) can analyze the projection space by identifying the vanishing point (32) and expand the distance information centered on the image vanishing point (32). The electronic device (100) can update the distance information included in the depth map (10) based on the vanishing point (32) information and obtain an updated depth map (10').

[0224] Here, the updated depth map (10') may include updated distance information included in the sparse depth map (10). Accordingly, the extended depth map (10') may include distance information for a space (20) with a wider field of view (FoV) than the sparse depth map (10).

[0225] The operation of the electronic device (100) to update distance information based on the vanishing point (32) information is described in detail below.

[0226] According to FIG. 10, the electronic device (100) can identify a plurality of object areas (31) included in third sensing data acquired through the RGB sensor. The electronic device (100) can identify at least one vanishing point (32) based on the identified plurality of object areas (31).

[0227] The electronic device (100) can identify edges based on the identified plurality of object regions (31). Here, the edges can be detected by an edge detection algorithm (e.g., a Canny edge detection algorithm). Specifically, the electronic device (100) can identify the boundary between two objects where the changes in R, G, and B values ​​are the greatest as an edge. The electronic device (100) can identify a plurality of edges by repeating the above process for the RGB segmentation image (30).

[0228] The electronic device (100) can identify a plurality of straight lines among the identified edges. Here, the straight lines can be detected using a straight line detection algorithm (e.g., the Hough Transform algorithm). Since the specific details of the straight line detection algorithm have been described in FIG. 6A, a redundant description will be omitted.

[0229] The electronic device (100) can identify one or more vanishing points (32) where the identified multiple straight lines converge. Specifically, the electronic device (100) can identify the vanishing points through at least one vanishing point detection algorithm. Such algorithms can be stored in the memory (120).

[0230] Here, the vanishing point detection algorithm may correspond to an algorithm that extracts multiple vanishing point candidates and repeats the same steps until the positions of the vanishing point candidates converge, or may detect one vanishing point (32) by repeating different steps.

[0231] For example, the electronic device (100) can obtain a vanishing point through the JL (J-Linkage) clustering algorithm. The JL clustering algorithm may correspond to an algorithm that classifies (clusters) a plurality of identified straight lines into a plurality of models to generate a plurality of vanishing point candidates and detects the vanishing point based on the similarity between the straight lines.

[0232] For example, the electronic device (100) can obtain a vanishing point through the Expectation-Maximization (EM) algorithm. The EM algorithm may correspond to an iterative algorithm that finds an estimate of a parameter having a maximum likelihood or a maximum a posteriori (MAP) probability in a probability model that depends on latent variables. That is, the EM algorithm may correspond to an algorithm that obtains multiple random vanishing point candidates, calculates the probability that multiple straight lines belong to each vanishing point candidate (E-step), updates the vanishing point candidates to a more probable position based on the calculated probability (M-step), and repeats the E-step and M-step until the vanishing point candidates converge to one point, thereby detecting a vanishing point.

[0233] However, the above-described algorithm is merely an example of an algorithm for detecting a vanishing point, and the electronic device (100) can obtain a vanishing point through various other types of vanishing point detection algorithms. Furthermore, the electronic device (100) can detect a vanishing point (32) through a single algorithm, and can obtain a vanishing point (32) by combining different algorithms.

[0234] Thereafter, the electronic device (100) can identify the location of the vanishing point (32) and the locations of multiple vanishing lines converging to the vanishing point (32) in the depth map (10) from the segmentation image (30). Specifically, the electronic device (100) can identify the location of the vanishing point (32) and the locations of multiple vanishing lines by aligning the depth map (10) with the segmentation image (30).

[0235] The electronic device (100) can obtain the location of the vanishing point (32) and the locations of multiple vanishing lines converging to the vanishing point (32) as vanishing point information.

[0236] The electronic device (100) can update the distance information included in the depth map (10) based on the acquired vanishing point (32) information. That is, the electronic device (100) can interpolate the distance information based on the vanishing point information. This will be described in detail below.

[0237] The electronic device (100) can identify an area between adjacent vanishing lines among a plurality of vanishing lines as a plane area (e.g., a floor, a wall, or a ceiling) moving away from a vanishing point (32). Here, the plane area may be an area on a depth map (10).

[0238] The electronic device (100) can identify the distance value of the vanishing point (32). Specifically, the electronic device (100) can identify the distance value of the point closest to the location of the vanishing point (32) as the distance value of the vanishing point (32). However, the present invention is not limited thereto.

[0239] The electronic device (100) can obtain distance values ​​at preset intervals based on the vanishing point (32) for a flat area. Specifically, the electronic device (100) can obtain a distance value that decreases from the distance value of the vanishing point (32) as it moves away from the vanishing point (32). Here, the degree to which the distance value decreases (e.g., the amount of change in the distance value per unit length) can increase as it moves away from the position of the vanishing point (32). The distance value obtained here can be referred to as an updated distance value, and the coordinates corresponding to the point where the distance value is obtained in the depth map (10) can be referred to as updated coordinates.

[0240] Accordingly, the electronic device (100) can obtain update distance information including update coordinates and update distance values. The electronic device (100) can obtain a new depth map (10') including the update distance information.

[0241] Meanwhile, returning to FIG. 9, if the sensing angle of the lidar sensor (110) is narrower than the sensing angle of the RGB sensor (140), a new depth map (10') with an expanded size can be obtained.

[0242] That is, the electronic device (100) can update the distance information included in the depth map (10) based on the RGB image acquired by the RGB sensor (140) by sensing a wider range than the field of view of the lidar sensor (110). The acquired updated distance information can include a wider range of coordinates than the existing distance information. Therefore, the new depth map (10') acquired by the electronic device (100) through the updated distance information can include distance information of a wider area than the previous depth map (10).

[0243] Accordingly, the electronic device (100) can obtain distance information with an angle of view (FoV) greater than that of the lidar sensor (110). That is, the electronic device (100) can update the distance information and obtain updated distance information for a wider range of space, greater than the angle of view according to the inherent characteristics of the lidar sensor (110).

[0244] FIG. 11 is a detailed block diagram illustrating a detailed configuration of an electronic device according to at least one embodiment of the present disclosure.

[0245] According to FIG. 11, the electronic device (100) may include at least one of a lidar sensor (110), a memory (120), an electronic device (100), an RGB sensor (140), a communication unit (150), a display (160), a projection unit (170), and a moving member (180).

[0246] The configuration illustrated in Fig. 11 is merely an example of various embodiments, and some configurations may be omitted and new configurations may be added. The contents already described in Fig. 2 are omitted.

[0247] The communication unit (150) is a configuration for the electronic device (100) to communicate with a plurality of external devices. The communication unit (150) can communicate with at least one external electronic device (200) or at least one server device.

[0248] The electronic device (100) may receive a signal requesting a depth map of a space of the external device from an external device or a server device through the communication unit (150), or may receive a signal requesting a depth map (10) of a space (20) of the electronic device (100) from the server device. Alternatively, the electronic device (100) may transmit a signal requesting a depth map (10) of a space (20) to the external device or the server device through the communication unit (150).

[0249] The communication unit (150) can transmit and receive various signals and data with external devices through various wired and wireless communication methods such as Bluetooth, AP-based Wi-Fi (Wi-Fi, Wireless LAN network), Zigbee, wired / wireless LAN (Local Area Network), WAN (Wide Area Network), Ethernet, IEEE 1394, HDMI (High-Definition Multimedia Interface), USB (Universal Serial Bus), MHL (Mobile High-Definition Link), AES / EBU (Audio Engineering Society / European Broadcasting Union), optical, coaxial, etc.

[0250] Meanwhile, the communication unit (150) may include a plurality of communication modules for performing different functions. According to an embodiment of the present disclosure, the electronic device (100) may separately include a communication module for communicating with an external electronic device (200) and a communication module for communicating with a server (300). For example, the electronic device (100) may communicate with the external electronic device (200) using a BT module, and the electronic device (100) may communicate with the server (300) using a Wi-Fi module. However, the present disclosure is not limited thereto.

[0251] The electronic device may further include an interface such as an HDMI port, DP, RGB, DVI, USB, Thunderbolt, etc. for receiving video / audio signals by being connected to external content sources. HMDI, DP, and Thunderbolt are ports that can transmit video and audio signals simultaneously. At least one processor (130) performs various processes such as demuxing, decoding, and scaling on content received from a content source through a communication unit (150) and these various interfaces to configure screen data, and provides the configured screen data to a display.

[0252] Meanwhile, the electronic device (100) can obtain distance information about space based on additional information received from an external device through the communication unit (150) described above.

[0253] The electronic device (100) can receive in-home map information from an external device and obtain a depth map based on the received information.

[0254] FIGS. 12 and 13 are drawings for explaining in-home map information in at least one embodiment of the present disclosure.

[0255] Referring to FIG. 12, there may not be a wall or other object within the sensing range of the electronic device (100) for projecting content. Here, the sensing range may refer to a range within which the lidar sensor (110) included in the electronic device (100) can recognize the surrounding environment. Here, the range within which the surrounding environment can be recognized may include a distance range and an angular range (or a field of view range).

[0256] If there is no wall or other object within the sensing range of the electronic device (100), at least one processor (130) may not obtain the first sensing data and the second sensing data and may identify that there is no object in the surroundings.

[0257] In the case above, the electronic device (100) can receive indoor map information from an external electronic device located in the house (1).

[0258] Here, the external electronic device (200) may be an air conditioner (200-1), a mobile device (200-2), a movable indoor robot (200-3), and a server capable of communicating with the indoor electronic devices. Although the external electronic devices are illustrated as the air conditioner (200-1), the mobile device (200-2), and the movable indoor robot (200-3), the external electronic devices are not limited thereto.

[0259] The external electronic device may be a device that performs a function similar to that of the electronic device (100), such as a portable projector. Furthermore, the external electronic device may include, for example, at least one of a television, a digital video disk (DVD) player, a smartphone, a tablet PC, an audio device, a refrigerator, a vacuum cleaner, an oven, a microwave oven, a washing machine, an air purifier, a set-top box, a home automation control panel, a security control panel, a media box, a camcorder, or an electronic picture frame. However, the present invention is not limited thereto.

[0260] Referring to FIG. 13, the electronic device (100) can communicate with external electronic devices (200), such as an air conditioner (200-1), a mobile device (200-2), a movable indoor robot (200-3), and a server (300) capable of communicating with indoor electronic devices.

[0261] The server (300) may be implemented as a server device, a cloud server device, etc., but is not limited thereto, and may be implemented as various devices such as a PC or laptop PC. In the present disclosure, the server (300) is illustrated and described as if it were a single server device, but the server (300) may be implemented as a plurality of servers.

[0262] For example, if an external electronic device (200) such as an air conditioner (200-1), a mobile device (200-2), or a movable indoor robot (200-3) is an Internet of Things (IoT) device equipped with a communication function, the server (300) may correspond to a device that can be connected to the external electronic devices (200) using a wireless network. In this case, the server (300) may build an Internet of Things system with the external electronic device (200), which is an Internet of Things device.

[0263] The electronic device (100) can request information within the map from the server (300) through the communication unit (150).

[0264] An external electronic device (200) may acquire map information or receive and store it from an external device. In particular, a movable indoor robot (200-3) may accumulate distance information of the surrounding space while moving within the home (1). The movable indoor robot (200-3) may acquire and store map information of the home (1) based on the accumulated distance information.

[0265] When the server (300) receives a request signal from the electronic device (100), it transmits a signal requesting in-home (1) map information to each external electronic device (200), and when each external electronic device (200) stores in-home (1) map information, it can transmit in-home (1) map information or distance information to the server (300).

[0266] The electronic device (100) can receive map information or distance information received from the server (300) from the external electronic device (200).

[0267] Accordingly, the electronic device (100) can obtain a new depth map for a space outside the sensing range based on the acquired distance information, the received in-home map information, and the new distance information.

[0268] Hereinafter, returning to the description of Fig. 11, the detailed configuration of the electronic device (100) will be described.

[0269] Referring to FIG. 11, the electronic device (100) may include a display (160). The electronic device (100) may further include a display.

[0270] The display (160) is a configuration for displaying the operating status, notification messages, UI screens, etc. of the electronic device (100). The display (160) may be implemented as various types of displays, such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, a PDP (Plasma Display Panel), etc. The display (160) may also include a driving circuit, a backlight unit, etc., which may be implemented as a type of a-si TFT (amorphous silicon thin film transistor), LTPS (low temperature poly silicon) TFT, OTFT (organic TFT), etc. Meanwhile, the display (160) may be implemented as a touch screen combined with a touch sensor, a flexible display, a three-dimensional display (3D display, three-dimensional display), etc. Alternatively, the display (160) may be implemented with only one or a plurality of light-emitting elements. The electronic device (100) can change the display status of the display (160) according to various states, such as when the electronic device (100) is turned on, when it is in a normal operating state, when it is low on power or in an error state, so that the user can intuitively understand the status of the electronic device (100).

[0271] However, the display (160) configuration is only a part of various embodiments, and the display (160) configuration may be omitted. That is, the electronic device (100) may be a device directly equipped with the display (160) or a device connected to an external electronic device. For example, when the electronic device (100) is implemented as a set-top box, a one-connect box, a projector, etc., the operations of the electronic device (100) described above may also be performed in an electronic device that does not include the display (160).

[0272] Meanwhile, the electronic device (100) may include a projection unit (170).

[0273] The projection unit (170) is a component that projects an image to the outside. According to various embodiments of the present disclosure, the projection unit (170) can be implemented with various projection methods (e.g., CRT (cathode-ray tube) method, LCD (Liquid Crystal Display) method, DLP (Digital Light Processing) method, laser method, etc.). For example, the CRT method has the same principle as a CRT monitor. The CRT method magnifies the image with a lens in front of the cathode-ray tube (CRT) and displays the image on the screen. Depending on the number of cathode-ray tubes, it is divided into a single-tube type and a three-tube type, and in the case of a three-tube type, the red, green, and blue cathode-ray tubes can be implemented separately.

[0274] Another example is the LCD method, which displays images by passing light from a light source through liquid crystals. LCD methods are divided into single-panel and three-panel types. In the three-panel type, light from a light source is separated into red, green, and blue by a dichroic mirror (a mirror that reflects only certain colors of light and transmits all others). The light then passes through the liquid crystals and is then refocused into a single point.

[0275] Another example is the DLP method, which displays images using a DMD (Digital Micromirror Device) chip. The DLP projection unit may include a light source, a color wheel, a DMD chip, a projection lens, etc. Light output from the light source can be colored as it passes through the rotating color wheel. The light passing through the color wheel is input to the DMD chip. The DMD chip is composed of numerous micromirrors. The DMD chip reflects the input light. The projection lens can play a role in magnifying the light reflected from the DMD chip to the size of the image.

[0276] As another example, a laser-based projection unit (170) includes a Diode Pumped Solid State (DPSS) laser and a galvanometer. To output various colors, DPSS lasers may be provided for each RGB color. The galvanometer uses a motor to rapidly rotate a mirror to reflect the laser. For example, the galvanometer can rotate the mirror at a maximum speed of 40 KHz / sec.

[0277] The electronic device (100) can acquire a depth map based on sensing data acquired from a lidar sensor (110) or an RGB sensor (140), and can acquire information on a projectible area within a space based on the acquired depth map. The memory (120) can store information on a plurality of projectible areas. Here, the projectible area refers to an area on which the electronic device (100) can project an image.

[0278] The projectable area may include areas that can provide users with uninterrupted images, such as walls, floors, areas where screens are installed, furniture areas, blind areas, and the like. Furthermore, "information regarding multiple projectable areas" may include information regarding the location of each of the multiple projectable areas, the flatness of each of the multiple projectable areas, and the color of each of the multiple projectable areas.

[0279] The electronic device (100) of the present disclosure can capture images of the user's surrounding space to obtain information on multiple projectible areas. Specifically, the electronic device (100) controls at least one of a lidar sensor (110) and an RGB sensor (140) to sense the surrounding space, and can obtain information on multiple projectible areas within the surrounding space based on the surrounding space.

[0280] The electronic device (100) can control the projection unit (170) to project an image onto a projectible area acquired based on the surrounding space.

[0281] Meanwhile, the electronic device (100) may include a movable member (180).

[0282] The electronic device (100) may include a movable member (180) at the lower portion. The movable member (180) is a component for moving the electronic device (100). To this end, the movable member (180) includes a motor, wheels, etc., and the electronic device (100) may be moved through the movement of the wheels.

[0283] When an event occurs that requires the electronic device (100) to move to a specific location within a space, the electronic device (100) checks map information about the space from the memory (120) and then sets a movement path to a target location based on the current location. For example, if the electronic device (100) determines that there is no obstacle on the straight path between the current location and the target location, the electronic device (100) can determine a straight movement path. The electronic device (100) can control the moving member (180) to move the main body of the electronic device (100) along the determined movement path. If the electronic device (100) determines that there is an obstacle on the straight path between the current location and the target location, the electronic device (100) can determine an avoidance path to avoid the obstacle. The electronic device (100) can identify the location of each obstacle within the space based on the map information and set an avoidance path. Thereafter, the electronic device (100) can control the driving unit (not shown) and the moving member (180) to move the main body of the electronic device (100) along the avoidance path.

[0284] As shown in FIG. 4, the moving member (180) is depicted as a wheel, but it may be implemented in the form of a cat filter when implemented, and when the electronic device (100) is implemented as a drone or the like, the moving member (180) may also be implemented as a propeller or the like.

[0285] Meanwhile, although the illustrated example shows the electronic device (100) as having a movable member (180), the movable member (180) may be a separate device. For example, the electronic device (100) may be combined with a movable device such as a robot vacuum cleaner, and may be mounted on the robot vacuum cleaner to control the movement of the robot vacuum cleaner and operate.

[0286] Meanwhile, the movable member (180) can adjust the projection direction of the electronic device (100). For example, the direction in which the projection device looks can be adjusted by adjusting the body position of the electronic device (100), or the projection shape can be adjusted by adjusting the position of a lens or mirror within the projection device.

[0287] Here, the projection direction refers to the direction in which the image in the form of a projection is projected, and may also be referred to as the direction in which the electronic device (100) is facing, the projection direction, the projection direction, etc. In the following, for the sake of ease of explanation, the projection direction is expressed as being changed, but it may also be expressed as being changed in the projection area. In this case, the change in the projection area does not mean a change in the screen size while maintaining the center point of the projection area, but a case in which the center point of the projection area is changed.

[0288] However, the configuration of the movable member (180) is only a part of various embodiments, and the configuration of the movable member (180) may be omitted. For example, the electronic device (100) may be a movable projector directly equipped with the movable member (180), or a movable projector that must be carried and moved by the user without the movable member (180).

[0289] If the electronic device (100) does not have a movable member (180), it can sense the surrounding space at a location placed by the user to obtain sensing data, and project an image onto a projectible area obtained based on the obtained sensing data.

[0290] Meanwhile, the electronic device (100) may include some additional configurations not shown in FIG. 13.

[0291] For example, the electronic device (100) may include a microphone for receiving a user's voice. The microphone may transmit the received user's voice to the electronic device (100). Subsequently, the electronic device (100) may input the received user's voice into a voice recognition model to perform voice recognition. For example, the electronic device (100) may perform STT (Speech to Text) on the user's voice to perform voice recognition on the user's voice. The microphone may receive the user's voice and transmit the received user's voice to the electronic device (100). Subsequently, the electronic device (100) may input the received user's voice into a voice recognition model to perform voice recognition. For example, the electronic device (100) may perform STT (Speech to Text) on the user's voice to perform voice recognition on the user's voice.

[0292] According to one embodiment of the present disclosure, when an electronic device (100) receives a user voice through a microphone to move to a new space, the electronic device (100) can control a driving unit (not shown) and a moving member (180) to move to a position where the space can be sensed. However, the present disclosure is not limited thereto.

[0293] For example, in addition to cases where the electronic device (100) is equipped with a microphone, the user's voice may be input as an analog voice signal into the microphone of an external device, such as a remote control. The remote control or the like may also digitize the analog voice signal and transmit it to the electronic device (100). However, this is not a limitation.

[0294] For example, based on the user's voice input via a microphone installed in the smartphone, the user can remotely control the electronic device (100) via the smartphone. Specifically, the smartphone can perform voice recognition functions via an installed remote control application. However, the external device that performs these voice recognition and remote control functions is not limited to the smartphone, and the same functions can also be performed via an AI speaker or other electronic devices on which applications can be installed.

[0295] FIG. 14 is a flowchart for explaining a method for controlling an electronic device according to at least one embodiment of the present disclosure.

[0296] According to FIG. 14, the electronic device (100) can acquire first sensing data through the lidar sensor by controlling the lidar sensor to sense space in a sensing direction corresponding to a preset angle according to a preset event (S1410). The preset event and preset angle have been described in the various embodiments described above, and thus, a redundant description thereof will be omitted.

[0297] The sensing direction of the lidar sensor can be sequentially changed to acquire multiple sets of second sensing data through the lidar sensor (S1420). Since the sensing direction of the lidar sensor and other details have been described in the various embodiments described above, a detailed explanation will be omitted.

[0298] A depth map including distance information of a space can be acquired based on first sensing data and a plurality of second sensing data (S1430). Distance information of an object area can be identified based on the first sensing data, a plurality of second sensing data, and third sensing data acquired by an RGB sensor. The operation of updating the distance information included in the depth map based on the distance information of the object area has been described in the various embodiments described above, so a redundant description thereof will be omitted.

[0299] The control method of Fig. 14 can be performed by an electronic device having a configuration similar to that of Fig. 2 or Fig. 11, but is not necessarily limited thereto, and may be performed by a device having a different configuration.

[0300] Additionally, the various embodiments described above may be implemented independently of each other, or may be implemented in whole or in part in combination with various other embodiments of the present disclosure.

[0301] Meanwhile, the methods according to the various embodiments of the present disclosure described above can be implemented only with a software upgrade or a hardware upgrade for an existing electronic device.

[0302] Additionally, the various embodiments of the present disclosure described above can also be performed through an embedded server provided in an electronic device, or an external server of the electronic device.

[0303] Meanwhile, according to the exemplary embodiments of the present disclosure, the various embodiments described above may be implemented as software containing instructions stored on a machine-readable storage medium (e.g., a computer). When such software or program is executed by an electronic device, the electronic device may execute various control methods as described in the various embodiments described above.

[0304] Such software or programs may be used while stored on a non-transitory computer-readable medium. Here, "non-transitory" means that the storage medium does not contain signals and is tangible, but does not distinguish between whether the data is stored semi-permanently or temporarily on the storage medium.

[0305] Additionally, according to one or more embodiments of the present disclosure, the methods according to the various embodiments described above may be provided as included in a computer program product. The computer program product may be traded as a commodity between sellers and buyers. The computer program product may be distributed online through an online application store or in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0306] In addition, each of the components (e.g., modules or programs) according to the various embodiments described above may be composed of a single or multiple entities, and some of the corresponding sub-components described above may be omitted, or other sub-components may be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity, which may perform the same or similar functions as those performed by each of the corresponding components prior to integration. Operations performed by modules, programs or other components according to various embodiments may be executed sequentially, in parallel, iteratively or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0307] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person having ordinary skill in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. In electronic devices, lidar sensor; comprising at least one processor; At least one processor, Based on the occurrence of a preset event, the lidar sensor is controlled to sense space in a sensing direction of the lidar sensor corresponding to a preset angle so that the lidar sensor acquires first sensing data. After controlling the lidar sensor to sense the space in the sensing direction of the lidar sensor corresponding to the preset angle, the direction of the lidar sensor is controlled so that the sensing direction of the lidar sensor is sequentially changed so that the lidar sensor acquires a plurality of second sensing data corresponding to the sequentially changed sensing direction, An electronic device that obtains a depth map including distance information of the space based on the first sensing data and the plurality of second sensing data.

2. In paragraph 1, An electronic device wherein the above preset angle is an angle parallel to the ground.

3. In paragraph 2, An electronic device wherein the above-described preset event is activation of the electronic device.

4. In paragraph 1, It further includes an RGB sensor; At least one processor, Controlling the RGB sensor to obtain third sensing data about the surroundings of the electronic device, Identifying multiple object areas based on the third sensing data, Identifying distance information of the plurality of object areas based on distance information included in the depth map, An electronic device that updates distance information included in the depth map based on distance information of the plurality of object areas identified above.

5. In paragraph 4, At least one processor, Identifying an area corresponding to each of the plurality of object areas in the depth map, An electronic device that updates an area corresponding to each of the plurality of object areas in the depth map with the same distance information.

6. In paragraph 4, At least one processor, Obtain information about a vanishing point identified based on the above plurality of object areas, An electronic device that updates distance information included in the depth map based on information about the acquired vanishing point.

7. In paragraph 4, At least one processor, Processing the first sensing data, the plurality of second sensing data, and the third sensing data acquired through the lidar sensor to have matching view points, An electronic device that updates distance information included in the depth map based on the first sensing data, the plurality of second sensing data, and the third sensing data, while the viewpoints are matched.

8. In paragraph 1, At least one processor, Obtain vanishing point information based on the first sensing data and the plurality of second sensing data, An electronic device that updates distance information included in the depth map based on information about the vanishing point.

9. In paragraph 1, At least one processor, An electronic device that controls the lidar sensor so that the sensing direction of the lidar sensor sequentially changes in the up-down direction while the left-right sensing direction range of the lidar sensor falls within a preset angular range.

10. A method for controlling an electronic device including a lidar sensor, A step of controlling the lidar sensor so that the lidar sensor acquires first sensing data by sensing space in a sensing direction of the lidar sensor corresponding to a preset angle based on the occurrence of a preset event; A step of controlling the direction of the lidar sensor so that the sensing direction of the lidar sensor is sequentially changed so that the lidar sensor acquires a plurality of second sensing data corresponding to the sequentially changed sensing direction, after controlling the lidar sensor to sense the space in the sensing direction of the lidar sensor corresponding to the preset angle; and A control method, comprising: a step of obtaining a depth map including distance information of the space based on the first sensing data and the plurality of second sensing data.

11. In paragraph 10, A control method wherein the above preset angle is an angle parallel to the ground.

12. In paragraph 11, A control method wherein the above-described preset event is activation of the electronic device.

13. In paragraph 10, The electronic device further comprises an RGB sensor; A step of controlling the RGB sensor to obtain third sensing data about the surroundings of the electronic device; A step of identifying a plurality of object areas based on the third sensing data; A step of identifying distance information of the plurality of object areas based on distance information included in the depth map; and A control method further comprising: a step of updating distance information included in the depth map based on distance information of the identified object area.

14. In paragraph 13, A step of identifying an area corresponding to each of the plurality of object areas in the depth map; and A control method further comprising: a step of updating an area corresponding to each of the plurality of object areas in the depth map with the same distance information.

15. A non-transitory computer-readable recording medium storing computer instructions that, when executed by a processor of an electronic device including a lidar sensor, cause the electronic device to perform an operation, wherein the operation is: A step of controlling the lidar sensor so that the lidar sensor acquires first sensing data by sensing space in a sensing direction of the lidar sensor corresponding to a preset angle based on the occurrence of a preset event; A step of controlling the direction of the lidar sensor so that the sensing direction of the lidar sensor is sequentially changed so that the lidar sensor acquires a plurality of second sensing data corresponding to the sequentially changed sensing direction, after controlling the lidar sensor to sense the space in the sensing direction of the lidar sensor corresponding to the preset angle; and A non-transitory computer-readable recording medium comprising: a step of obtaining a depth map including distance information of the space based on the first sensing data and the plurality of second sensing data.

Images