Electronic device and object measurement method thereof

Abstract

An electronic device and an object measurement method thereof are disclosed. According to an embodiment, an electronic device includes a display, a memory, and a processor operatively connected to the display and the memory. The memory stores instructions, wherein the instructions cause the processor to generate first depth information in a first direction of an object, generate a first point cloud of the object based on the first depth information, generate a first bounding box containing one or more points of the first point cloud, generate second depth information in a second direction of the object, wherein the second direction is different from the first direction, generate a second point cloud of the object based on the second depth information, generate a second bounding box containing one or more points of the second point cloud, generate a third bounding box by combining the first bounding box and the second bounding box, and display the third bounding box on the display. Certain other embodiments are also possible.

Description

Technical Field

[0001] The present disclosure relates generally to an electronic device, and more particularly, to an electronic device capable of measuring the size of an object using information of adjacent objects. Background Art

[0002] Using augmented reality technology, a 3D virtual image can be superimposed on a real-world background image captured by a camera, and the combined image can be provided to the user. Portable electronic devices such as smartphones (hereinafter referred to as "electronic devices") can be designed to provide users with various types of content based on augmented reality technology.

[0003] An exemplary application of augmented reality technology is the use of virtual images to provide information about real-world objects. One such type of information about real-world objects is the object's dimensions. For example, a user can obtain dimensional information about a cup using an electronic device that can capture an image of the cup, measure the cup's dimensions from the image, and display dimensional information about a box containing the cup. Summary of the Invention

[0004] Conventionally, object size measurement is achieved by measuring the distance between user-specified points on a captured real-world image. For example, when an electronic device displays a preview image from its camera, the user can sequentially select the top and bottom endpoints of an object to measure the distance between the two selected points. The object's size can then be determined based on the measurement results.

[0005] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the embodiments.

[0006] According to a disclosed embodiment, an electronic device includes a display, a memory, and a processor operably connected to the display and the memory. The memory stores instructions that, when executed by the processor, cause the processor to perform the following operations: generate first depth information in a first direction of an object; generate a first point cloud of the object based on the first depth information; generate a first bounding box containing one or more points of the first point cloud; generate second depth information in a second direction of the object, wherein the second direction is different from the first direction; generate a second point cloud of the object based on the second depth information; generate a second bounding box containing one or more points of the second point cloud; generate a third bounding box by combining the first bounding box and the second bounding box; and display the third bounding box on the display.

[0007] According to a disclosed embodiment, a method for measuring an object of an electronic device includes: generating first depth information in a first direction of the object; generating a first point cloud of the object based on the first depth information; generating a first bounding box including one or more points of the first point cloud; generating second depth information in a second direction of the object, wherein the second direction is different from the first direction; generating a second point cloud of the object based on the second depth information; generating a second bounding box including one or more points of the second point cloud; generating a third bounding box by combining the first bounding box and the second bounding box; and displaying the third bounding box. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In the drawings, the same or similar reference numerals may be used for the same or similar components.

[0009] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0010] Figure 1 is a block diagram illustrating a configuration of an electronic device in a network environment according to a disclosed embodiment;

[0011] Figure 2 is a diagram illustrating an electronic device and an object according to a disclosed embodiment;

[0012] Figure 3 is a block diagram illustrating an electronic device according to a disclosed embodiment;

[0013] Figure 4 is a block diagram illustrating hardware and software for generating 3-dimensional coordinates of an object of an electronic device according to a disclosed embodiment;

[0014] Figure 5 is a diagram illustrating exemplary screen displays for explaining a process in which an electronic device guides aiming at an object to be measured according to a disclosed embodiment;

[0015] Figure 6 is a diagram illustrating an exemplary screen display for explaining a process of generating a point cloud of an object by an electronic device according to a disclosed embodiment;

[0016] Figure 7A and Figure 7B is a diagram illustrating an exemplary screen display for explaining a process of removing a reference surface from a point cloud by an electronic device according to a disclosed embodiment;

[0017] Figure 8 is a diagram illustrating exemplary screen displays for explaining a process of generating a 3-dimensional bounding box by an electronic device according to a disclosed embodiment;

[0018] 9A to 9Dis a diagram illustrating exemplary screen displays for explaining a process in which an electronic device displays guide information related to object measurement according to a disclosed embodiment;

[0019] Figure 10 is a diagram illustrating an exemplary screen display for interpreting a point cloud of an object generated by an electronic device according to a disclosed embodiment;

[0020] Figure 11 is a diagram illustrating exemplary screen displays for explaining a process in which an electronic device generates two bounding boxes for an object according to a disclosed embodiment;

[0021] Figure 12 is a diagram illustrating exemplary screen displays for explaining a process of updating a bounding box of an electronic device according to a disclosed embodiment;

[0022] Figure 13A and Figure 13B is a diagram illustrating exemplary screen displays for explaining a process of an electronic device generating a bounding box of a 2-dimensional object according to a disclosed embodiment; and

[0023] Figure 14 is a flowchart illustrating an object measurement method of an electronic device according to a disclosed embodiment. DETAILED DESCRIPTION

[0024] Certain disclosed embodiments are directed to providing an electronic device and an object measurement method thereof that can automatically and accurately measure the size of a real-world object.

[0025] Figure 1 1 is a block diagram illustrating an electronic device 101 in a network environment 100 according to various embodiments. Figure 1, the electronic device 101 in the network environment 100 can communicate with the electronic device 102 via the first network 198 (e.g., a short-range wireless communication network), or communicate with the electronic device 104 or the server 108 via the second network 199 (e.g., a long-range wireless communication network). According to an embodiment, the electronic device 101 can communicate with the electronic device 104 via the server 108. According to an embodiment, the electronic device 101 may include a processor 120, a memory 130, an input device 150, a sound output device 155, a display device 160, an audio module 170, a sensor module 176, an interface 177, a connection terminal 178, a haptic module 179, a camera module 180, a power management module 188, a battery 189, a communication module 190, a subscriber identification module (SIM) 196, or an antenna module 197. In some embodiments, at least one of the components (e.g., the display device 160 or the camera module 180) may be omitted from the electronic device 101, or one or more other components may be added to the electronic device 101. In some embodiments, some of the components may be implemented as a single integrated circuit. For example, the sensor module 176 (eg, a fingerprint sensor, an iris sensor, or an illumination sensor) may be implemented as embedded in the display device 160 (eg, a display).

[0026] The processor 120 may run, for example, software (e.g., program 140) to control at least one other component of the electronic device 101 connected to the processor 120 (e.g., a hardware component or a software component), and may perform various data processing or calculations. According to one embodiment, as at least part of the data processing or calculation, the processor 120 may load commands or data received from another component (e.g., sensor module 176 or communication module 190) into the volatile memory 132, process the commands or data stored in the volatile memory 132, and store the resulting data in the non-volatile memory 134. Depending on the embodiment, the processor 120 may include a main processor 121 (e.g., a central processing unit (CPU) or an application processor (AP)) and an auxiliary processor 123 (e.g., a graphics processing unit (GPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is operationally independent of or combined with the main processor 121. Additionally or alternatively, the auxiliary processor 123 may be adapted to consume less power than the main processor 121, or adapted to be specifically used for a designated function. The auxiliary processor 123 may be implemented separately from the main processor 121 or as part of the main processor 121 .

[0027] When the main processor 121 is in an inactive (e.g., sleep) state, the auxiliary processor 123 (rather than the main processor 121) may control at least some of the functions or states related to at least one component (e.g., the display device 160, the sensor module 176, or the communication module 190) among the components of the electronic device 101, or when the main processor 121 is in an active state (e.g., running an application), the auxiliary processor 123 may control at least some of the functions or states related to at least one component (e.g., the display device 160, the sensor module 176, or the communication module 190) together with the main processor 121. Depending on the embodiment, the auxiliary processor 123 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., the camera module 180 or the communication module 190) that is functionally related to the auxiliary processor 123.

[0028] The memory 130 may store various data used by at least one component of the electronic device 101 (e.g., the processor 120 or the sensor module 176). The various data may include, for example, software (e.g., the program 140) and input data or output data for commands related thereto. The memory 130 may include a volatile memory 132 or a non-volatile memory 134.

[0029] The program 140 may be stored as software in the memory 130 , and may include, for example, an operating system (OS) 142 , middleware 144 , or applications 146 .

[0030] The input device 150 may receive commands or data from outside the electronic device 101 (e.g., a user) to be used by other components of the electronic device 101 (e.g., the processor 120). The input device 150 may include, for example, a microphone, a mouse, a keyboard, or a digital pen (e.g., a stylus).

[0031] The sound output device 155 can output sound signals to the outside of the electronic device 101. The sound output device 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as playing multimedia or records, and the receiver can be used for incoming calls. Depending on the embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0032] The display device 160 can visually provide information to the outside of the electronic device 101 (e.g., a user). The display device 160 may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling a corresponding one of the display, the holographic device, and the projector. Depending on the embodiment, the display device 160 may include a touch circuit adapted to detect a touch or a sensor circuit adapted to measure the strength of the force caused by the touch (e.g., a pressure sensor).

[0033] The audio module 170 can convert sound into an electrical signal, and vice versa. According to an embodiment, the audio module 170 can obtain sound via the input device 150, or output sound via the sound output device 155 or an earphone of an external electronic device (e.g., electronic device 102) directly (e.g., wired) or wirelessly connected to the electronic device 101.

[0034] The sensor module 176 can detect an operating state (e.g., power or temperature) of the electronic device 101 or an environmental state (e.g., a user's state) outside the electronic device 101, and then generate an electrical signal or data value corresponding to the detected state. Depending on the embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illumination sensor.

[0035] The interface 177 may support one or more specific protocols to be used to connect the electronic device 101 directly (e.g., wired) or wirelessly to an external electronic device (e.g., the electronic device 102). Depending on the embodiment, the interface 177 may include, for example, a High-Definition Multimedia Interface (HDMI), a Universal Serial Bus (USB) interface, a Secure Digital (SD) card interface, or an audio interface.

[0036] The connection end 178 may include a connector, wherein the electronic device 101 can be physically connected to an external electronic device (e.g., the electronic device 102) via the connector. Depending on the embodiment, the connection end 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0037] The haptic module 179 may convert the electrical signal into mechanical stimulation (eg, vibration or motion) or electrical stimulation that can be recognized by the user via his sense of touch or kinesthetic sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.

[0038] The camera module 180 may capture still images or moving images. Depending on the embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

[0039] The power management module 188 may manage power supply to the electronic device 101. According to an embodiment, the power management module 188 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0040] The battery 189 may power at least one component of the electronic device 101. According to an embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0041] The communication module 190 may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (e.g., electronic device 102, electronic device 104, or server 108), and perform communication via the established communication channel. The communication module 190 may include one or more communication processors capable of operating independently from the processor 120 (e.g., an application processor (AP)) and supporting direct (e.g., wired) communication or wireless communication. Depending on the embodiment, the communication module 190 may include a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules can communicate with an external electronic device via a first network 198 (e.g., a short-range communication network such as Bluetooth, Wireless Fidelity (Wi-Fi) Direct, or Infrared Data Association (IrDA)) or a second network 199 (e.g., a long-range communication network such as a cellular network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))). These various types of communication modules can be implemented as a single component (e.g., a single chip), or these various types of communication modules can be implemented as multiple components separated from each other (e.g., multiple chips). The wireless communication module 192 can identify and authenticate the electronic device 101 in a communication network (such as the first network 198 or the second network 199) using user information (e.g., an International Mobile Subscriber Identity (IMSI)) stored in the user identification module 196.

[0042] Antenna module 197 can transmit or receive signals or power to or from the outside of electronic device 101 (e.g., an external electronic device). Depending on the embodiment, antenna module 197 may include an antenna comprising a radiating element formed of a conductive material or conductive pattern formed in or on a substrate (e.g., a PCB). Depending on the embodiment, antenna module 197 may include multiple antennas. In this case, at least one antenna suitable for the communication scheme used in a communication network (such as first network 198 or second network 199) may be selected from the multiple antennas by, for example, communication module 190 (e.g., wireless communication module 192). Signals or power can then be transmitted or received between communication module 190 and the external electronic device via the selected at least one antenna. Depending on the embodiment, additional components other than the radiating element (e.g., a radio frequency integrated circuit (RFIC)) may also be formed as part of antenna module 197.

[0043] At least some of the above components can be connected to each other via an inter-peripheral communication scheme (e.g., a bus, general-purpose input output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and communicatively transmit signals (e.g., commands or data) therebetween.

[0044] According to an embodiment, commands or data can be transmitted or received between the electronic device 101 and the external electronic device 104 via the server 108 coupled with the second network 199. Each of the electronic devices 102 and 104 can be the same type as or different from the electronic device 101. According to an embodiment, all or some of the operations to be executed by the electronic device 101 can be executed at one or more of the external electronic devices 102, 104, or server 108. For example, if the electronic device 101 is to automatically perform a function or a service or is to request a function or a service in response to a request from a user or another device, the electronic device 101, instead of or in addition to executing the function or the service, can request at least some of the function or the service to be executed by the one or more external electronic devices. The one or more external electronic devices receiving the request can execute at least some of the requested function or service, or perform other functions or other services related to the request, and transfer an outcome of the execution to the electronic device 101. The electronic device 101 can provide the outcome, with or without further processing of the outcome, as at least part of a reply to the request. To that end, a cloud computing, distributed computing, or client-server computing technology can be used, for example.

[0045] An electronic device according to various embodiments can be one of various types of electronic devices. The electronic devices can include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. According to an embodiment of the disclosure, the electronic devices are not limited to those described above.

[0046] It should be understood that the various embodiments of the present disclosure and the terms used therein are not intended to limit the technical features set forth herein to specific embodiments, but rather include various changes, equivalents or alternative forms for corresponding embodiments. For the description of the accompanying drawings, similar reference numerals may be used to refer to similar or related elements. It will be understood that the nouns in the singular form corresponding to the term may include one or more things, unless the relevant context clearly indicates otherwise. As used herein, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C" and "at least one of A, B or C" may include any one or all possible combinations of the items listed together with the corresponding phrase in the multiple phrases. As used herein, terms such as "1st" and "2nd" or "first" and "second" may be used to simply distinguish corresponding components from another component, and do not limit the components in other aspects (e.g., importance or order). It will be understood that if an element (e.g., a first element) is referred to as being “combined with another element (e.g., a second element)”, “combined to another element (e.g., a second element)”, “connected with another element (e.g., a second element)”, or “connected to another element (e.g., a second element)”, when the term “operably” or “communicatively” is used or when the term “operably” or “communicatively” is not used, it means that the element can be directly (e.g., wired) connected to the other element, wirelessly connected to the other element, or connected to the other element via a third element.

[0047] As used herein, the term "module" may include units implemented in hardware, software, or firmware, and may be used interchangeably with other terms (e.g., "logic," "logic block," "portion," or "circuit"). A module may be a single integrated component adapted to perform one or more functions or the smallest unit or portion of the single integrated component. For example, depending on an embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0048] The various embodiments described herein can be implemented as software (e.g., program 140) comprising one or more instructions stored in a storage medium (e.g., internal memory 136 or external memory 138) that can be read by a machine (e.g., electronic device 101). For example, under the control of a processor, a processor (e.g., processor 120) of the machine (e.g., electronic device 101) can call at least one of the one or more instructions stored in the storage medium and execute the at least one instruction with or without the use of one or more other components. This enables the machine to be operable to perform at least one function according to the called at least one instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The machine-readable storage medium can be provided in the form of a non-transitory storage medium. The term "non-transitory" only means that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but the term does not distinguish between data being semi-permanently stored in the storage medium and data being temporarily stored in the storage medium.

[0049] According to an embodiment, the method according to various embodiments of the present disclosure may be included and provided in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be published in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)), or may be published online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™), or may be distributed (e.g., downloaded or uploaded) directly between two user devices (e.g., smart phones). If published online, at least a portion of the computer program product may be temporarily generated, or at least a portion of the computer program product may be at least temporarily stored in a machine-readable storage medium (such as a manufacturer's server, an application store's server, or a memory of a forwarding server).

[0050] According to various embodiments, each component (for example, module or program) in the above-mentioned components may include a single entity or multiple entities. According to various embodiments, one or more components in the above-mentioned components may be omitted, or one or more other components may be added. Alternatively or additionally, multiple components (for example, module or program) may be integrated into a single component. In this case, according to various embodiments, the integrated component may still perform the one or more functions of each component in the multiple components in the same or similar manner as a corresponding component in the multiple components before integration. According to various embodiments, the operations performed by module, program or another component may be performed sequentially, in parallel, repeatedly or in a heuristic manner, or one or more operations in the operations may be run or omitted in different orders, or one or more other operations may be added.

[0051] Figure 2 is a diagram illustrating an electronic device and an object according to a disclosed embodiment.

[0052] According to an embodiment, the electronic device 210 may be, but is not limited to, a portable electronic device such as a smartphone and a tablet personal computer (PC). The electronic device 210 may include at least one sensor (e.g., a time-of-flight (ToF) sensor 211 and a red, green, and blue (RGB) image sensor 212) for capturing an image within a predetermined viewing angle and creating depth information (or a depth map) of an object 220 included in the captured image.

[0053] Reference Figures 3 to 14 A method in which the electronic device 210 generates depth information of the object 220 using the ToF sensor 211 and / or the RGB image sensor 212 is described in detail.

[0054] The object 220 for which depth information can be obtained is not affected by Figure 2 The embodiments shown are limited. Figure 2 For the example of capturing an image of the cup 221 placed on the table 222 , the electronic device 210 may also capture an image of another object such as a human body to obtain depth information of the object.

[0055] Although the description in the present disclosure is directed to an exemplary case of obtaining depth information of the cup 221 in a captured image and automatically displaying the bounding box and size of the cup 221 automatically measured based on the depth information (e.g., as Figures 5 to 12 ), but the methods of certain disclosed embodiments can be used to measure the dimensions of various types of 3-dimensional or 2-dimensional objects (e.g., a picture drawn on a flat surface).

[0056] The electronic device 210 can create a point cloud based on the depth information of the object 220 in the photographed image, and continue to create a bounding box of the point cloud.

[0057] According to an embodiment, the electronic device 210 can create a plurality of point clouds (e.g., a first point cloud and a second point cloud) based on depth information (e.g., first depth information and second depth information) obtained at two or more positions (e.g., a first direction and a second direction), create a plurality of bounding boxes (e.g., a first bounding box to a third bounding box) from the point clouds, and display the bounding boxes.

[0058] After displaying the created bounding boxes, if it is determined that the bounding boxes are not accurate, the electronic device 210 can display guide information prompting the user to move the electronic device 210 to photograph additional images (or depth information) of the corresponding object 220 at different positions.

[0059] Figure 3 FIG. 1 is a block diagram illustrating an electronic device according to an embodiment of the disclosure.

[0060] Referring to FIG. 1, Figure 3 The electronic device 300 can include a ToF sensor 310, an RGB sensor 320, a display 330, a processor 340, and a memory 350, at least one of which can be omitted or replaced. The electronic device 300 can include Figure 1 at least some of the configuration and / or functions of the electronic device.

[0061] The components constituting the electronic device 300 are contained in a case, and some components that emit light (e.g., the display 330) or receive light (e.g., the ToF sensor 310) from the outside of the case can be partially exposed to the outside of the case.

[0062] According to an embodiment, the display 330 can display an image and can be implemented with at least one of various types of display devices such as a liquid crystal display (LCD) or a light emitting diode (LED) display. The display 330 can include Figure 1 at least some of the configuration and / or functions of the display device 160. The display 330 can be exposed through an opening formed in one surface (e.g., a front surface) of the case.

[0063] The display 330 can be a touch screen display capable of sensing a touch or hovering (i.e., proximity touch) gesture made by one or more fingers or a stylus pen of a user.

[0064] According to an embodiment, the ToF sensor 310 can obtain depth information of an object using ToF technology. For example, the ToF sensor 310 can emit infrared light and calculate the arrival time of the light reflected from the object to measure the distance from the ToF sensor 310 to the portion of the object that reflects the light. Based on the distance measured by the ToF sensor 310, the depth information of the object can be generated.

[0065] The ToF sensor 310 may generate depth information from a plurality of image frames acquired from a plurality of positions in a plurality of directions using ToF technology.

[0066] According to an embodiment, the RGB image sensor 320 may capture an image of a subject and generate image data including red (R), green (G), and blue (G) data. The RGB image sensor 320 may acquire the image data using one of various technologies such as a charge coupled device (CCD) and a complementary metal oxide semiconductor (CMOS).

[0067] According to an embodiment, the processor 340 may generate depth information of the object based on two pieces of image data of the object acquired by the RGB image sensor in different directions.

[0068] According to an embodiment, the electronic device 300 may include at least one of the ToF sensor 310 and the RGB image sensor 320 and / or may include an additional sensor capable of generating depth information of an object in addition to the ToF sensor 310 and the RGB image sensor 320 .

[0069] According to an embodiment, the memory may include, but is not limited to, volatile memory and non-volatile memory. The memory 350 may include Figure 1 The memory 350 may also store at least some of the configuration and / or functionality of the memory. Figure 1 At least some of the programs in the program.

[0070] Memory 350 may store various instructions that may be executed by processor 340. These instructions may include arithmetic and logical operations, instructions for manipulating data, and input / output control commands.

[0071] According to an embodiment, the processor 340 may be configured to control components of the electronic device 300 to perform communication-related operations and data processing operations, and may include Figure 1The processor 340 may be functionally, operatively, and / or electronically connected to the components of the electronic device 300, namely, the ToF sensor 310, the RGB image sensor 320, the display 330, and the memory 350. The processor 340 may include a microprocessor or any suitable type of processing circuit, such as one or more general-purpose processors (e.g., ARM-based processors), digital signal processors (DSPs), programmable logic devices (PLDs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), graphics processing units (GPUs), video card controllers, and the like. Furthermore, it will be appreciated that when a general-purpose computer accesses code for implementing the processes described herein, the execution of the code transforms the general-purpose computer into a special-purpose computer for performing the processes described herein. The specific functions and steps provided in the figures may be implemented in hardware, software, or a combination of both, and may be executed in whole or in part within the programming instructions of a computer. Furthermore, it will be understood and appreciated by those skilled in the art that a "processor" or "microprocessor" may be hardware in the claimed disclosure.

[0072] Processor 340 is not limited to performing arithmetic operations and data processing functions of electronic device 300. Certain disclosed embodiments are directed to a method for automatically and accurately measuring the size of a real-world object and guiding a user to perform more accurate measurements. The operations of processor 340 described below may be performed after loading corresponding instructions stored in memory 350.

[0073] According to an embodiment, the processor 340 may generate first depth information in a first direction of the object.

[0074] According to an embodiment, the processor 340 may generate first depth information based on a first ToF frame captured by the ToF sensor 310 located in a first direction of the object. The ToF sensor 310 may measure the distance between points of each object located within the field of view, and the processor 340 may generate the first depth information based on the distance.

[0075] According to an alternative embodiment, the processor 340 may generate first depth information based on the difference between a third RGB frame acquired by the RGB image sensor 320 in the third direction of the object and a first RGB frame acquired by the RGB image sensor 320 in the first direction of the object. The first and third directions of the object may form a predetermined angle, so the first and third RGB frames may be image frames captured in different directions of the object. Therefore, the processor 340 may acquire first depth information of the object based on the difference between the first and third RGB frames acquired in different directions of the object.

[0076] According to an embodiment, the processor 340 can generate a first point cloud of the object based on the first depth information. The point cloud includes points captured on the surface of the 3-dimensional object. Each point constituting the point cloud can be identified by coordinate information with respect to a reference point.

[0077] According to an embodiment, the processor 340 can generate the first point cloud having only points above the reference surface. That is, in the generated point cloud, the processor 340 can remove points on the reference surface.

[0078] In a case where an object placed on a specific reference surface (e.g., a table and a floor) is measured, because the reference surface is included in the image frame (e.g., a ToF frame and an RGB image frame), the depth information and the point cloud of the reference surface can be included. The processor 340 can remove the point cloud of the reference surface to generate a bounding box for only the object.

[0079] According to an embodiment, the processor 340 can detect points having the same normal vector (or within an error range) among points constituting the point cloud. Then, the processor 340 can identify points adjacent to each other among the points having the same normal vector based on their coordinate information. According to an alternative embodiment, the processor 340 can determine a surface having the largest number of points having the same coordinate value in the z-axis (e.g., a z-axis in the Figure 2

[0080] By removing the reference surface as detected above, a first point cloud consisting of points corresponding to the object can be generated. In a case where there are a plurality of objects in the photographed image frame (e.g., a ToF frame and an RGB image frame), the objects can be separated by removing the reference surface. The processor 340 can generate a point cloud and a bounding box for the separated objects.

[0081] According to an embodiment, the processor 340 can generate a first bounding box based on the first point cloud. The bounding box is formed as a smallest cuboid into which the object can be put. For example, the processor 340 can generate the first bounding box based on minimum and maximum coordinate values obtained from coordinate information of each point of the first point cloud in x-axis, y-axis, and z-axis.

[0082] According to an embodiment, in a case where a plurality of point clouds are generated for a plurality of objects included in the photographed image frame (e.g., a ToF frame and an RGB image frame), the processor 340 can generate a bounding box for each object due to the removal of the reference surface.

[0083] According to an embodiment, the processor can display the first bounding box on the display 330. Here, the processor 340 can display the first bounding box such that the first bounding box is overlaid on an image of the object photographed by the RGB image sensor 320. In a case where there are a plurality of objects, the bounding box for each object is overlaid on the respective object. ​

[0084] Because the first bounding box generated above is generated based on only one set of depth information, it may be inaccurate. The low accuracy is due to the difficulty of measuring 3-D depth information from only the first direction. According to the disclosed embodiment, the electronic device 300 can use multiple sets of depth information obtained from different directions to more accurately calculate the bounding box and value.

[0085] According to an embodiment, the processor 340 may generate second depth information in a second direction of the object. Here, the second direction is different from the first direction and may form a predetermined angle with the first direction horizontally or vertically relative to the object.

[0086] The second depth information may be generated in the same manner as for the first depth information. For example, the processor 340 may generate the second depth information based on a second ToF frame acquired by the ToF sensor 310 in a second direction of the object. The processor 340 may also generate the second depth information based on a difference between a first RGB frame acquired by the RGB image sensor 320 in the first direction of the object and a second RGB frame acquired by the RGB image sensor 320 in the second direction of the object.

[0087] According to an embodiment, the processor 340 may generate a second point cloud of the object based on the second depth information. The processor 340 may generate a second point cloud having only points above the reference surface among the point clouds generated based on the second depth information.

[0088] According to an embodiment, processor 340 may generate a second bounding box based on the second point cloud. The second bounding box may be partially different from the first bounding box. For example, the second bounding box may have a size that is extended (or reduced) in the x-direction, y-direction, and z-direction relative to the first bounding box. Here, processor 340 may display the second bounding box so that it is overlaid on the image of the object captured by RGB image sensor 320.

[0089] According to an embodiment, the third bounding box may be generated by combining the first bounding box and the second bounding box. As described above, since the first bounding box and the second bounding box are measured in different directions of the object, their sizes are different.

[0090] The third bounding box may have a rectangular parallelepiped shape that can fit into the first and second bounding boxes. That is, the third bounding box may be generated based on the minimum and maximum values ​​of the x-axis, y-axis, and z-axis coordinate values ​​of the first and second bounding boxes.

[0091] According to an embodiment, the processor 340 may display the third bounding box on the display 330. The processor 340 may display the third bounding box such that the third bounding box is overlaid on the image of the object acquired by the RGB image sensor 320.

[0092] According to an embodiment, the processor 340 may display the third bounding box and digital information. The digital information may include the edge length and volume of the third bounding box. The processor 340 may generate the digital information based on the minimum and maximum coordinate values ​​of the point cloud contained in the third bounding box.

[0093] According to an embodiment, if it is determined that the first bounding box is inaccurate, the processor 340 may display, on the display 330 , guidance information prompting generation of another bounding box in a different direction.

[0094] According to an embodiment, if the first bounding box is partially different from the area including the actual object, the processor 340 may recognize the first bounding box as inaccurate and display guide information prompting to generate a second bounding box in a different direction.

[0095] According to an optional embodiment, if it is determined that the object extracted from the first RGB image is partially out of the viewing angle of the RGB image sensor 320 , the processor 340 may display guide information prompting a position shift of the electronic device 300 .

[0096] According to an optional embodiment, if the number of points constituting the first point cloud is less than a threshold value, the processor 340 may display guidance information prompting the user to shift the position of the electronic device 300 closer to the object.

[0097] According to an optional embodiment, if the number of first point clouds acquired from the first ToF frame photographed by the ToF sensor 310 is less than a threshold, the processor 340 may display guidance information prompting to acquire depth information using the RGB image sensor 320 .

[0098] Then refer to 9A to 9D Describes how to display boot information.

[0099] According to an embodiment, the electronic device 300 may generate a bounding box for a 2-dimensional object (eg, a rectangle and an ellipse) on a plane.

[0100] If any 3D object is not detected in the image frame captured by the ToF sensor 310 and / or the RGB image sensor 320 , or if the user selects a 2D object measurement option, the processor 340 may perform a process for acquiring depth information of the 2D object.

[0101] The processor 340 may generate a point cloud based on depth information of the 2-dimensional object, and continue to generate a point cloud mask by cutting out points forming a shape of the 2-dimensional object from the generated point cloud.

[0102] The processor 340 can calculate a minimum x-axis coordinate value and a minimum y-axis coordinate value of points of the point cloud mask and a maximum x-axis coordinate value and a maximum y-axis coordinate value, and generate a bounding box in the form of a 2-dimensional shape (e.g., a rectangle) based on the calculated coordinate values.

[0103] The processor 340 can display the generated bounding box on the display 330 so that the bounding box is overlaid on the image of the object together with digital information including a length of at least one side of the bounding box and / or an area of the boundary.

[0104] Figure 4 is a block diagram illustrating hardware and software of an electronic device for generating a 3-dimensional coordinate of an object according to an embodiment of the disclosure.

[0105] Referring to Figure 4 , the electronic device 400 can include the following hardware components: a processor 440, an RGB image sensor 420, a ToF sensor 410, an inertial measurement unit (IMU) sensor 460, a display 430, and a memory 450. The processor 440 can execute one or more processes or threads for generating a point cloud and / or a bounding box of an object.

[0106] According to an embodiment, if a user selects to execute an object measurement function, the electronic device 400 can display an image of an object photographed by the RGB image sensor 420, measurement information (e.g., a point cloud, a bounding box, and digital information), and guide information through a graphical user interface 441.

[0107] The ToF sensor 410 can generate and transmit at least one ToF frame of an object to the processor 440, and the processor 440 can include a depth map generator 442 that generates depth information based on the ToF frame. The electronic device 400 can include an additional sensor capable of generating depth information.

[0108] The processor 440 can include a camera pose relocalizer 443 that relocalizes a camera pose of the RGB image sensor 420.

[0109] The IMU sensor 460 can acquire various information related to a position and a motion of the electronic device 400, such as a rotation amount, acceleration, and position information of the electronic device 400, using at least one of a gyroscope, an accelerometer, and an electromagnetic sensor. The processor 440 can include an IMU noise filter 444 capable of filtering noise from information acquired by the IMU sensor 460.

[0110] The processor 440 can relocalize a camera pose by means of the camera pose relocalizer 443 using a sensing value of the IMU sensor 460. At this time, the processor 440 can reorganize image frames (e.g., ToF frames).

[0111] The processor 440 may include a local iterative closest point (ICP) module 446 capable of generating a point cloud based on the depth information generated by the depth map generator 442 and the camera pose relocated by the camera pose relocalizer 443 .

[0112] The local ICP module 446 may generate a bounding box for the object based on the point cloud.

[0113] The processor 440 may display the bounding box and numerical information (eg, length, area, and volume) of the bounding box on the display 430 using a graphical user interface, and may store the bounding box and the numerical information in the memory 450 .

[0114] Figure 5 is a diagram illustrating exemplary screen displays for explaining a process in which an electronic device guides aiming at an object to be measured according to a disclosed embodiment.

[0115] Reference Figure 5 , the user can operate the electronic device 500 to display the image generated by the RGB image sensor (eg, Figure 3 The processor ( Figure 3 The processor 340 in the image processing apparatus may display a sighting image 565 for guiding the object to be measured to be arranged at the center of the screen.

[0116] If the user adjusts the angle of the electronic device 500 so that the aiming image 565 is placed at the center of the object, the processor may automatically or in response to a touch input made by the user, with the aid of a ToF sensor (e.g., Figure 3 The ToF sensor 310 in the embodiment of the present invention acquires a first ToF frame. The first ToF frame may include information about a cup 571 as an object to be measured and a table 572 on which the cup 571 is placed.

[0117] The processor may also acquire an RGB image frame with the aid of an RGB image sensor.

[0118] The processor may acquire first depth information from the first ToF frame. The first depth information may include information about the depth of all objects (e.g., cup 571 and table 572). In this case, since the objects are in contact with each other, they may be recognized as a single object.

[0119] Figure 6 is a diagram illustrating exemplary screen displays for explaining a process of an electronic device generating a point cloud of an object according to a disclosed embodiment.

[0120] According to an embodiment, a processor (e.g., Figure 5The processor 340 in the image processing apparatus may generate a first point cloud 680 based on the first depth information and display the first point cloud 680 on the display 630.

[0121] The point cloud may include points constituting the surface of a 3D object. The points constituting the point cloud may each have coordinate information relative to a reference point.

[0122] Reference Figure 6 , the processor can be used in the image sensor (e.g., Figure 3 The points of the first point cloud 680 are displayed on an image captured by the RGB image sensor 320 in the image, so that the points are overlaid on the objects in the image. Here, the first point cloud 680 may include points measured on the cup 671 and the table 672 as objects in the screen.

[0123] Figure 7A and Figure 7B is a diagram illustrating exemplary screen displays for explaining a process of removing a reference surface from a point cloud by an electronic device according to a disclosed embodiment.

[0124] Reference Figure 7A , even if the object that the user intends to measure is the cup 771 , the first point cloud temporarily acquired in the process of measuring the size of the cup 771 placed on a reference surface such as a table may include information about both the cup 771 and the table.

[0125] According to an embodiment, a processor (e.g., Figure 3 The processor 340 in FIG. 7 may generate a first point cloud 781, wherein the first point cloud 881 includes only points above the reference surface in the provisional first point cloud generated based on the first depth information. In other words, the processor may remove points at or below the reference surface (e.g., a table) in the provisional first point cloud.

[0126] According to an embodiment, the processor may detect points having the same normal vector (or within an error range) among the points constituting the first point cloud 781, and identify adjacent points among the points having the same normal vector based on their coordinate information. Because the points corresponding to the table are all arranged on the same plane, their normal vectors may be perpendicular to the z-axis. The processor may identify adjacent points that have the same normal vector (or within an error range) as the reference surface.

[0127] According to an optional embodiment, the processor may determine a surface having a maximum number of points having the same z-axis coordinate value (or within a tolerance range) in the temporarily generated point cloud as the reference surface.

[0128] Reference Figure 7A , by Figure 6The reference surface is removed from the temporary point cloud shown in to obtain a point cloud 781 of the cup 771 .

[0129] Reference Figure 7B , in the case where a plurality of objects are included in a captured image frame (eg, a ToF frame and an RGB image frame), the point clouds 781a and 781b of the respective objects 771a and 771b may be separated from each other by removing a reference surface.

[0130] In this case, the processor may generate first point clouds 781a and 781b for the separated objects 771a and 771b and then proceed to generate and display a bounding box for each object.

[0131] Figure 8 is a diagram illustrating exemplary screen displays for explaining a process of generating a 3-dimensional bounding box by an electronic device according to a disclosed embodiment.

[0132] According to an embodiment, a processor (e.g., Figure 3 The processor 340 in the embodiment may be configured to be operable based on the first point cloud (e.g., Figure 7A The processor generates a first bounding box 890 based on the first point cloud 781 in the image. The bounding box may be generated in the shape of a minimum cuboid into which the object can fit. For example, the processor may generate the first bounding box 890 based on the minimum and maximum coordinate values ​​obtained from the coordinate information of each point in the first point cloud in the x-axis, y-axis, and z-axis.

[0133] The processor may display the first bounding box 890 as an overlay on the image of the object 871 .

[0134] The processor may display the first bounding box 890 and digital information 899. Here, the digital information 899 may include the edge length and volume of the first bounding box 890. The processor may generate the digital information 899 based on the minimum coordinate value and the maximum coordinate value of the point cloud included in the first bounding box 890.

[0135] Reference Figure 8 , by referring to Figures 5 to 8 The described process may produce a bounding box that is too small for object 871 to fit within it, i.e., the bounding box may be inaccurate. Measuring an object using only one ToF frame may produce a bounding box that is inaccurately sized relative to the actual object. Similarly, an inaccurate bounding box may be produced when measuring an object in an excessively dark or bright area or when using a portion of an image of the object (e.g., a ToF frame).

[0136] According to an embodiment, if it is determined that the bounding box generated as described above is inaccurate, the electronic device 800 may display guide information prompting the user to move the electronic device 800 to a different location to capture another image (or depth information) of the corresponding object.

[0137] Hereinafter, a process for displaying guide information is described with reference to 9A to 9D A process for displaying guide information is described.

[0138] 9A to 9D is a diagram illustrating exemplary screen displays of a process for explaining an electronic device to display guide information related to a measurement of an object according to an embodiment of the disclosure.

[0139] According to an embodiment, in a case where the first bounding box 990 generated as described above is different from an actual area of the object, the processor (e.g., the processor 340 in Figure 3 may display guide information 935 prompting to generate a bounding box again in a different direction.

[0140] Referring to Figure 9A , the first bounding box 990 overlaid on an image of the object 971 captured by the RGB image sensor (e.g., the RGB image sensor 320 in Figure 3 is smaller in size than an actual area of the object 971.

[0141] According to an embodiment, the processor can extract an area corresponding to the object 971 in a first RGB image captured by the RGB image sensor, determine that the first bounding box is overlaid on the area of the object 971 in the first RGB image, and compare a size of the overlapped portion with a threshold value. If the comparison result indicates that the size of the overlapped portion is smaller than the threshold value, it means that the first bounding box 990 is inaccurate; thus, the processor can display the guide information 935 on the display 930.

[0142] Referring to Figure 9B , the first bounding box can cover a portion of the object 971. This is because the object 971 is partially within a field of view of the ToF camera when the first ToF frame is acquired to generate the first depth information.

[0143] According to another embodiment, if it is determined that the area corresponding to the object 971 in the RGB image is partially outside a field of view of the RGB image sensor, the processor can display guide information 936 prompting to generate a bounding box again in a different direction.

[0144] Because the ToF sensor and the RGB image sensor are in the electronic device, the ToF sensor and the RGB image sensor can have the same field of view. Thus, the processor can determine whether the object is entirely within the field of view of the ToF sensor based on the RGB image acquired by the RGB image sensor. The processor can determine whether the object is partially outside the field of view based on pixel values at a vertical edge and a horizontal edge of the RGB image.

[0145] Referring to Figure 9C, because the electronic device 900 is too far away from the object 971, the first point cloud 982 generated as described above may be composed of a relatively small number of points.

[0146] According to an embodiment, if the number of points of the first point cloud 982 is equal to or less than a threshold value, the processor may display on the display 930 guide information 937 prompting to generate the bounding box again from a closer distance.

[0147] Reference Figure 9D When point cloud 983 of object 971 is generated when the ambient light is too dark or too bright, point cloud 983 may consist of a small number of points because the ToF sensor emits light in the infrared band. In this case, because the RGB image sensor will provide a more accurate point cloud, the processor may display guidance information 938 prompting the use of the RGB image sensor.

[0148] According to an embodiment, if the number of points in the first point cloud generated based on the first ToF frame is equal to or less than the threshold, the processor may display on the display guidance information prompting the use of the RGB image sensor to generate a bounding box. The processor may also check the ambient brightness based on the RGB image frame captured by the RGB image sensor, and if it is determined that the ambient brightness is too bright or too dark, display on the display guidance information 938 prompting the use of the RGB image sensor to generate a bounding box.

[0149] Figure 10 is a diagram illustrating an exemplary screen display for interpreting a point cloud of an object generated by an electronic device according to a disclosed embodiment.

[0150] Before displaying the first bounding box (e.g. Figure 8 After the first bounding box 890 in the 9A to 9D After the guidance information described in the embodiment is obtained, the second depth information is obtained. For example, the user may move the electronic device 1000 to a new position (eg, a second position) according to the guidance information to capture an image of the object from the new position. Figures 10 to 12 In the embodiment of the present invention, it is assumed that the electronic device is horizontally moved around the object 1071 to measure the object 1071 in a second direction that forms a predetermined angle with the first direction. However, the moving direction is not limited to the disclosed embodiment, and the second direction may be a direction inclined upward or downward from the first direction or a direction in the opposite direction of the first direction.

[0151] According to an embodiment, the processor may generate second depth information in a second direction of the object.

[0152] The processor may generate a second point cloud 1080 based on the second depth information. The second point cloud 1080 may be generated by using the same process as that used for the first point cloud.

[0153] The processor may display the second point cloud 1080 such that points of the second point cloud 1080 are overlaid on the corresponding objects 1071 and 1072 .

[0154] Figure 11 is a diagram illustrating exemplary screen displays for explaining a process in which an electronic device generates two bounding boxes for an object according to a disclosed embodiment.

[0155] Processor (e.g. Figure 3 The processor 340 in the embodiment may be configured to obtain a point cloud from a second point cloud (e.g., Figure 10 The second point cloud 1080 in FIG. 11 is removed from the reference surface and generates a second bounding box 1192. The second bounding box 1192 can be generated in the same manner as for the first bounding box 1191.

[0156] Reference Figure 11 , the electronic device may display a first bounding box 1191 and a second bounding box 1192 on the display 1130. The first bounding box 1191 and the second bounding box 1192 may each have an overlapping portion overlapping with the object and a non-overlapping portion not overlapping with the object.

[0157] Figure 12 is a diagram illustrating exemplary screen displays for explaining a process of updating a bounding box of an electronic device according to a disclosed embodiment.

[0158] According to an embodiment, a processor (e.g., Figure 3 The processor 340 in FIG. 1 may convert the first bounding box (eg, Figure 11 1191) and a second bounding box (e.g., Figure 11 The second bounding box 1192 in is combined to generate a third bounding box 1293.

[0159] The third bounding box 1293 may be a rectangular parallelepiped having a size sufficient to fit the first and second bounding boxes. That is, the third bounding box 1293 may be generated based on the minimum and maximum values ​​of the x-axis, y-axis, and z-axis coordinate values ​​of the first and second bounding boxes.

[0160] The processor may display the third bounding box on the display 1230 such that the third bounding box is overlaid on the object 1271 .

[0161] According to an embodiment, the processor may display the third bounding box and digital information 1299 of the third bounding box 1293. Here, the digital information 1299 may include the edge length and volume of the third bounding box. The processor may generate the digital information 1299 based on the minimum and maximum coordinate values ​​of the points of the point cloud corresponding to the third bounding box 1293.

[0162] Figure 13A and Figure 13B is a diagram illustrating exemplary screen displays for explaining a process of an electronic device generating a bounding box of a 2-dimensional object according to a disclosed embodiment.

[0163] According to an embodiment, the electronic device 1300 (eg, Figure 3 The electronic device 300 in FIG. 3 may generate a bounding box for a 2-dimensional object (eg, a rectangle and an ellipse) on a plane.

[0164] Reference Figure 13A , a paper document 1371 may be placed on a table 1372. If you are trying to use a ToF sensor (e.g. Figure 3 ToF sensor 310 in ) and / or RGB image sensor (e.g., Figure 3 If no 3-dimensional object is detected after measuring a 3-D object with the RGB sensor 320 in FIG. 1 , or if the user selects a 2-dimensional object measurement option, the processor (e.g., Figure 3 The processor 340 in the may perform processing for generating a bounding box for a 2-dimensional object.

[0165] The processor may generate point clouds denoted by reference numerals 1381 and 1382 based on depth information acquired by means of the ToF sensor and / or the RGB image sensor, and display the point clouds 1381 and 1382 on the display 1330. The point clouds 1381 and 1382 may include all points corresponding to the paper document 1371 and the table 1372.

[0166] The processor may generate a point cloud mask having points corresponding to the paper document 1371 as a 2D object cut out from the generated point cloud. For example, the processor may detect an area having a predetermined shape using various edge extraction methods and generate a point cloud mask by cutting out the area.

[0167] The processor 340 may generate a 2-dimensional bounding box (eg, a rectangle) based on the minimum x-axis coordinate value and the minimum y-axis coordinate value and the maximum x-axis coordinate value and the maximum y-axis coordinate value of the points of the point cloud mask.

[0168] Reference Figure 13B , the processor may display the generated bounding box such that the bounding box is overlaid on the image of the paper document 1371 as a 2-dimensional object. According to an embodiment, the processor may display digital information 1399 including at least one of a side length and / or an area of ​​the bounding box of the paper document 1371 as a 2-dimensional object.

[0169] according to Figure 13A and Figure 13BIn the embodiment of the present invention, unlike conventional techniques, the electronic device 1300 can automatically calculate the size of a 2D object without any user intervention in setting reference points for measurement. The proposed method can accurately calculate the size of a bounding box even in the absence of some depth information.

[0170] According to an embodiment, the electronic device 300 may include a display 330, a memory 350, and a processor 340 operatively connected to the display 330 and the memory 350. The memory 350 may store instructions, wherein the instructions, when executed by the processor, cause the processor to perform the following operations: generate first depth information of an object in a first direction; generate a first point cloud of the object based on the first depth information; generate a first bounding box including one or more points of the first point cloud; generate second depth information of the object in a second direction, wherein the second direction is different from the first direction; generate a second point cloud of the object based on the second depth information; generate a second bounding box including one or more points of the second point cloud; generate a third bounding box by combining the first bounding box and the second bounding box; and display the third bounding box on the display 330.

[0171] According to an embodiment, the instructions may further cause the processor to display digital information including the volume of the third bounding box and the edge length of the third bounding box on the display 330 .

[0172] According to an embodiment, the instructions may further cause the processor to generate the digital information based on a minimum coordinate value and a maximum coordinate value of the points in the third bounding box.

[0173] According to an embodiment, the electronic device 300 may further include an RGB image sensor 320 , and the instructions may further cause the processor to display a third bounding box on an RGB image including the object captured by the RGB image sensor 320 .

[0174] According to an embodiment, the instructions may also cause the processor to perform the following operations: extract an object from a first RGB image captured by the RGB image sensor 320 in a first direction of the object; identify an overlapping area where the first bounding box overlaps with an area of ​​the object extracted from the first RGB image; and based on a size of the overlapping area being equal to or less than a threshold, display on the display 330 guiding information prompting acquisition of second depth information in a second direction.

[0175] According to an embodiment, the instructions may further cause the processor to display on the display 330 guide information prompting acquisition of the second depth information in the second direction, wherein the display of the guide information may be based on at least a portion of the object being outside the field of view of the RGB image sensor.

[0176] According to an embodiment, the instructions may further cause the processor to display, on the display 330 , guidance information prompting the user to acquire the second depth information at a position closer to the object based on the number of points in the first point cloud being equal to or less than a threshold.

[0177] According to an embodiment, the instructions may further cause the processor to detect a reference surface in a first point cloud generated based on the first depth information, wherein the first point cloud includes points above the reference surface.

[0178] According to an embodiment, the instructions may further cause the processor to generate a plurality of first point clouds corresponding to a plurality of objects separated from each other by removing points corresponding to the reference surface in the first point cloud, and generate first bounding boxes corresponding to the plurality of objects.

[0179] According to an embodiment, the electronic device 300 may also include a time-of-flight (ToF) sensor 310, and the instructions may also cause the processor to generate first depth information based on a first ToF frame captured by the ToF sensor 310 in a first direction of the object, and generate second depth information based on a second ToF frame captured by the ToF sensor 310 in a second direction of the object.

[0180] According to an embodiment, the electronic device 300 may also include an RGB image sensor 320, and the instructions may also cause the processor to display guidance information on the display 330 prompting the use of the RGB image sensor 320 to obtain the second depth information based on the number of points in the first point cloud obtained from the first ToF frame being equal to or less than a threshold.

[0181] According to an embodiment, the electronic device 300 may further include an RGB image sensor 320, and the instructions may further cause the processor to perform the following operations: generate first depth information based on a first difference between a third RGB frame captured by the RGB image sensor 320 in a third direction of the object and a first RGB frame captured by the RGB image sensor 320 in a first direction of the object, and generate second depth information based on a second difference between the first RGB frame and a second RGB frame captured by the RGB image sensor 320 in a second direction of the object.

[0182] Figure 14 is a flowchart illustrating an object measurement method of an electronic device according to a disclosed embodiment.

[0183] exist Figure 14 In the embodiment, reference can be made to Figures 1 to 12 The electronic device described (e.g., Figure 3 The electronic device 300 in the embodiment executes the object measurement method, and the above technical features are omitted below.

[0184] In operation 1410, an electronic device (eg, Figure 3the electronic device 300) can generate first depth information in a first direction of the object (for example, Figure 2 According to an embodiment, the electronic device can acquire the first depth information of the object by means of a ToF sensor (for example, Figure 3 According to an embodiment, the electronic device can acquire the first depth information of the object by means of a ToF sensor (for example, Figure 3 According to an embodiment, the electronic device can acquire the first depth information of the object by means of a ToF sensor (for example,

[0185] At operation 1420, the electronic device can generate a first point cloud of the object based on the first depth information. According to an embodiment, the electronic device can generate a first point cloud consisting of only points above the reference surface among point clouds generated based on the first depth information.

[0186] At operation 1430, the electronic device can generate a first bounding box including the first point cloud. The electronic device can generate the first bounding box using minimum x-axis, y-axis, and z-axis coordinate values and maximum x-axis, y-axis, and z-axis coordinate values of the points obtained from coordinate information of the points constituting the first point cloud.

[0187] The electronic device can display the first bounding box so that the first bounding box is overlaid on the object. If it is determined that the first bounding box is not accurate, the electronic device can display guide information prompting generation of a second bounding box in a different direction of the object on the display.

[0188] According to an embodiment, if it is determined that the first bounding box is partially different from a region including the actual object, the processor can identify that the first bounding box is not accurate, and display guide information prompting generation of a second bounding box in a different direction.

[0189] According to an alternative embodiment, if the object extracted from the first RGB image is partially outside the angle of view of the RGB image sensor, the processor can display guide information prompting a shift in the position of the electronic device.

[0190] According to an alternative embodiment, if the number of points constituting the first point cloud is less than a threshold value, the processor can display guide information prompting a shift in the position of the electronic device closer to the object.

[0191] According to an alternative embodiment, if the number of points constituting the first point cloud acquired from the first ToF frame photographed by the ToF sensor 310 is less than a threshold value, the processor can display guide information prompting acquisition of depth information using the RGB image sensor.

[0192] How to display the guide information has been described with reference to 9A to 9D

[0193] ​The electronic device can generate second depth information in a second direction of the object, at operation 1440. The second depth information can be generated in the same manner as the first depth information described with reference to operation 1410.

[0194] The electronic device can generate a second point cloud of the object based on the second depth information, at operation 1450. The second point cloud can be generated in the same manner as the first point cloud described with reference to operation 1420.

[0195] The electronic device can generate a second bounding box including the second point cloud, at operation 1460. The second bounding box can be generated in the same manner as the first bounding box described with reference to operation 1430.

[0196] The electronic device can generate a third bounding box by combining the first bounding box and the second bounding box, at operation 1470.

[0197] The electronic device can display the third bounding box, at operation 1480. The electronic device can also display digital information (e.g., edge length and volume) of the third bounding box.

[0198] According to an embodiment, an object measurement method of an electronic device can include generating first depth information in a first direction of an object, generating a first point cloud of the object based on the first depth information, generating a first bounding box including one or more points of the first point cloud, generating second depth information in a second direction of the object, wherein the second direction is different from the first direction, generating a second point cloud of the object based on the second depth information, generating a second bounding box including one or more points of the second point cloud, generating a third bounding box by combining the first bounding box and the second bounding box, and displaying the third bounding box.

[0199] According to an embodiment, the method can further include displaying digital information including a volume of the third bounding box and an edge length of the third bounding box.

[0200] According to an embodiment, the digital information can be generated based on minimum and maximum coordinate values of points in the third bounding box.

[0201] According to an embodiment, the method can further include extracting the object from a first red, green, blue (RGB) image captured by an RGB image sensor 320 in the first direction of the object, identifying an overlap region in which the first bounding box overlaps a region of the object extracted from the first RGB image, and displaying guide information prompting acquisition of second depth information in the second direction based on a size of the overlap region being equal to or less than a threshold value.

[0202] According to an embodiment, the method may further include: displaying, on the display 330 , guide information prompting acquisition of the second depth information in the second direction, wherein the display of the guide information may be based on at least a portion of the object being outside the viewing angle of the RGB image sensor 320 .

[0203] According to an embodiment, the method may further include: displaying guide information prompting to acquire the second depth information at a position closer to the object, based on the number of points in the first point cloud being equal to or smaller than a threshold.

[0204] According to an embodiment, generating a first point cloud of the object may further include detecting a reference surface in the first point cloud generated based on the first depth information, wherein the first point cloud may include points above the reference surface.

[0205] According to an embodiment, a time-of-flight (ToF) sensor 310 may be used to generate the first depth information.

[0206] As described above, the electronic device and object measurement method thereof disclosed in certain embodiments are advantageous because the electronic device can automatically and accurately measure the size of a real-world object and guide the user to adopt a more accurate measurement method.

[0207] Specific embodiments of the above-described embodiments of the present disclosure may be implemented in hardware, firmware, or by software or computer code that may be stored in a recording medium (such as a CD ROM, a digital versatile disk (DVD), a magnetic tape, a RAM, a floppy disk, a hard disk, or a magneto-optical disk) or by the execution of computer code initially stored on a remote recording medium or a non-transitory machine-readable medium downloaded over a network and to be stored on a local recording medium, so that the methods described herein may be presented by software stored on a recording medium using a general-purpose computer or a dedicated processor or in programmable or dedicated hardware (such as an ASIC or FPGA). As will be understood in the art, a computer, processor, microprocessor controller, or programmable hardware includes a memory component, such as RAM, ROM, flash memory, etc., that can store or receive software or computer code that implements the processing methods described herein when accessed and executed by the computer, processor, or hardware.

[0208] While the disclosure has been shown and described with reference to various embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the disclosure as defined by the appended claims and their equivalents.

Claims

1. An electronic device comprising: monitor; Memory; as well as a processor operatively connected to the display and the memory, The memory stores instructions, wherein when the instructions are executed by the processor, the electronic device performs the following operations: receiving user input for selecting a 2D measurement or a 3D measurement of an object, In response to recognizing that the object is arranged in the screen of the display, generating a first point cloud of the object based on first depth information of the object, generating a first bounding box based on the first point cloud of the object, wherein the first bounding box overlaps with a first overlapping region of the first object image, determining whether the amount of the first overlapping area exceeds a threshold, providing first numerical information of the area or volume of the first bounding box on a display if the amount of the first overlapping area exceeds the threshold, If the amount of the first overlapping area does not exceed the threshold, then: determining a direction for moving the electronic device that enables the electronic device to acquire a second object image and generate a second bounding box that overlaps a second overlapping region of the second object image, wherein the second overlapping image is larger than the first overlapping image, Guidance information prompting the user to move the electronic device in the direction is displayed on the display.

2. The electronic device according to claim 1, wherein The instructions also cause the electronic device to generate first digital information based on minimum and maximum coordinate values ​​of the points in the first bounding box.

3. The electronic device according to claim 1, wherein: The instructions further cause the electronic device to generate a second point cloud of the object based on second depth information of the object in response to the electronic device moving in the direction, wherein the second depth information is generated by the electronic device at the position to which the electronic device is moved. generating a second bounding box based on the second point cloud of the object, wherein the second bounding box overlaps with a second overlapping region of the second object image and is different from the first bounding box, and Second numerical information of the area or volume of the second bounding box is provided on the display.

4. The electronic device according to claim 3, wherein: The instructions further cause the electronic device to generate a third bounding box by combining the first bounding box and the second bounding box, and Third numerical information of the area or volume of the third bounding box is displayed on the display.

5. The electronic device according to claim 1, wherein The instructions further cause the electronic device to display, on the display, another guide information for guiding arrangement of the object in the screen of the display by the mobile electronic device.

6. The electronic device according to claim 1, wherein The instructions further cause the electronic device to display, on the display, another guide information prompting to acquire second depth information at a position closer to the object, based on the number of points in the first point cloud being equal to or less than a threshold.

7. The electronic device according to claim 1, wherein: The instructions further cause the electronic device to detect a reference surface in a first point cloud generated based on the first depth information, wherein the first point cloud includes points above the reference surface.

8. The electronic device according to claim 7, wherein: The instructions also cause the electronic device to perform the following operations: generating a plurality of first point clouds corresponding to a plurality of objects separated from each other by removing points corresponding to the reference surface in the first point cloud, A first bounding box corresponding to the plurality of objects is generated.

9. The electronic device according to claim 1, further comprising: Time of Flight ToF sensor, The instructions further cause the electronic device to generate first depth information based on a first ToF frame captured by the ToF sensor.

10. A method for measuring an object of an electronic device, the method comprising: receiving user input for selecting a 2D measurement or a 3D measurement of an object; In response to recognizing that the object is arranged in the screen of the display, generating a first point cloud of the object based on first depth information of the object; generating a first bounding box based on the first point cloud of the object, wherein the first bounding box overlaps with a first overlapping region of the first object image, determining whether the amount of the first overlapping area exceeds a threshold, providing first numerical information of the area or volume of the first bounding box on a display if the amount of the first overlapping area exceeds the threshold, If the amount of the first overlapping area does not exceed the threshold, then: determining a direction for moving the electronic device that enables the electronic device to acquire a second object image and generate a second bounding box that overlaps a second overlapping region of the second object image, wherein the second overlapping image is larger than the first overlapping image, Guidance information prompting the user to move the electronic device in the direction is displayed on the display.

11. The method according to claim 10, wherein: The first digital information is generated based on minimum and maximum coordinate values ​​of points in the first bounding box.

12. The method of claim 10, further comprising: In response to the electronic device moving in the direction, generating a second point cloud of the object based on second depth information of the object, wherein the second depth information is generated by the electronic device located at the position to which the electronic device is moved, generating a second bounding box based on the second point cloud of the object, wherein the second bounding box overlaps with a second overlapping region of the second object image and is different from the first bounding box, and Second numerical information of the area or volume of the second bounding box is provided on the display.

13. The method of claim 12, further comprising: Producing a third bounding box by combining the first bounding box and the second bounding box, and Third numerical information of the area or volume of the third bounding box is displayed on the display.

14. The method of claim 11, further comprising: Another guide information for guiding arrangement of the object in the screen of the display by the mobile electronic device is displayed on the display.

15. The method of claim 11, wherein: The step of generating a first point cloud of the object further comprises: A reference surface is detected in a first point cloud generated based on the first depth information, wherein the first point cloud includes points above the reference surface.

Images