Mobile terminal and control method thereof

Abstract

The present invention relates to a mobile terminal including an illumination unit and a control method thereof. The mobile terminal according to an embodiment of the present invention includes: an illumination unit; a camera; and a control unit for controlling the illumination unit to irradiate light to an object to be photographed by the camera and controlling the camera to photograph the object irradiated with the illumination light, wherein the control unit determines the material of the object based on illumination information irradiated to the object photographed by the camera.

Description

Technical Field

[0001] The present disclosure relates to a mobile terminal having a lighting device, and more particularly, to a mobile terminal having a lighting device for identifying depth information related to an image or material of an object to be photographed. Background Art

[0002] Terminals can be divided into mobile / portable terminals and fixed terminals according to mobility. In addition, mobile terminals can be classified into handheld types and vehicle-mounted types according to whether users can directly carry them.

[0003] Mobile terminals have become increasingly multifunctional. Examples of these functions include data and voice communications, capturing images and videos via a camera, recording audio, playing music files via a speaker system, and displaying images and videos on a display. Some mobile terminals include additional functions that support playing electronic games, while other terminals are configured as multimedia players. Specifically, recently, mobile terminals can receive broadcast signals and multicast signals to allow viewing of videos and television programs.

[0004] As becoming multifunctional, the mobile terminal may be allowed to capture still images or moving images, play music or video files, play games, receive broadcasts, etc., so as to be implemented as an integrated multimedia player.

[0005] Recently, as the performance of cameras has been improved, various functions using cameras have been developed. For example, a function of shooting high-quality still images or videos or generating 3D images using depth information (or depth values) of images received through cameras has been actively developed.

[0006] For these various functions using the camera, the role of the light emitting element is important. Here, the light emitting element is used to emit light to a space corresponding to an image received by the camera.

[0007] Therefore, it is necessary to develop a light emitting element for performing various functions using a camera and a method of controlling the light emitting element. Summary of the invention

[0008] Technical issues

[0009] Accordingly, an aspect of the present disclosure is to provide a mobile terminal having lighting capable of irradiating in an optimized manner for extracting depth information regarding an image photographed through a camera, and a control method thereof.

[0010] Another aspect of the present disclosure is to provide a mobile terminal having lighting capable of irradiating a material for extracting an object to be photographed through a camera in an optimized manner and a control method thereof.

[0011] Technical Solution

[0012] A mobile terminal may include: a lighting unit; a camera; and a controller configured to control the lighting unit to irradiate lighting light on an object to be photographed by the camera, and control the camera to photograph the object irradiated with the lighting light. The controller may be configured to determine a material of the object based on information related to the lighting light irradiated on the object photographed by the camera.

[0013] In an implementation, the lighting unit may include: a first light source configured to irradiate patterned light; and a second light source configured to irradiate uniform light.

[0014] In an implementation, the controller may turn on the first light source and the second light source alternately.

[0015] In an implementation, the controller may turn off the second light source when the first light source is turned on within a first time period, and turn off the first light source when the second light source is turned on within a second time period after the first time period has elapsed.

[0016] In an implementation, the first light source may include a plurality of light emitting elements. The plurality of light emitting elements may be grouped into a plurality of groups. Each of the plurality of groups may include the same number of light emitting elements.

[0017] In an implementation, the controller may control the light emitting elements included in the plurality of groups in a preset manner.

[0018] In an implementation, the controller may control the lighting unit so that the first light emitting element and the second light emitting element included in each of the plurality of groups output light of different wavelengths.

[0019] In an implementation, the controller may group the plurality of groups into a plurality of units, and control the lighting unit to emit light for each unit without overlap.

[0020] In an implementation, each of the plurality of units may include the same number of groups. While controlling one of the plurality of units, the controller may control the lighting unit so that the light emitting elements included in the remaining units do not emit light.

[0021] In an implementation, the controller may control the lighting unit so that the first light emitting element and the second light emitting element included in each of the plurality of groups output light at different powers.

[0022] In an implementation, the controller may control all of the plurality of light emitting elements to emit light at different powers based on time.

[0023] In an implementation, the controller may control all of the plurality of light emitting elements to emit light at a first power within a first time period, and control all of the plurality of light emitting elements to emit light at a second power different from the first power within a second time period after the first time period has elapsed.

[0024] In an implementation, the controller may group the plurality of light emitting elements into a plurality of units, and control the lighting unit to emit light at different powers based on time for each of the plurality of units.

[0025] In an implementation, the controller may control the lighting unit so that the first light emitting element and the second light emitting element included in each of the plurality of groups output light with different beam profiles.

[0026] In an implementation, the controller may control the lighting unit so that the first light emitting element and the second light emitting element included in each of the plurality of groups output light in different patterns.

[0027] In an implementation, the controller may control the lighting unit so that the first light emitting element and the second light emitting element included in each of the plurality of groups irradiate light of the same pattern but with different sizes of light spots.

[0028] In an implementation, the first light source may include an illumination lens configured to irradiate different patterns of light.

[0029] In implementations, the first light source may include a variable aperture configured to illuminate different patterns of light.

[0030] In an implementation, the first light source may include a diffractive optical element configured to irradiate different patterns of light.

[0031] A method for controlling a mobile terminal may include the following steps: controlling a lighting unit to irradiate illumination light to an object to be photographed by a camera; controlling a camera to photograph the object irradiated with the illumination light; and determining a material of the object based on information related to the illumination light irradiated to the object photographed by the camera.

[0032] Beneficial Effects

[0033] The present disclosure may provide a new mobile terminal capable of determining a material of an object to be photographed by using lighting.

[0034] The present disclosure may provide a new control method capable of determining the material of an object to be photographed in an optimized manner by varying an illumination pattern. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1a is a block diagram of a mobile terminal according to one embodiment of the present disclosure.

[0036] Figure 1b and Figure 1c is a conceptual diagram illustrating one example of a mobile terminal according to the present disclosure viewed from different directions.

[0037] Figure 2 is a conceptual diagram illustrating a camera and a lighting device provided in a mobile terminal related to the present disclosure.

[0038] Figure 3 is a conceptual diagram showing a configuration of a mobile terminal according to one implementation of the present disclosure.

[0039] Figure 4 is a conceptual diagram illustrating a pattern light source according to one implementation of the present disclosure.

[0040] Figure 5 is a conceptual diagram illustrating a control method of a lighting unit according to the present disclosure.

[0041] Figure 6 , Figure 7 , Figure 8 , Fig. 9 , Fig.10 , Fig.11 , Fig.12 , Fig.13a , Fig.13b , Fig.14 , Fig.15 , Fig.16 , Fig.17 , Fig.18 , Fig.19 , Fig. 20 and Fig.21 is a conceptual diagram illustrating a method of controlling illumination (or an emitter) used by a mobile terminal to recognize a material of an object to be photographed. DETAILED DESCRIPTION

[0042] A description will now be given in detail according to the exemplary embodiments disclosed herein with reference to the accompanying drawings. In order to briefly describe with reference to the accompanying drawings, the same or similar reference numerals may be provided for the same or equivalent components, and their description will not be repeated. Generally, suffixes such as "module" and "unit" can be used to refer to elements or components. Such suffixes are used herein only to facilitate the description of the specification, and the suffixes themselves are not intended to give any special meaning or function. When describing the present disclosure, if a detailed description of related known functions or structures is considered to be unnecessary to deviate from the main purpose of the present disclosure, such description has been omitted, but it will be understood by those skilled in the art. The accompanying drawings are used to help easily understand the technical concept of the present disclosure, and it should be understood that the concept of the present disclosure is not limited by the accompanying drawings. The concept of the present disclosure should be interpreted as extending to any modified form, equivalent form and alternative form beyond the accompanying drawings.

[0043] It will be understood that although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are generally used only to distinguish one element from another.

[0044] It will be understood that when an element is referred to as being "connected to" another element, the element may be connected to the other element or intervening elements may also be present. In contrast, when an element is referred to as being "directly connected to" another element, there are no intervening elements.

[0045] A singular expression may include a plural expression unless it clearly indicates a different meaning from the context.

[0046] Terms such as “including” or “having” used herein should be understood to indicate that there are a plurality of components, functions or steps disclosed in the specification, and it should also be understood that more or fewer components, functions or steps may be used as well.

[0047] The mobile terminals presented herein can be implemented using various different types of terminals. Examples of these terminals include cellular phones, smart phones, user equipment, laptop computers, digital broadcast terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigators, portable computers (PCs), slate PCs, tablet PCs, ultrabooks, wearable devices (e.g., smart watches, smart glasses, head mounted displays (HMDs)), etc.

[0048] Only as non-limiting example, will be further described with reference to the mobile terminal of specific type.However, these teachings are equally applicable to other types of terminals, such as those types described above.In addition, these teachings can also be applied to fixed terminals such as digital TV, desktop computers, etc.

[0049] Reference Figure 1a to Figure 1c , Figure 1a is a block diagram of a mobile terminal according to an implementation of the present disclosure, Figure 1b and Figure 1c is a conceptual diagram illustrating one example of a mobile terminal viewed from different directions.

[0050] The mobile terminal 100 may be shown having components such as a wireless communication unit 110, an input unit 120, a sensing unit 140, an output unit 150, an interface unit 160, a memory 170, a controller (or control unit) 180, and a power supply unit 190. It will be understood that implementing all of the illustrated components is not a requirement, and greater or fewer components may alternatively be implemented.

[0051] In more detail, the wireless communication unit 110 may generally include one or more modules that allow communication, such as wireless communication between the mobile terminal 100 and a wireless communication system, communication between the mobile terminal 100 and another mobile terminal, or communication between the mobile terminal 100 and an external server. In addition, the wireless communication unit 110 may generally include one or more modules that connect the mobile terminal 100 to one or more networks.

[0052] The wireless communication unit 110 may include one or more of a broadcast receiving module 111 , a mobile communication module 112 , a wireless Internet module 113 , a short-range communication module 114 , and a location information module 115 .

[0053] The input unit 120 may include a camera 121 or an image input unit for obtaining an image or video, a microphone 122 as a kind of audio input device for inputting an audio signal, and a user input unit 123 (e.g., a touch key, a mechanical key, etc.) for allowing a user to input information. Data (e.g., audio, video, image, etc.) may be obtained through the input unit 120 and may be analyzed and processed according to a user command.

[0054] The sensing unit 140 may generally be implemented using one or more sensors configured to sense internal information of the mobile terminal, the surrounding environment of the mobile terminal, user information, etc. For example, the sensing unit 140 may include at least one of a proximity sensor 141, an illumination sensor 142, a touch sensor, an acceleration sensor, a magnetic sensor, a gravity sensor, a gyro sensor, a motion sensor, an RGB sensor, an infrared (IR) sensor, a finger scan sensor, an ultrasonic sensor, an optical sensor (e.g., a camera 121), a microphone 122, a battery gauge, an environmental sensor (e.g., a barometer, a hygrometer, a thermometer, a radiation detection sensor, a thermal sensor, and a gas sensor, etc.), and a chemical sensor (e.g., an electronic nose, a health sensor, a biosensor, etc.). The mobile terminal disclosed herein may be configured to utilize information obtained from one or more sensors of the sensing unit 140 and combinations thereof.

[0055] The output unit 150 may be generally configured to output various types of information, such as audio, video, tactile output, etc. The output unit 150 may be shown to have at least one of a display unit 151, an audio output module 152, a tactile module 153, and an optical output module 154. The display unit 151 may have an interlayer structure or an integrated structure with a touch sensor to implement a touch screen. The touch screen may be used as a user input unit 123, which provides an input interface between the mobile terminal 100 and the user, and at the same time provides an output interface between the mobile terminal 100 and the user.

[0056] The interface unit 160 serves as an interface with various types of external devices connected to the mobile terminal 100. For example, the interface unit 160 may include any wired or wireless port, an external power port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, etc. In some cases, the mobile terminal 100 may perform various control functions associated with the connected external device in response to the external device being connected to the interface unit 160.

[0057] The memory 170 is generally implemented to store data supporting various functions or features of the mobile terminal 100. For example, the memory 170 may be configured to store applications executed in the mobile terminal 100, data or instructions for the operation of the mobile terminal 100, etc. Some of these applications may be downloaded from an external server via wireless communication. Other applications may be installed in the mobile terminal 100 at the time of manufacture or shipment, which is usually the case for the basic functions of the mobile terminal 100 (e.g., answering calls, making calls, receiving messages, sending messages, etc.). Applications may be stored in the memory 170, installed in the mobile terminal 100, and executed by the controller 180 to perform operations (or functions) of the mobile terminal 100.

[0058] In addition to operations associated with applications, the controller 180 is generally used to control the overall operation of the mobile terminal 100. The controller 180 may provide or process information or functions suitable for the user by processing signals, data, information, etc. input or output by the above-mentioned various components or activating applications stored in the memory 170.

[0059] In addition, the controller 180 can control Figure 1a At least some of the components shown are activated to execute an application stored in the memory 170. In addition, the controller 180 may control at least two of those components included in the mobile terminal 100 to activate an application.

[0060] The power supply unit 190 may be configured to receive external power or provide internal power so as to supply appropriate power required to operate elements and components included in the mobile terminal 100. The power supply unit 190 may include a battery, which may be configured to be embedded in the terminal body or configured to be detachable from the terminal body.

[0061] At least some components may operate cooperatively to implement the operation, control or control method of the mobile terminal according to various embodiments disclosed herein. In addition, the operation, control or control method of the mobile terminal may be implemented on the mobile terminal by activating at least one application stored in the memory 170.

[0062] Hereinafter, before describing various embodiments implemented by the mobile terminal 100, reference will be made to Figure 1a The above components are described in more detail.

[0063] First, with respect to the wireless communication unit 110, the broadcast receiving module 111 is generally configured to receive broadcast signals and / or broadcast related information from an external broadcast management entity via a broadcast channel. The broadcast channel may include a satellite channel, a terrestrial channel, or both. In some embodiments, two or more broadcast receiving modules may be used to facilitate simultaneous reception of two or more broadcast channels or to support switching between broadcast channels.

[0064] The mobile communication module 112 may send wireless signals to and / or receive wireless signals from one or more network entities. Typical examples of network entities include base stations, external mobile terminals, servers, etc. These network entities form a part of a mobile communication network, which is constructed according to a technical standard or communication method for mobile communication (e.g., Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), CDMA 2000 (Code Division Multiple Access 2000), EV-DO (Enhanced Optimized Voice Data or Enhanced Voice Data Only), Wideband CDMA (WCDMA), High Speed ​​Downlink Packet Access (HSDPA), HSUPA (High Speed ​​Uplink Packet Access), Long Term Evolution (LTE), LTE-A (Long Term Evolution-Advanced), etc.).

[0065] The wireless signals may include various types of data according to a voice call signal, a video call signal, or text / multimedia message transmission / reception.

[0066] The wireless Internet module 113 refers to a module for wireless Internet access. This module may be internally or externally connected to the mobile terminal 100. The wireless Internet module 113 may transmit and / or receive wireless signals via a communication network according to wireless Internet technology.

[0067] Examples of such wireless Internet access include wireless LAN (WLAN), wireless fidelity (Wi-Fi), Wi-Fi Direct, Digital Living Network Alliance (DLNA), wireless broadband (WiBro), world interoperability for microwave access (WiMAX), high-speed downlink packet access (HSDPA), high-speed uplink packet access (HSUPA), long term evolution (LTE), advanced LTE (LTE-A), etc. The wireless Internet module 113 may transmit / receive data according to one or more of these wireless Internet technologies as well as other Internet technologies.

[0068] The wireless Internet module 113 performs such wireless Internet access as part of a mobile communication network when wireless Internet access is implemented according to, for example, WiBro, HSDPA, HSUPA, GSM, CDMA, WCDMA, LTE, LTE-A, etc. Therefore, the Internet module 113 may cooperate with the mobile communication module 112 or function as the mobile communication module 112.

[0069] The short-range communication module 114 is configured to facilitate short-range communications. Suitable technologies for implementing such short-range communications include BLUETOOTH TM , Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, Near Field Communication (NFC), Wireless Fidelity (Wi-Fi), Wi-Fi Direct, Wireless USB (Wireless Universal Serial Bus), etc. The short-range communication module 114 generally supports wireless communication between the mobile terminal 100 and a wireless communication system, communication between the mobile terminal 100 and another mobile terminal 100, or communication between a mobile terminal and a network where another mobile terminal 100 (or an external server) is located via a wireless local area network. An example of a wireless local area network is a wireless personal area network.

[0070] Here, another mobile terminal (which may be configured similarly to the mobile terminal 100) may be a wearable device (e.g., a smart watch, smart glasses, or a head mounted display (HMD)) capable of exchanging data with the mobile terminal 100 (or cooperating with the mobile terminal 100). The short-range communication module 114 may sense or identify the wearable device and allow communication between the wearable device and the mobile terminal 100. In addition, when the sensed wearable device is a device verified as communicating with the mobile terminal 100, for example, the controller 180 may send at least a portion of the data processed in the mobile terminal 100 to the wearable device via the short-range communication module 114. Therefore, the user of the wearable device may use the data processed in the mobile terminal 100 on the wearable device. For example, when a call is received in the mobile terminal 100, the user may use the wearable device to answer the call. In addition, when a message is received in the mobile terminal 100, the user may use the wearable device to view the received message.

[0071] The location information module 115 is generally configured to detect, calculate, derive or identify the location (or current location) of the mobile terminal. For example, the location information module 115 includes a global positioning system (GPS) module, a Wi-Fi module or both. For example, when the mobile terminal uses a GPS module, the location of the mobile terminal can be acquired using a signal sent from a GPS satellite. As another example, when the mobile terminal uses a Wi-Fi module, the location of the mobile terminal can be acquired based on information related to a wireless access point (AP), which sends a wireless signal to the Wi-Fi module or receives a wireless signal from the Wi-Fi module. If necessary, the location information module 115 may alternatively or additionally work with any other module of the wireless communication unit 110 to obtain data related to the location of the mobile terminal. The location information module 115 is a module for acquiring a location (or current location), and may not be limited to a module for directly calculating or acquiring the location of a mobile terminal.

[0072] Examples of these inputs include audio, image, video data, and user input. Image and video input are often obtained using one or more cameras 121. These cameras 121 can process image frames of still pictures or videos obtained by image sensors in video or image capture mode. The processed image frames can be displayed on the display unit 151 or stored in the memory 170. In addition, the camera 121 can be arranged in a matrix configuration so that multiple images with various angles or focal points can be input to the mobile terminal 100. In addition, the camera 121 can be arranged in a stereoscopic arrangement to obtain a left image and a right image for realizing a stereoscopic image.

[0073] The microphone 122 processes the external audio signal into electrical audio (sound) data. The processed audio data may be processed in various ways according to the functions executed in the mobile terminal 100. If necessary, the microphone 122 may include various noise removal algorithms to remove undesirable noise generated in the process of receiving the external audio signal.

[0074] The user input unit 123 is a component that allows user input. Such user input can enable the controller 180 to control the operation of the mobile terminal 100. The user input unit 123 may include one or more of a mechanical input element (e.g., a mechanical key, button, membrane switch, roller, touch switch, etc. located on the front and / or back or side of the mobile terminal 100) or a touch-sensitive input element. As an example, the touch-sensitive input element may be a virtual key, soft key or visual key displayed on the touch screen by software processing, or a touch key set at a position other than the touch screen on the mobile terminal. On the other hand, the virtual key or visual key may be displayed on the touch screen in various shapes (e.g., graphics, text, icon, video, or a combination thereof).

[0075] The sensing unit 140 is generally configured to sense one or more of the internal information of the mobile terminal, the surrounding environment information of the mobile terminal, the user information, etc., and generate a corresponding sensing signal. The controller 180 generally cooperates with the sensing unit 140 to control the operation of the mobile terminal 100 based on the sensing signal or to perform data processing, functions or operations associated with the application installed in the mobile terminal. The sensing unit 140 can be implemented using any of a variety of sensors, some of which will now be described in more detail.

[0076] The proximity sensor 141 refers to a sensor that senses the presence of an object approaching a surface or an object located near a surface without mechanical contact using an electromagnetic field, infrared rays, etc. The proximity sensor 141 may be arranged at an inner area of ​​the mobile terminal covered by the touch screen or near the touch screen.

[0077] When the touch screen is implemented as a capacitive type, the proximity sensor 141 can sense the proximity of the pointer relative to the touch screen by the change of the electromagnetic field in response to the approach of the conductive object. When the touch screen is implemented as a capacitive type, the proximity sensor 141 can sense the proximity of the pointer relative to the touch screen by the change of the electromagnetic field in response to the approach of the conductive object. In this case, the touch screen (touch sensor) can also be classified as a proximity sensor.

[0078] The term "proximity touch" will often be mentioned herein to indicate a scenario in which a pointer is set to approach a touch screen without contacting the touch screen. The term "contact touch" will often be mentioned herein to indicate a scenario in which a pointer makes physical contact with a touch screen. For a position corresponding to a proximity touch of a pointer relative to the touch screen, such a position will correspond to a position of the pointer perpendicular to the touch screen. The proximity sensor 141 can sense proximity touches and proximity touch patterns (e.g., distance, direction, speed, time, position, moving state, etc.). Typically, the controller 180 processes data corresponding to proximity touches and proximity touch patterns sensed by the proximity sensor 141 and outputs visual information on the touch screen. In addition, the controller 180 can control the mobile terminal 100 to perform different operations or process different data (or information) depending on whether the touch to a point on the touch screen is a proximity touch or a contact touch.

[0079] The touch sensor senses a touch (or touch input) applied to the touch screen (or display unit 151) using any of a variety of touch methods. Examples of such touch methods include resistive, capacitive, infrared, and magnetic field types, among others.

[0080] As an example, the touch sensor may be configured to convert a change in pressure applied to a specific portion of the display unit 151 or a capacitance occurring at a specific portion of the display unit 151 into an electric input signal. The touch sensor may also be configured to sense not only a touched position and a touched area, but also a touch pressure and / or a touch capacitance. A touch object is generally used to apply a touch input to the touch sensor. Typical examples of touch objects include fingers, touch pens, styluses, pointers, and the like.

[0081] When a touch input is sensed by the touch sensor, a corresponding signal may be sent to a touch controller. The touch controller may process the received signal and then send the corresponding data to the controller 180. Thus, the controller 180 may sense which area of ​​the display unit 151 is touched. Here, the touch controller may be a component independent of the controller 180, the controller 180, or a combination thereof.

[0082] In addition, the controller 180 can perform the same or different controls according to the type of touch object that touches the touch screen or a touch key provided in addition to the touch screen. For example, whether the same control or different control is performed according to the object providing the touch input can be determined based on the current operating state of the mobile terminal 100 or the currently executed application.

[0083] The touch sensor and the proximity sensor can be implemented alone or in combination to sense various types of touches, including short (or tap) touch, long touch, multi-touch, drag touch, flick touch, zoom in touch, swipe touch, hover touch, etc.

[0084] If necessary, an ultrasonic sensor may be implemented to identify position information related to a touch object using ultrasonic waves. For example, the controller 180 may calculate the position of a wave generation source based on information sensed by an illumination sensor and a plurality of ultrasonic sensors. Since light is much faster than ultrasonic waves, the time it takes for light to reach an optical sensor is much shorter than the time it takes for ultrasonic waves to reach an ultrasonic sensor. This fact may be used to calculate the position of a wave generation source. For example, the position of a wave generation source may be calculated using a time difference relative to the time it takes for ultrasonic waves to reach a sensor based on light as a reference signal.

[0085] The camera 121 , which has been depicted as a component of the input unit 120 , typically includes at least one camera sensor (CCD, CMOS, etc.), a photoelectric sensor (or image sensor), and a laser sensor.

[0086] Implementing the camera 121 with a laser sensor may allow the detection of a touch of a physical object relative to a 3D stereoscopic image. The photoelectric sensor may be laminated on or overlapped with the display device. The photoelectric sensor may be configured to scan the movement of a physical object approaching the touch screen. In more detail, the photoelectric sensor may include photodiodes and transistors (TR) in rows and columns to scan the content received at the photoelectric sensor using an electrical signal that varies according to the amount of light applied. That is, the photoelectric sensor may calculate the coordinates of the physical object based on the change in light, thereby obtaining the position information of the physical object.

[0087] The display unit 151 is generally configured to output information processed in the mobile terminal 100. For example, the display unit 151 may display execution screen information of an application executed at the mobile terminal 100 or User Interface (UI) and Graphic User Interface (GUI) information in response to the execution screen information.

[0088] In addition, the display unit 151 may be implemented as a stereoscopic display unit for displaying a stereoscopic image.

[0089] A typical stereoscopic display may adopt a stereoscopic display scheme such as a stereoscopic scheme (glasses scheme), an autostereoscopic scheme (glasses-free scheme), a projection scheme (holographic scheme), and the like.

[0090] The audio output module 152 may receive audio data from the wireless communication unit 110 or output audio data stored in the memory 170 in modes such as a signal reception mode, a call mode, a recording mode, a voice recognition mode, a broadcast reception mode, etc. The audio output module 152 may provide audible outputs related to a particular function performed by the mobile terminal 100 (e.g., a call signal reception sound, a message reception sound, etc.). The audio output module 152 may also be implemented as a receiver, a speaker, a buzzer, etc.

[0091] The haptic module 153 may be configured to generate various tactile effects that the user feels, perceives, or experiences. A typical example of the tactile effect generated by the haptic module 153 is vibration. The intensity, pattern, etc. of the vibration generated by the haptic module 153 may be controlled by user selection or setting of the controller. For example, the haptic module 153 may output different vibrations in a combined manner or in a sequential manner.

[0092] In addition to vibration, the tactile module 153 can generate various other tactile effects, including stimulation effects such as vertical movement of a pin arrangement toward contact with the skin, a jet force or suction force of air through a jet hole or a suction opening, touch to the skin, contact of electrodes, electrostatic force, etc., effects of reproducing the sensation of cold and hot using an element that can absorb or generate heat, and the like.

[0093] The haptic module 153 may be implemented to enable the user to feel the haptic effect through a muscle sense such as the user's fingers or arm, in addition to transferring the haptic effect through direct contact. Two or more haptic modules 153 may be provided according to the particular configuration of the mobile terminal 100 .

[0094] The optical output module 154 may output a signal for indicating the occurrence of an event using light from a light source. Examples of events occurring in the mobile terminal 100 may include message reception, call signal reception, missed call, alarm, schedule reminder, email reception, information reception through an application, and the like.

[0095] The signal output by the optical output module 154 may be implemented to cause the mobile terminal to emit monochromatic light or multiple colors of light. For example, as the mobile terminal senses that the user has viewed the event that has occurred, the signal output may be terminated.

[0096] The interface unit 160 functions as an interface for external devices to be connected to the mobile terminal 100. For example, the interface unit 160 may receive data transmitted from an external device, receive power to be delivered to elements and components within the mobile terminal 100, or transmit internal data of the mobile terminal 100 to such an external device. The interface unit 160 may include a wired or wireless headset port, an external power port, a wired or wireless data port, a memory card port, a port for connecting a device having an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, and the like.

[0097] The identification module may be a chip that stores various information for verifying the use authority of the mobile terminal 100, and may include a user identification module (UIM), a subscriber identification module (SIM), a universal subscriber identification module (USIM), etc. In addition, a device having an identification module (also referred to herein as an "identification device") may take the form of a smart card. Therefore, the identification device may be connected to the mobile terminal 100 via the interface unit 160.

[0098] When the mobile terminal 100 is connected to an external cradle, the interface unit 160 may be used as a passage that enables power from the cradle to be supplied to the mobile terminal 100, or may be used as a passage that enables various command signals input from the cradle by a user to be transmitted to the mobile terminal. Various command signals or power input from the cradle may be used as a signal for recognizing that the mobile terminal is correctly mounted on the cradle.

[0099] The memory 170 may store programs for supporting operations of the controller 180 and store input / output data (eg, phonebook, messages, still images, videos, etc.) The memory 170 may store data related to various patterns of vibration and audio output in response to a touch input on the touch screen.

[0100] The memory 170 may include one or more types of storage media, including a flash memory type, a hard disk type, a solid state disk (SSD) type, a silicon magnetic disk (SDD) type, a micro multimedia card type, a card type memory (e.g., SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. The mobile terminal 100 may also operate in relation to a network storage device that performs a storage function of the memory 170 on a network such as the Internet.

[0101] The controller 180 may generally control operations related to application programs and general operations of the mobile terminal 100. For example, when the state of the mobile terminal satisfies a preset condition, the controller 180 may set or release a lock state for restricting a user from inputting a control command with respect to an application.

[0102] The controller 180 may also perform control and processing associated with voice calls, data communications, video calls, etc., or perform pattern recognition processing to recognize handwriting input or drawing input made on the touch screen as characters or images, respectively. In addition, the controller 180 may control one or a combination of those components to implement various exemplary embodiments disclosed herein.

[0103] The power supply unit 190 receives external power or provides internal power and supplies appropriate power required to operate various elements and components included in the HMD 100 under the control of the controller 180. The power supply unit 190 may include a battery, which is generally rechargeable or detachably coupled to the terminal body for charging.

[0104] The power supply unit 190 may include a connection port. The connection port may be configured as one example of the interface unit 160, and an external charger for supplying power to recharge the battery may be electrically connected to the connection port.

[0105] As another example, the power supply unit 190 may be configured to recharge the battery wirelessly without using a connection port. In this example, the power supply unit 190 may receive power delivered from an external wireless power transmitter using at least one of an inductive coupling method based on magnetic induction or a magnetic resonance coupling method based on electromagnetic resonance.

[0106] Various embodiments described herein may be implemented in a computer-readable medium, machine-readable medium, or similar medium using, for example, software, hardware, or any combination thereof.

[0107] Reference Figure 1b and Figure 1c , the disclosed mobile terminal 100 includes a straight-type terminal body. However, alternatively, the mobile terminal 100 can be implemented in any of a variety of different configurations. Examples of these configurations include watch-type, clip-type, glasses-type, or folding, flip-type, slide-type, rotating and swing-type (where two or more bodies are combined with each other in a manner that can move relative to each other) or a combination thereof. The discussion herein will often involve a specific type of mobile terminal. However, these teachings about a specific type of mobile terminal will generally also apply to other types of mobile terminals.

[0108] Here, the mobile terminal 100 is regarded as at least one assembly, and the terminal body may be understood as a concept involving the assembly.

[0109] The mobile terminal 100 will generally include a housing (e.g., a frame, a housing, a cover, etc.) that forms the appearance of the terminal. In this embodiment, the housing is formed using a front housing 101 and a rear housing 102. Various electronic components are inserted into the space formed between the front housing 101 and the rear housing 102. In addition, at least one intermediate housing may be provided between the front housing 101 and the rear housing 102.

[0110] The display unit 151 is shown disposed at the front side of the terminal body to output information. As shown, a window 151a of the display unit 151 may be mounted to the front case 101 to form a front surface of the terminal body together with the front case 101.

[0111] In some embodiments, electronic components may also be mounted to the rear housing 102. Examples of these electronic components mounted to the rear housing 102 may include a removable battery, an identification module, a memory card, etc. In this case, a rear cover 103 is shown covering the electronic components, and the cover may be removably coupled to the rear housing 102. Therefore, when the rear cover 103 is removed from the rear housing 102, the electronic components mounted on the rear housing 102 are exposed to the outside.

[0112] As shown, when the back cover 103 is coupled to the rear case 102, the side surface of the rear case 102 may be partially exposed. In some cases, when coupled, the rear case 102 may also be completely shielded by the back cover 103. In addition, the back cover 103 may include an opening for exposing the camera 121b or the second audio output module 152b to the outside.

[0113] The housings 101 , 102 , 103 may be formed by injection molding of a synthetic resin, or may be formed of a metal such as stainless steel (STS), aluminum (Al), titanium (Ti), or the like.

[0114] As an alternative to the example in which a plurality of housings form an internal space for accommodating components, the mobile terminal 100 may be configured such that one housing forms the internal space. In this case, the mobile terminal 100 having a single body is formed such that synthetic resin or metal extends from the side surface to the rear surface.

[0115] In addition, the mobile terminal 100 may include a waterproof unit (not shown) for preventing water from entering the terminal body. For example, the waterproof unit may include a waterproof member between the window 151a and the front housing 101, between the front housing 101 and the rear housing 102, or between the rear housing 102 and the back cover 103 to seal the internal space when those housings are connected.

[0116] The mobile terminal 100 may include a display unit 151, first and second audio output modules 152a and 152b, a proximity sensor 141, an illumination sensor 142, an optical output module 154, first and second cameras 121a and 121b, first and second manipulation units 123a and 123b, a microphone 122, an interface unit 160, and the like.

[0117] The following, such as Figure 1b and Figure 1c As shown, an exemplary mobile terminal 100 will be described, in which the front surface of the terminal body is shown to have a display unit 151, a first audio output module 152a, a proximity sensor 141, an illumination sensor 142, an optical output module 154, a first camera 121a and a first manipulation unit 123a, the side surface of the terminal body is shown to have a second manipulation unit 123b, a microphone 122 and an interface unit 160, and the rear surface of the terminal body is shown to have a second audio output module 152b and a second camera 121b.

[0118] However, those components may not be limited to the arrangement described. Some components may be omitted or rearranged or arranged on different surfaces. For example, the first manipulation unit 123a may be arranged on another surface of the terminal body, and the second audio output module 152b may be arranged on a side surface of the terminal body other than the rear surface of the terminal body.

[0119] The display unit 151 is generally configured to output information processed in the mobile terminal 100. For example, the display unit 151 may display execution screen information of an application executed at the mobile terminal 100 or User Interface (UI) and Graphic User Interface (GUI) information in response to the execution screen information.

[0120] The display unit 151 may include at least one of a liquid crystal display (LCD), a thin film transistor-LCD (TFT-LCD), an organic light emitting diode (OLED), a flexible display, a three-dimensional (3D) display, and an electronic ink display.

[0121] The display unit 151 may be implemented using two display devices according to its configuration type. For example, a plurality of display units 151 may be arranged on one side (spaced apart from each other or these devices may be integrated), or these devices may be arranged on different surfaces.

[0122] The display unit 151 may include a touch sensor that senses a touch relative to the display unit 151 to receive a control command in a touch manner. Therefore, when a touch is applied to the display unit 151, the touch sensor may sense the touch, and the controller 180 may generate a control command corresponding to the touch. Content input in a touch manner may be characters, numbers, instructions of various patterns, or a menu item that can be specified.

[0123] On the other hand, the touch sensor may be configured in the form of a film having a touch pattern and disposed between the window 151a and a display (not shown) on the rear surface of the window, or may be a metal wire patterned directly on the rear surface of the window. Alternatively, the touch sensor may be formed integrally with the display. For example, the touch sensor may be disposed on a substrate of the display, or may be disposed inside the display.

[0124] Thus, the display unit 151 may form a touch screen together with the touch sensor, in which case the touch screen may be used as a user input unit (123, see Figure 1a ). In some cases, the touch screen may replace at least some functions of the first manipulation unit 123a.

[0125] The first audio output module 152a may be implemented as a receiver that transmits a call sound to a user's ear, and the second audio output module 152b may be implemented as a speaker that outputs various alarm sounds or multimedia reproduction request sounds.

[0126] The window 151a of the display unit 151 may include a sound hole for emitting the sound generated from the first audio output module 152a. However, the present disclosure is not limited thereto, and the sound may be released along the assembly gap between the structures (e.g., the gap between the window 151a and the front housing 101). In this case, in terms of appearance, the hole independently formed to output the audio sound is not visible or hidden, thereby further simplifying the appearance of the mobile terminal 100.

[0127] The optical output module 154 may be configured to output light for indicating the occurrence of an event. Examples of these events may include message reception, call signal reception, missed call, alarm, schedule reminder, email reception, information reception through an application, etc. When the user has viewed the event that occurred, the controller 180 may control the optical output module 154 to stop light output.

[0128] The first camera 121 a may process image frames (eg, still images or moving images) obtained by an image sensor in a photographing mode or a video call mode, and the processed image frames may then be displayed on the display unit 151 or stored in the memory 170 .

[0129] The first manipulation unit 123a and the second manipulation unit 123b are examples of the user input unit 123, which can be manipulated by the user to provide input to the mobile terminal 100. The first manipulation unit 123a and the second manipulation unit 123b may also be collectively referred to as a manipulation portion. The first manipulation unit 123a and the second manipulation unit 123b may adopt any method of a tactile manner that allows the user to perform manipulation (e.g., touch, push, scroll, etc.) using a sense of touch. The first manipulation unit 123a and the second manipulation unit 123b may also be manipulated by proximity touch, hovering touch, etc., without the user's sense of touch.

[0130] The drawings are illustrated based on that the first manipulation unit 123a is a touch key, but the present disclosure may not necessarily be limited thereto. For example, the first manipulation unit 123a may be configured using a mechanical key or a combination of a touch key and a key.

[0131] The content received by the first manipulation unit 123a and the second manipulation unit 123b can be set in various ways. For example, the user can use the first manipulation unit 123a to input commands such as menu, home key, cancel, search, etc., and the user can use the second manipulation unit 123b to input commands such as controlling the volume output from the first audio output module 152a or the second audio output module 152b, switching to a touch recognition mode of the display unit 151, etc.

[0132] On the other hand, as another example of the user input unit 123, a rear input unit (not shown) may be provided on the rear surface of the terminal body. The user may manipulate the rear input unit to input commands for controlling the operation of the mobile terminal 100. Content input may be set in various ways. For example, the user may use the rear input unit to input commands such as power on / off, start, end, scroll, etc., control the volume output from the first audio output module 152a or the second audio output module 152b, switch to a touch recognition mode of the display unit 151, etc. The rear input unit may be implemented in a form that allows touch input, push input, or a combination thereof.

[0133] The rear input unit may be disposed to overlap the display unit 151 of the front surface in the thickness direction of the terminal body. As an example, the rear input unit may be disposed at an upper end portion of the rear surface of the terminal body so that the user can easily manipulate it with an index finger when the user grasps the terminal body with one hand. However, the present disclosure may not be limited thereto, and the position of the rear input unit may be changeable.

[0134] When the rear input unit is disposed on the rear surface of the terminal body, a new user interface can be implemented using the rear input unit. In addition, the above-mentioned touch screen or rear input unit can replace at least part of the functions of the first manipulation unit 123a located on the front surface of the terminal body. Therefore, when the first manipulation unit 123a is not disposed on the front surface of the terminal body, the display unit 151 can be implemented to have a larger screen.

[0135] On the other hand, the mobile terminal 100 may include a finger scan sensor that scans a user's fingerprint. The controller may use the fingerprint information sensed by the finger scan sensor as an authentication means. The finger scan sensor may be installed in the display unit 151 or the user input unit 123.

[0136] The microphone 122 may be disposed in a plurality of places and configured to receive stereo sound. The microphone 122 may be disposed in a plurality of places and configured to receive stereo sound.

[0137] The interface unit 160 may serve as a path that allows the mobile terminal 100 to interface with external devices. For example, the interface unit 160 may be at least one of a connection terminal for connecting to another device (e.g., earphones, external speakers, etc.), a port for near field communication (e.g., an infrared data association (IrDA) port, a Bluetooth port, a wireless LAN port, etc.), or a power terminal for supplying power to the mobile terminal 100. The interface unit 160 may be implemented in the form of a slot for accommodating an external card (e.g., a subscriber identity module (SIM), a user identity module (UIM), or a memory card for information storage).

[0138] A second camera 121b may be further mounted to a rear surface of the terminal body. The second camera 121b may have an image capturing direction substantially opposite to that of the first camera 121a.

[0139] The second camera 121b may include a plurality of lenses arranged along at least one line. The plurality of lenses may be arranged in a matrix form. These cameras may be referred to as "array cameras". When the second camera 121b is implemented as an array camera, images may be captured in various ways using a plurality of lenses, and images with better quality may be obtained.

[0140] The lighting unit 124 may be disposed adjacent to the second camera 121b. When an image of a subject is captured using the camera 121b, the lighting unit 124 may illuminate the subject.

[0141] A second audio output module 152b may be further provided on the terminal body. The second audio output module 152b may be combined with the first audio output module 152a to implement a stereo function and may also be used to implement a hands-free mode for call communication.

[0142] At least one antenna for wireless communication may be provided on the terminal body. The antenna may be embedded in the terminal body or formed in the housing. For example, the broadcast receiving module 111 (see Figure 1a ) can be retracted into the terminal body. Alternatively, the antenna can be formed in the form of a film attached to the inner surface of the rear cover 103, or a housing containing a conductive material can be used as the antenna.

[0143] The terminal body is provided with a power supply unit 190 (see Figure 1a ). The power supply unit 190 may include a battery 191 which is installed in the terminal body or detachably coupled to the outside of the terminal body.

[0144] The battery 191 may receive power via a power line connected to the interface unit 160. In addition, the battery 191 may be (re)charged wirelessly using a wireless charger. Wireless charging may be achieved through magnetic induction or electromagnetic resonance.

[0145] On the other hand, the drawings show that the back cover 103 is coupled to the rear case 102 for shielding the battery 191 to prevent the battery 191 from being separated and to protect the battery 191 from external impact or foreign matter. When the battery 191 is detachable from the terminal body, the back cover 103 may be detachably coupled to the rear case 102.

[0146] An accessory for protecting the appearance or assisting or extending the functions of the mobile terminal 100 may be further provided on the mobile terminal 100. As one example of an accessory, a cover or bag for covering or accommodating at least one surface of the mobile terminal 100 may be provided. The cover or bag may cooperate with the display unit 151 to extend the functions of the mobile terminal 100. Another example of an accessory may be a touch pen for assisting or extending touch input on the touch screen.

[0147] In addition, a mobile terminal related to the present disclosure may extract (detect, determine, sense) depth information from an image captured by a camera using a camera and a lighting device (a lighting unit, a lighting module, etc., which will be described together).

[0148] In addition, the mobile terminal related to the present disclosure may use a camera and an illumination device to capture (or generate) a 3D image. For example, the mobile terminal related to the present disclosure may convert (generate) a 2D image captured by a camera into a 3D image based on the extracted depth information. In another example, the mobile terminal related to the present disclosure may determine the distance to an object based on the light irradiated from the illumination device, and capture (or generate) a 3D image through a camera based on the distance to the object.

[0149] Hereinafter, a method for extracting depth information from an image captured using a camera and an illumination device will be described in detail with reference to the accompanying drawings. Hereinafter, extraction of depth information from an image captured by a camera will be described, but the relevant content may also be inferred and applied to capturing (or generating) a 3D image in the same / similar manner.

[0150] The mobile terminal 100 related to the present disclosure can extract the image data through the camera 121 (please refer to Figure 1a ) receives (or captures) depth information.

[0151] The image received through the camera may be referred to as a preview image. In detail, the preview image may refer to an image received through the camera in real time. The preview image may change as the mobile terminal having the camera 121 moves due to an external force or as an object moves.

[0152] For example, an image captured by a camera may refer to an image obtained by capturing a preview image. For example, an image may be captured when an image capture button output on a display unit of a mobile terminal is touched, when a user gesture associated with capturing a preview image is sensed through a preview image, or when a physical button set in the mobile terminal is pressed.

[0153] The image described in the present disclosure may refer to at least one of a preview image and a captured image.

[0154] The depth information described in the present disclosure may be a depth value. The depth information may refer to a distance (or distance value) between a pixel corresponding to an object included in an image and a mobile terminal (particularly, a camera).

[0155] For example, when the distance between an object corresponding to a specific pixel of an image and the mobile terminal is n, the depth information of the specific pixel may be a specific value corresponding to n. The specific value corresponding to n may be n or a value converted by a preset algorithm.

[0156] In addition, when the coordinates of the image are set to the x-axis and the y-axis perpendicular to the x-axis, the depth information may be a value corresponding to the z-axis perpendicular to the x-axis and the y-axis. The absolute value of the depth information may increase as the distance between the object and the mobile terminal increases.

[0157] Depth information can be used in various fields. For example, depth information can be used to capture / generate 3D stereo images (stereoscopy), to generate 3D printing data used in 3D printers, or to detect the movement of objects (or subjects) around a mobile terminal.

[0158] The mobile terminal related to the present disclosure may extract depth information of an image received (or photographed) by a camera in various ways. For example, the controller 180 may extract depth information by a stereoscopic vision scheme using at least two cameras to extract depth information, a structured light scheme using light emitting elements arranged to form a preset pattern to extract depth information, a time-of-flight (ToF) scheme that extracts depth information based on the time it takes for light emitted from a light emitting element to be reflected to return, or any combination thereof.

[0159] Hereinafter, extraction of depth information using a structured light scheme among the above schemes will be mainly described.

[0160] The structured light scheme is a scheme of emitting light to an object by controlling a plurality of light emitting elements arranged to have a preset pattern, sensing the light reflected from the object, and then extracting depth information based on the sensed light (or a pattern of the sensed light).

[0161] In detail, the structured light scheme is a scheme for extracting depth information by irradiating light to an object from a plurality of light emitting elements arranged in a preset pattern and calculating a shift amount of returned reflected light relative to the preset pattern (or a shift amount of a reflected light pattern).

[0162] For example, the controller 180 of the mobile terminal related to the present disclosure may control a plurality of light emitting elements arranged in a preset pattern to emit light to an object. Figure 1a The sensing unit 140 senses light reflected and returned from the object.

[0163] Here, the controller 180 may extract depth information of the image received through the camera 121 based on the sensing result. For example, the controller 180 may extract depth information of the image received through the camera 121 by comparing a pattern formed by reflected and returned light with a preset pattern.

[0164] In detail, the controller 180 can extract depth information of the image received by the camera 121 by comparing a preset pattern in which a plurality of light emitting elements emit light toward an object (or a preset pattern in which the plurality of light emitting elements are arranged) with a pattern formed by reflected and returned light (or light spots) and calculating an amount of displacement (or a changed form, a changed distance, a changed direction, etc.) of each reflected and returned light (or light spot) relative to the preset pattern or an amount of displacement of the returned light pattern.

[0165] In another example, in a structured light scheme, the controller 180 may extract depth information of an image received through the camera 121 by comparing a time when light emitted from a plurality of light emitting elements is reflected to return and an intensity of the reflected and returned light.

[0166] To this end, a plurality of light emitting elements may be formed to emit light toward a space corresponding to an image received through the camera 121 .

[0167] The preset pattern may be determined (or set) by the user, or may be predetermined when the product of the mobile terminal is manufactured. In addition, the preset pattern may be changed according to a user request or by the control of the controller.

[0168] In addition, the plurality of light emitting elements may emit infrared light. In addition, the light emitting element may be a laser diode that changes an electrical signal into an optical signal. For example, the light emitting device may be a vertical cavity surface emitting laser (VCSEL).

[0169] In the present disclosure, a structured light scheme can be used to extract depth information of an image through one camera (infrared camera or 3D camera), and even when an object has a single color, depth information can be extracted. In addition, the accuracy of the depth information can be enhanced by combining the structured light scheme with a stereoscopic vision scheme using at least two cameras or combining the structured light scheme with a ToF scheme.

[0170] The ToF scheme may be a scheme that measures depth information of an image by calculating a time when light directly irradiated on an object returns as reflected light.

[0171] A stereoscopic vision scheme may be a scheme in which a plurality of cameras (e.g., two cameras) are symmetrically arranged and the disparity (or difference in distance or space) between an image received by a first camera (e.g., a left camera) among the plurality of cameras and an image received by a second camera (e.g., a right camera) among the plurality of cameras is used to extract depth information of the images received by the cameras.

[0172] A mobile terminal related to the present disclosure may use stereoscopic vision, structured light technology, ToF technology, or a combination of at least two or more of these technologies.

[0173] Figure 2 is a conceptual diagram illustrating a camera and a lighting device provided in a mobile terminal related to the present disclosure.

[0174] like Figure 2 As shown in (a), the mobile terminal related to the present disclosure may have a plurality of cameras 121b and 121c on one surface thereof. Here, one surface of the mobile terminal 100 may be at least one of a rear surface, a front surface, and a side surface of a body of the mobile terminal.

[0175] exist Figure 2 In (a), it is shown that a plurality of cameras 121b and 121c are provided on the rear surface of the body of the mobile terminal.

[0176] In addition, the lighting device 200 of the present disclosure may be provided on one surface on which the plurality of cameras 121 b and 121 c are provided.

[0177] The lighting device 200 may include a plurality of light emitting elements, and as described above, the lighting device 200 may irradiate light having a preset pattern to extract depth information of an image through a structured light scheme. Here, the plurality of light emitting elements (or the plurality of light sources) may be, for example, VCSELs.

[0178] like Figure 2 As shown in (a), the mobile terminal of the present disclosure can extract depth information of an image received through a camera by combining a stereoscopic vision scheme with a structured light scheme using a plurality of cameras 121a and 121b and a lighting device 200 capable of irradiating light of a preset pattern.

[0179] However, not limited to this, although multiple cameras 121a and 121b are set on one surface of the body of the mobile terminal, the mobile terminal 100 of the present disclosure can use any one of the stereoscopic vision scheme, the structured light scheme and the ToF scheme or by combining at least two schemes to extract depth information of the image received by the camera.

[0180] However, it is not limited to this. Figure 2 As shown in (b), the mobile terminal 100 of the present disclosure may use one camera 121 and the lighting device 200, use only the structured light scheme, use only the ToF scheme, or use a combination of the structured light scheme and the ToF scheme to extract depth information of an image received through the camera.

[0181] In addition, as described above in the structured light scheme, the lighting device 200 provided in the mobile terminal 100 related to the present disclosure may irradiate light to form (or have) a predetermined pattern. The lighting device 200 may include a plurality of light emitting elements. Here, the light emitting element may be the above-mentioned VCSEL.

[0182] A plurality of light emitting elements may be formed to have a preset pattern, or only some of the light emitting elements may be turned on to irradiate light of a preset pattern.

[0183] For example, a plurality of light emitting elements (or a die including a plurality of light emitting elements) may be referred to as a VCSEL array.

[0184] The controller 180 of the mobile terminal related to the present disclosure may individually control each of the plurality of light emitting elements (plurality of light sources) included in the lighting device 200. In detail, the controller 180 may individually turn on or off the plurality of light emitting elements provided in the lighting device 200. In addition, the controller 180 may individually control the emission intensity of the plurality of light emitting elements provided in the lighting device (or lighting unit) 200. In addition, the controller 180 may individually control (determine) the emission timing of the plurality of light emitting elements provided in the lighting device 200.

[0185] The lighting devices 200 may be individually turned on or off, vary emission intensity, or change emission timing under the control of the controller 180. Therefore, the pattern of light irradiated from the lighting device 200 (ie, a preset pattern) may vary.

[0186] In this way, in the lighting device 200 included in the mobile terminal of the present disclosure, the pattern of the illuminated light (or the intensity of the light, the viewpoint of the light) can be changed by individually controlling multiple light-emitting elements (multiple VCSELs), and from this perspective, the lighting device 200 can be called active lighting.

[0187] In addition, the lighting device 200 related to the present disclosure may irradiate light (or light spots) of a preset pattern so that the light can be used to extract depth information of an image. Here, the light spot may refer to an area (or point) where an object is irradiated with light or an area (or point) where a mobile terminal (or lighting device 200, camera, or sensing unit) is irradiated with light reflected from an object.

[0188] Here, in the present disclosure, since the plurality of light emitting elements included in the lighting device 200 are laser diodes (e.g., VCSELs), when the plurality of light emitting elements emit light, the light (laser) is irradiated on a narrow area (or point) of the object. Therefore, a light spot can be formed in the object. In addition, in the present disclosure, based on the light (laser) reflected from the object to return to the mobile terminal, the light spot irradiated on the object can be detected.

[0189] In addition, the lighting device 200 may include a diffractive optical element (DOE). The DOE may be configured to diffract light (laser light) output from the light emitting element.

[0190] DOE can diffract the light output from the light emitting element into multiple light beams. In the present disclosure, diffracted light (laser) can be understood as split light, duplicated light, refracted part of light, etc. In the case where one light output from the light emitting element is diffracted (or split) into multiple light beams by DOE, the sum of the intensities of the multiple light beams can be equal to the intensity of the one light.

[0191] In other words, the intensity of each of the plurality of light beams (ie, any one of the plurality of light beams diffracted by the DOE) may be weaker than the intensity of the one light beam before entering the DOE.

[0192] Furthermore, the lighting device of the present disclosure can output a larger number of light beams (light spots) than the number of the plurality of light emitting elements using DOE.

[0193] For example, when the number of multiple light-emitting elements is n and the number of light beams (light spots) output when one light beam passes through the DOE is m, the lighting device 200 of the present disclosure can output n*m light beams (light spots) (or irradiate n*m ​​light beams to the object).

[0194] In the present disclosure, the lighting device 200 may have a plurality of light emitting elements and a DOE, and the DOE may diffract light output from the plurality of light emitting elements so that a predetermined pattern is formed with respect to each of the plurality of light emitting elements.

[0195] That is, the lighting device 200 of the present disclosure may include a DOE for diffracting light so that each light source has a predetermined pattern. In other words, the DOE included in the lighting device 200 may diffract light so that one light beam output from one light emitting element forms a predetermined pattern. Therefore, multiple light beams output from multiple light emitting elements may be diffracted to form a predetermined pattern and pass through the DOE.

[0196] Furthermore, as described above, the mobile terminal according to one implementation of the present disclosure may not only extract depth information related to an image, but also extract or determine the material of an object to be photographed through a camera.

[0197] Hereinafter, a control method of a lighting unit (lighting device) and a camera for extracting or determining a material of an object to be photographed will be described in more detail with reference to the accompanying drawings.

[0198] Figure 3 is a conceptual diagram showing a configuration of a mobile terminal according to one implementation of the present disclosure.

[0199] First, the mobile terminal of the present disclosure may include a lighting unit 124 (or the above-mentioned lighting device 200 ), a camera 121 , and a controller (or control unit) 180 that interacts with and controls the camera 121 and the lighting unit 124 .

[0200] Specifically, the controller 180 may control the lighting unit 124 to irradiate lighting (or light) to a subject to be photographed by the camera 121 , and control the camera 121 to photograph the subject.

[0201] The controller 180 may determine the material of the object based on information about illumination (or light) irradiated to the object to be photographed by the camera.

[0202] Specifically, Figure 3 As shown in (a), the controller 180 may include a lighting control module 181 for controlling lighting and a depth / material calculation module 182 for identifying (calculating) depth information related to an image and at least one of a material of an object included in the image based on information related to lighting (or light) reflected from an object to be received in the camera (or lighting information included in an image captured by the camera).

[0203] like Figure 3 As shown in (a) and (b) of FIG. 1 , the lighting control module 181 and the depth / material calculation module 182 may be configured as independent modules, or may be configured as software components included in the controller 180 .

[0204] like Figure 3 As shown in (b), in this paper Figure 2 The camera 121 disclosed in (a) may be configured as a plurality of cameras 121b and 121c.

[0205] In addition, the lighting unit 124 may include a first light source 124 a configured to irradiate pattern light (or structured light) and a second light source 124 b configured to irradiate uniform light (or surface light).

[0206] The lighting unit 124 may include a first light source (pattern light source) 124 a and a second light source (surface light source) 124 b .

[0207] The first light source (pattern light source) 124a may be configured to irradiate light of a preset pattern.

[0208] The preset pattern refers to a light pattern that a light beam output from a pattern light source irradiates onto an object to be photographed through a position where a plurality of vertical cavity surface emitting lasers (VCSELs) are arranged, an optical replication method of a diffractive optical element (DOE), and the like.

[0209] Specifically, the pattern light source may include a plurality of VCSELs (and a diffractive optical element (DOE)) to emit a plurality of light spots to an object in a preset pattern.

[0210] As an example, a plurality of VCSELs may be disposed on a die in an arrangement corresponding to a preset pattern, so that when the lighting device is not equipped with a DOE, a plurality of light spots illuminate an object in a preset pattern.

[0211] In addition, the second light source (surface light source) 124b may be configured to irradiate uniform light.

[0212] For example, the second light source (surface light source) 124b may uniformly irradiate light to a region (space) where light is output.

[0213] For example, the same amount of light beams may illuminate a first region illuminated by light output from the second light source (surface light source) 124b and a second region different from the first region, and the intensity (or brightness) of light illuminated to each region may be the same.

[0214] like Figure 3 As shown, the lighting unit 124 of the present disclosure may include a pattern light source 124a and a surface light source 124b.

[0215] Here, the pattern light source 124a and the surface light source 124b may output infrared rays. For example, the wavelength of the infrared rays irradiated from the pattern light source 124a and the surface light source 124b may be 940 nm.

[0216] In addition, the camera 121 of the present disclosure may include a high-resolution sensor. For example, the high-resolution sensor may be an infrared sensor.

[0217] When the high-resolution sensor is an infrared sensor, the camera 121 of the present disclosure may be referred to as an infrared camera (or IR camera).

[0218] The high-resolution sensor may be configured to sense only the light output from the lighting unit 124 (or the wavelength of the illumination light irradiated from the lighting unit 124 ).

[0219] For example, when the wavelength of infrared rays output from the pattern light source 124 a and the surface light source 124 b is 940 nm, the high-resolution sensor may receive only light of the 940 nm wavelength.

[0220] In addition, the controller 180 of the present disclosure may include a lighting control module 181 that controls the lighting unit 124 and a depth / material calculation module 182 that controls a high-resolution sensor (or camera 121 ).

[0221] That is, the operations / functions / controls performed by the lighting control module 181 and the depth / material calculation module 182 may be understood to be performed by the controller 180 .

[0222] On the other hand, the pattern light source 124a, the surface light source 124b, and the lighting control module 181 may be electrically connected to or disconnected from the mobile terminal, and may be understood to be included in a physically detachable lighting device (or lighting module).

[0223] In addition, the high-resolution sensor 121 and the depth / material calculation module 182 may be electrically connected to or disconnected from the mobile terminal and may be understood as being included in a physically detachable camera (or camera module).

[0224] Figure 4 is a conceptual diagram illustrating a pattern light source according to one implementation of the present disclosure.

[0225] Reference Figure 4 , the first light source (pattern lighting) 124 a of the lighting unit 124 according to one implementation of the present disclosure may include a printed circuit board 900 .

[0226] The printed circuit board 900 may include a printed circuit board (PCB) or a printed wiring board (PWB).

[0227] The printed circuit board 900 may be provided with a circuit printed (placed, mounted) thereon and capable of transmitting an electrical signal. At least one component shown in FIG. 1 may be provided on the printed circuit board 900, and the component may be configured to send and receive an electrical signal.

[0228] The first light source 124 a may include a sub-mount (not shown) mounted (arranged, coupled) on the printed circuit board 900 .

[0229] In addition, the first light source 124a related to the present disclosure may include a die 210 mounted on a submount and having a plurality of light emitting elements.

[0230] The sub-mount may perform a heat dissipation function. Specifically, the sub-mount may be configured to dissipate heat generated by the tube core 210 disposed on the sub-mount (specifically, heat generated due to light emission of a plurality of light-emitting elements disposed on the tube core 210). For example, the sub-mount may be made of a heat dissipation material and may be referred to as a heat sink.

[0231] The die 210 may be provided with a plurality of light emitting elements. For example, each of the plurality of light emitting elements may be a laser diode or a VCSEL.

[0232] The first light source 124a according to the present disclosure may include a pad inserted into a hole formed in the sub-mount, connected to the die 210 in a wire bonding manner, and electrically connected to the printed circuit board 900 so that the die 210 and the printed circuit board 900 are electrically connected through the sub-mount.

[0233] That is, refer to Figure 4 According to the present disclosure, the first light source 124a may include a printed circuit board 900, a sub-mount mounted on the printed circuit board 900, a tube core 210 mounted on the sub-mount and having a plurality of light-emitting elements, and a pad inserted into a hole formed in the sub-mount, connected to the tube core 210 in a wire bonding manner and electrically connected to the printed circuit board 900, so that the tube core 210 and the printed circuit board 900 are electrically connected through the sub-mount.

[0234] The number of wire bonds, holes, and pads provided may be multiple. For example, the number of holes and pads may correspond to the number of wire bonds on die 210.

[0235] The sub-mount may be provided with a plurality of holes. The holes formed in the sub-mount may be formed through the sub-mount. In one example, the holes may include through holes.

[0236] In each hole, a pad may be inserted (placed, arranged). The pad may be made of a conductive material. The pad may be referred to as a bonding pad.

[0237] The pads may be inserted into holes formed through the submount to connect to the die 210 via wire bonding.

[0238] The pads may be inserted through the holes of the submount to be electrically connected to the printed circuit board 900 .

[0239] The pads inserted through the holes formed through the sub-mount may allow the die 210 and the printed circuit board 900 to be electrically connected to each other through the sub-mount.

[0240] Specifically, due to the pads inserted through the sub-mount, the plurality of light emitting elements disposed on the die 210 and the printed circuit board 900 (circuitry disposed thereon) may be electrically connected through the sub-mount.

[0241] In addition, the area of ​​the submount may be larger than the area of ​​the die 210. This is for wire bonding between the die 210 and pads inserted through holes formed through the submount.

[0242] On the other hand, Figure 4 As shown, the first light source 124a according to the present disclosure may include a bracket 930 disposed on the printed circuit board 900 to surround (cover, include, accommodate) the die 210 and the sub-mount.

[0243] The support 930 may be formed to have an inner space between the die 210 and the support 930. The inner space may preferably be in a vacuum state without air, but may not be in a vacuum state.

[0244] The bracket 930 may be formed to surround (cover, include, accommodate) the die 210 , the sub-mount, the pads inserted through the holes formed through the sub-mount, and the wire bonding connecting the die 210 and the pads.

[0245] The bracket 930 may function as a housing because the bracket 930 surrounds the die 210 , the submount, the wire bonds, the pads, etc. Furthermore, the bracket 930 may be configured as a single housing or a combination of multiple housings.

[0246] Furthermore, an inner space may be formed between the support 930 and the die 210 .

[0247] The first light source 124a of the present disclosure may include a diffractive optical element (DOE) 220 disposed in an inner space between the bracket 930 and the tube core 210 and mounted (arranged) by the bracket 930. The DOE 220 may be located in the optical axis direction of the plurality of light emitting elements so that light output from the plurality of light emitting elements of the tube core 210 can pass therethrough.

[0248] The DOE 220 may be selectively set or not set according to the user's design.

[0249] In addition, instead of a diffractive optical element, the first light source 124a of the present disclosure may use a refractive optical element (e.g., at least one of a grating unit array (GCA), a reflector / prism array, a microlens array, a fly-eye lens, and a birefringent element, or a combination of two of them) to diffract or split the light emitted from the light-emitting element into multiple light beams.

[0250] The following description of DOE may also be inferred and applied in the same / similar manner to the case where a refractive optical element is used or the case where a DOE and a refractive optical element are combined.

[0251] The bracket 930 of the first light source 124a of the present disclosure may be provided with a lens 230a through which light output from a plurality of light emitting elements provided on the die 210 is transmitted. That is, the lens 230a may be installed (placed) in the bracket 930 and located in the optical axis direction of the plurality of light emitting elements so that light emitted from the plurality of light emitting elements (or light passing through the DOE 220) may be transmitted therethrough.

[0252] Light output from the plurality of light emitting elements disposed on the die 210 may be diffracted (split or replicated) by the DOE 220, and then the diffracted (split or replicated) light may be incident on and transmitted through the lens 230a to be output to an external space.

[0253] The support 930 may be provided with a lens 230a through which light output from a plurality of light emitting elements provided on the die 210 is transmitted. The lens 230a may include at least one of a microlens array, a reflective lens, a collimating lens, a grating cell array (GCA), a reflector / prism array, a fly-eye lens, a birefringent element, and the like.

[0254] The lens 230 may allow light output from the plurality of light emitting elements 210 a , 210 b , 210 c , and 210 d or light having passed through the DOE 220 to be transmitted therethrough.

[0255] The lens 230 provided in the lighting device 200 of the present disclosure may be a general lens 230 a or a telecentric lens.

[0256] Figure 5 is a conceptual diagram illustrating a control method of a lighting unit according to the present disclosure.

[0257] The controller 180 may capture a 3D image using the lighting unit 124 and the camera 121 .

[0258] Here, capturing a 3D image should be understood as a concept including extracting depth information, extracting edge information, and extracting brightness information.

[0259] In addition, the controller 180 may extract / decide / determine the material of the object to be photographed using the lighting unit 124 and the camera 121 .

[0260] The controller 180 (or the lighting control module 181 ) may independently control the pattern light source 124 a and the surface light source 124 b .

[0261] In addition, the controller 180 may control the pattern light source 200 a and the surface light source 200 b in a preset manner in order to extract high-resolution depth information or a material of an object using one camera (eg, an infrared camera).

[0262] For example, the controller 180 may alternately turn on the first light source (the pattern light source 124 a ) and the second light source (the surface light source 124 b ).

[0263] like Figure 5 As shown, when the first light source (pattern illumination light) 124a is turned on within a first time (segment), the controller 180 may turn off the second light source (uniform illumination light, surface illumination) 124b, and when the second light source 124b is turned on within a second time (segment) after the first time period, the first light source 124a is turned off.

[0264] The controller 180 may control the lighting unit 124 so that pattern light irradiated from the pattern light source 124 a and surface light irradiated from the surface light source 124 b may be alternately output for each frame in which the camera 121 (or a high-resolution sensor) receives (acquires) an image.

[0265] For example, the controller 180 may control the surface light source 124 b not to emit light while the pattern light source 124 a emits light, and control the pattern light source 124 a not to emit light while the surface light source 124 b emits light.

[0266] For example, the controller 180 may control the lighting unit 124 so that the first light source 124 a emits light when the camera 121 captures a first frame, and the second light source 124 b emits light when the camera 121 captures a second frame after capturing the first frame.

[0267] In detail, while photographing the first frame, the pattern light source 124a may emit light (on) and the surface light source 124b may not emit light (off). On the other hand, while photographing the second frame, the surface light source 124b may emit light and the pattern light source 124a may not emit light.

[0268] Subsequently, the controller 180 may control the pattern light source 124 a to emit light again when the camera 121 captures a third frame after capturing the second frame, and control the surface light source 124 b to emit light when the camera 121 captures a fourth frame after capturing the third frame.

[0269] Similarly, the controller 180 may control the surface light source 124 b not to emit light while photographing the third frame, and control the pattern light source 124 a not to emit light while photographing the fourth frame.

[0270] On the other hand, the controller 180 may control the pattern light source 124a to emit light during at least part of the time (period) of capturing the first frame (third frame), and control the surface light source 124b to emit light during at least part of the time of capturing the second frame (or fourth frame).

[0271] For example, the controller 180 may control the lighting unit 124 so that the pattern light source 124a or the surface light source 124b emits light during the entire time of shooting one frame, or any one of the pattern light source 124a and the surface light source 124b does not emit light during a predetermined time (segment). The at least part of the time may refer to the time obtained by subtracting the predetermined time from the total time of shooting one frame.

[0272] The reason why both the pattern light source 200a and the surface light source 200b do not emit light for a predetermined time during the time for photographing one frame is to synchronize the time when the aperture of the camera is opened with the time when the illumination is turned on and to prevent interference between light beams emitted from the surface light source and the pattern light source.

[0273] Thereafter, the controller 180 may determine / extract / decide the material of the object included in the image using a preset algorithm based on a difference between images acquired by the camera 121 while alternately turning on and off the first light source (pattern light source) and the second light source (surface light source).

[0274] As described above, the mobile terminal of the present disclosure may extract depth information regarding an image photographed by a camera using structured light and a surface light source.

[0275] In addition, the present disclosure may provide a mobile terminal including a camera and a lighting unit configured to distinguish (determine, recognize) a material of an object (ie, a subject to be photographed by a camera).

[0276] To this end, the present disclosure may determine / extract / decide the material of an object by considering the power, wavelength, beam profile (light distribution), etc. of the lighting (i.e., light or auxiliary light) irradiated from the lighting unit, and thus determine whether the object is a real object or a false object (e.g., whether the object is a printed picture).

[0277] There are multiple factors for identifying an object. Color, size, material (or texture), and shape can correspond to these factors.

[0278] In addition, the mobile terminal of the present disclosure is intended to further provide a function of determining a material in addition to the function of an existing camera.

[0279] To this end, the present disclosure may divide the auxiliary light into multiple segments (or groups or units) so that the lighting unit emits lighting light of different wavelengths in each time (segment).

[0280] The sensor (camera) can sequentially acquire the object's response to light beams with wavelengths 1, 2, 3 and 4.

[0281] Since the material of an object (subject) responds to the wavelength of the light beam, the material can be identified from the responses to multiple representative wavelengths.

[0282] In addition, the mobile terminal of the present disclosure may determine the material of an object from a response to the power of the auxiliary light of a specific wavelength by irradiating illumination light (auxiliary light) of a specific wavelength from the illumination unit.

[0283] That is, the mobile terminal of the present disclosure adopts a method of identifying (determining) a material by emitting auxiliary light of different powers in respective times (periods) and checking a response to the auxiliary light.

[0284] In another method, the mobile terminal of the present disclosure may recognize (determine) the material (or texture) of the object by sequentially using different auxiliary light beams (ie, uniform illumination light and pattern illumination light).

[0285] The mobile terminal of the present disclosure can obtain texture from the object's response (reaction) to (with) uniform light such as an LED, obtain a beam profile (optical profile) from patterned lighting light, and combine the obtained texture and beam profile to determine (extract, identify) the material of the object.

[0286] The mobile terminal of the present disclosure may also determine the material of an object using deep learning and artificial intelligence (AI) from a single RGB camera.

[0287] The mobile terminal of the present disclosure may improve the accuracy of determining the material of an object by combining at least two of the above methods.

[0288] A mobile terminal including a depth camera (illumination unit and camera) to which the technology of the present disclosure is applied can determine (extract or decide) the material of an object while showing a depth performance (depth information recognition performance) superior to that of existing cameras. Therefore, the mobile terminal of the present disclosure can increase the reliability of recently emerging financial, security, and object recognition.

[0289] Return to reference Figure 3 According to the present disclosure, a mobile terminal including a depth camera capable of determining (extracting or identifying) materials may include a camera 121, a lighting unit 124 (or an auxiliary light unit), a lighting control module 181 (or a control unit), and a depth / material calculation module 182 (or a calculation unit).

[0290] The lighting unit 124 (or the auxiliary light unit) may include a uniform auxiliary light source (or the second light source) and a pattern auxiliary light source (or the first light source). Alternatively, the uniform auxiliary light source may not be included.

[0291] The uniform auxiliary light source is illumination that emits uniform light, and may be configured as LED, LED+DOE, VCSEL (LD)+DOE, or the like.

[0292] Patterned auxiliary light can be configured as VCSEL, illumination lens and / or DOE (optional).

[0293] A method of implementing the auxiliary light will be described in more detail later with reference to the accompanying drawings.

[0294] The camera unit (or camera 121) may include sensors (may be configured as sensors), and each sensor may include an image sensor, an optical filter, and a lens. The camera unit may use one to four sensors.

[0295] For example, the sensor of the camera unit may be configured by using a general monochrome sensor and an RGB sensor for a short wavelength at the same time or using only one of a sensor for a short wavelength or an RGBIR sensor.

[0296] Sensors can be configured as general sensors, high-speed sensors with high frame rates, event sensors, etc.

[0297] The auxiliary light source (first light source) may be provided in multiple segments.

[0298] In the case of a general sensor with a low frame rate, the frame rate can be delayed when obtaining information by projecting a pattern over the entire area.

[0299] Therefore, the camera of the mobile terminal of the present disclosure may be provided with a high-speed sensor having a frame rate of several hundred fps (higher than the frame rate of several tens of fps equipped in general sensors) or a high-speed event sensor that recognizes information per pixel.

[0300] Furthermore, in the case of a general sensor, a global shutter sensor may be provided instead of a rolling shutter sensor. Thus, the present disclosure may be used not only for applications of depth cameras but also for applications of general 2D cameras and IR cameras.

[0301] In addition, since depth information as well as material information may be obtained in view of the use of pattern auxiliary light, the present invention may be advantageous in the application of a depth camera.

[0302] Additionally, the sensor may generally be configured as a short wavelength monochromatic sensor.

[0303] The optical filter of the sensor may be located between the sensor and the lens. An optical filter suitable for the wavelength of the uniform illumination light and the pattern auxiliary light may be used.

[0304] For example, if the uniform auxiliary light uses a wavelength of 940 nm and the pattern auxiliary light uses a wavelength in the range of 430 nm to 650 nm, the filter of the sensor may be configured as a broadband wavelength transmission bandpass filter covering the entire wavelength of 430 nm to 940 nm.

[0305] The control unit (controller 180) may perform lighting control, camera control, and lighting camera control.

[0306] The lighting control may be performed to control each segment of the auxiliary light source having a plurality of segments to be turned on / off in sequence.

[0307] When LED is used, the control unit (controller 180) can control the LED uniform auxiliary light and pattern auxiliary light to turn on / off in sequence, and control the sensor to receive such light. LED light as a uniform surface light source can be used to obtain the shape, features and 2D original image of the object. On the other hand, pattern auxiliary light as a beam profile (light distribution) of patterns and light spots can be used to obtain depth information.

[0308] Both uniform auxiliary light and patterned auxiliary light can be used to obtain material information.

[0309] Illumination camera control can be performed to synchronize the sensor exposure time with the on / off time of the illumination.

[0310] Camera control can be performed to control automatic exposure such as AE, and in the case of a stereo camera, to control the synchronization between the two sensors.

[0311] The calculation unit (depth / material calculation module 182) may acquire material and 3D depth images using signals obtained from sensors based on at least one of pattern shift, size, intensity, and distribution of light points (patterns) and features (feature points) of a 2D original image.

[0312] Image synthesis from RGB images may also be performed using a separately provided RGB camera or using an RGBIR sensor.

[0313] Reference Figure 4 , the pattern auxiliary light (first light source) for determining or extracting a material according to the present disclosure may include a VCSEL, a DOE, and a lens.

[0314] The uniform auxiliary light (second light source) may be configured as an LED (or VCSEL) diffuser.

[0315] Figure 4The first light source shown may include a VCSEL (LED or LD) lens, a VCSEL (LED or LD) diffraction element (the diffraction element is a diffuser, a DOE, a microlens array, a grating, a HOE, etc.), a VCSEL (LED or LD) lens diffraction element and a VCSEL (LED or LD) diffraction element lens (the lens is a refractive lens and a reflector / prism / dividing mirror, etc.).

[0316] The auxiliary light for determining the material of the object may be configured as uniform auxiliary light (LED or LD) or / and pattern auxiliary light (VCSEL or LD). The auxiliary light may be used in plural. The auxiliary light may include AF or / and aperture and OIS actuators.

[0317] VCSELs create the basic pattern and create spots of light.

[0318] VCSELs can project patterns onto objects by creating the viewing angle of a lens.

[0319] Diffractive elements can increase the number of light patterns by replicating the basic pattern of the VCSEL.

[0320] The controller 180 may determine the material of the object in consideration of the following items: For example, the controller 180 may determine the material of the object based on reflectivity.

[0321] In order to determine the material based on the reflectivity, it is necessary to know the reflectivity at the reference distance. Information about the reflectivity at the reference distance may be pre-stored in a memory.

[0322] The reflectivity (response signal of the sensor) can be determined by the following factors.

[0323] - Lighting parameters such as wavelength, power intensity, frequency, etc.

[0324] -Measurement conditions: distance, angle, ambient light

[0325] -Measure objects: material (or texture), color, shape

[0326] -Active lighting is more advantageous than passive methods that do not use lighting. (It is necessary to know the amount and wavelength of lighting (light).)

[0327] - The change in reflectivity with angle should also be considered.

[0328] - In the case of the face, the reflectivity of the skin is different from the reflectivity of the pupil.

[0329] For example, it is necessary to consider data based on the characteristics of a particular subject (human skin, pupils, hair, and ethnicity).

[0330] On the other hand, it is not easy to determine the material of an object in various environments. However, in the present disclosure, the material can be determined by the relatively different responses to the wavelength of the light beam.

[0331] An example of material classification according to reflectivity is as follows.

[0332] -0%-25%: Black

[0333] -26%-50%: Stone, Brick

[0334] -51%-75%: cotton, skin

[0335] -76%-100%: Glass, white

[0336] * Applying the example of material classification based on reflectivity to vehicles, the following results are obtained.

[0337] -0%-40%: Safety belt

[0338] -40%-50%: face, skin

[0339] -50%-70%: cotton, seat cover

[0340] -60%-80%: Seat cover

[0341] Hereinafter, a method of controlling pattern illumination (first illumination) for determining a material of an object will be described in more detail.

[0342] Figure 6 , Figure 7 , Figure 8 , Fig. 9 , Fig.10 , Fig.11 , Fig.12 , Fig.13a , Fig.13b , Fig.14 , Fig.15 , Fig.16 , Fig.17 , Fig.18 , Fig.19 , Fig. 20 and Fig.21 is a conceptual diagram illustrating a method of controlling lighting used by a mobile terminal to determine (recognize) a material of an object to be photographed.

[0343] Reference Figure 6 , the first light source 124a may include a plurality of light emitting elements 600a, 600b, 600c and 600d, 600.

[0344] The plurality of light emitting elements 600 may be grouped into a plurality of groups 610 - a , 610 - b , . . . , 610 - n .

[0345] A group may refer to the above-mentioned segment.

[0346] Each group may include the same number of light emitting elements 600a, 600b, 600c, 600d (four).

[0347] The controller 180 may control the light emitting elements included in the plurality of groups 610 - a , 610 - b , . . . , 610 - n in a preset manner.

[0348] For example, Figure 6 As shown, the controller (control unit) 180 can control the lighting unit 124 so that the first light-emitting element 600a (for example, the light-emitting element located at position 1x1) and the second light-emitting element 600b (for example, the light-emitting element located at position 1x2) included in each of the multiple groups 610-a, 610-b, ..., 610-n output light beams of different wavelengths.

[0349] For example, the controller 180 may control the lighting unit 124 so that the first light emitting element 600a, the second light emitting element 600b, the third light emitting element 600c and the fourth light emitting element 600d belonging to each group emit illumination light of the first wavelength, the second wavelength, the third wavelength and the fourth wavelength, respectively.

[0350] Reference Figure 6 , the controller 180 may group the light emitting elements used to determine (extract) the material into n segments (groups), and may control the corresponding n segments to project light sequentially with a time difference.

[0351] At this time, each segment (group) can project light of a different wavelength.

[0352] The controller 180 may determine the material of the object using the fact that each material exhibits different reflectivity with respect to wavelengths.

[0353] The controller 180 may determine the material according to the response to each region / material by controlling the segments 1 to 4 to project (irradiate, emit) light to a specific object / material (subject) in a sequential manner.

[0354] The controller 180 may also determine the material by examining the response of only one of segments 1-4 or the responses of multiple segments 1-4.

[0355] Additionally, the controller 180 may use AI algorithms / data based on RGB or IR raw images to determine the material of objects that exhibit similar reflection / scattering responses but are made of different materials.

[0356] In the mobile terminal of the present disclosure, the number of segments (groups) and the number of power adjustment steps may be randomly set.

[0357] Multiple light emitting elements may be arranged in a random pattern as well as Figure 6Regular pattern shown.

[0358] Each segment (group) may have any shape other than a rectangular shape.

[0359] Segments (groups) may have regions that are independent of each other or that intersect each other.

[0360] Each of the examples described herein can be used independently, and some examples can be used in combination.

[0361] exist Figure 6 In , only the VCSEL structure is described, but the DOE and the lens can be placed on the VCSEL to build auxiliary lighting together.

[0362] like Figure 7 As shown, the controller 180 may group the plurality of groups 610 - a , 610 - b , . . . , 610 - n into a plurality of units 900 a , 900 b , 900 c , and 900 d .

[0363] Here, the unit may refer to the above-mentioned segments or a medium group (or a large group) in which those groups are sub-grouped.

[0364] The controller 180 may control the lighting unit 124 to emit light for each unit without time overlap.

[0365] The plurality of units 900a, 900b, 900c, and 900d may include the same number of groups.

[0366] The controller 180 may control the lighting unit 124 so that one of the plurality of units (eg, 900a) is turned on while light emitting elements included in the other units 900b, 900c, and 900d do not emit light.

[0367] That is, the controller 180 may control the lighting units so that a plurality of units irradiate light with a time difference, rather than controlling the entire light emitting elements.

[0368] In the case of controlling the first unit 900a among the plurality of units, as Figure 7 As shown, the controller 180 may control the lighting unit so that the first light emitting element and the second light emitting element included in each of the plurality of groups included in the first unit 900a output light beams of different wavelengths.

[0369] Furthermore, while the first light emitting element emits light, the second to fourth light emitting elements may not emit light, and thereafter, the second light emitting element, the third light emitting element, and the fourth light emitting element may sequentially emit light.

[0370] As another example, in Figure 6 In the case of , wavelengths 1 to 4 can be alternately turned on / off for the entire area (without segments). Figure 7In the case of , groups of respective output wavelengths 1 to 4 may be grouped into units (segments) 1 to 4, which may then be alternately turned on / off for each wavelength.

[0371] Therefore, the controller 180 can control those segments so that units (segments) 1 to 4 of wavelength 1 are turned on sequentially, units 1 to 4 of the second wavelength 2 are turned on sequentially, units 1 to 4 of the third wavelength 3 are turned on sequentially, and then units 1 to 4 of the fourth wavelength 4 are turned on sequentially.

[0372] Since the entire area is divided into segments (cells), illumination control can be performed for each area / wavelength and the material of a specific area can be determined (extracted, detected, identified).

[0373] like Figure 8 As shown, the controller 180 may control the lighting unit so that the first light emitting elements 600a and the second light emitting elements 600b included in the plurality of groups 610-a, 610-b, ..., 610-n respectively emit lights of different powers.

[0374] For example, the controller 180 may control the lighting unit so that the first light-emitting elements 600a included in multiple groups output light of a first power for low-reflection materials, the second light-emitting elements 600b output light of a second power for medium-reflection materials (26% to 50%), the third light-emitting elements 600c output light of a third power for another medium-reflection material (51% to 70%), and the fourth light-emitting elements 600d output light of a fourth power for high-reflection materials (76% to 100%).

[0375] In this case, the controller 180 may control the first to fourth light emitting elements to sequentially emit light in such a manner that the first to fourth light emitting elements output light without temporal overlap.

[0376] In addition, the controller 180 can control the camera to shoot the first frame when the first light emitting element outputs light, shoot the second frame when the second light emitting element outputs light, shoot the third frame when the third light emitting element outputs light, and shoot the fourth frame when the fourth light emitting element outputs light.

[0377] That is, the controller 180 may determine the material of the object by controlling groups each having a different light power intensity.

[0378] The controller 180 may determine the material of the object using reflection / scattering differences for respective materials.

[0379] The controller 180 may determine the material according to the response to each region / material by controlling the first to fourth light emitting elements to project (irradiate, emit) toward a specific object / material in a sequential manner.

[0380] The controller 180 may determine the material by checking a response of only one of the first light emitting elements of each of the first to fourth light emitting elements or a plurality of responses of the first to fourth light emitting elements.

[0381] Figure 8 An example is shown in which the first to fourth light emitting elements emit light to the entire area in a divided manner based on power.

[0382] The number of light emitting elements included in each group and the number of power adjustment steps may be randomly set.

[0383] Similarly, if Fig. 9 As shown, the controller 180 may group the plurality of groups into a plurality of units 900a, 900b, 900c, and 900d, and control the light emitting elements to emit light for the respective units without time overlap.

[0384] exist Fig. 9 In the case of Figure 8 The illustrated method of controlling the lighting units controls the light emitting elements for each unit 900a, 900b, 900c and 900d, so that the first to fourth light emitting elements output light of different powers in different time periods.

[0385] As another example, in Figure 8 In the case of , wavelengths 1 to 4 can be alternately turned on / off for the entire area (without segments). Fig. 9 In the case of , groups of respective output wavelengths 1 to 4 may be grouped into units (segments) 1 to 4, which may then be alternately turned on / off for each wavelength.

[0386] Therefore, the controller 180 can control those segments so that units (segments) 1 to 4 of power 1 are turned on sequentially, units 1 to 4 of power 2 are turned on sequentially, units 1 to 4 of power 3 are turned on sequentially, and then units 1 to 4 of power 4 are turned on sequentially.

[0387] In addition, if Fig.10 As shown, the controller 180 may control all of the plurality of light emitting elements 600 to emit light at different powers over time.

[0388] For example, Fig.10 As shown in (a), the controller 180 can control all the multiple light emitting elements to emit light of the first power within the first time (segment). Fig.10 As shown in (b), the controller 180 may control all of the plurality of light emitting elements to emit light at a second power different from the first power within a second time period after the first time period elapses.

[0389] For example, Fig.10As shown in (a) to (d) of FIG. 1 , the controller 180 may control the lighting unit 124 so that the entire light emitting element gradually outputs light of weaker power as time passes.

[0390] The controller 180 may project light on the entire area of ​​the object while sequentially changing the power.

[0391] A structure showing four physically different types of transmitters with different optical powers Figure 6 Differently, the controller 180 can adjust the power over the entire area by controlling the driver to change the input. Therefore, the entire area can use only one type of transmitter.

[0392] The controller 180 may control the power for each time (period). The number of power adjustment steps may be randomly set. Not only regular patterns but also random patterns may be available.

[0393] like Fig.11 As shown, the controller 180 may group the plurality of light emitting elements into a plurality of units 900a, 900b, 900c, and 900d, and control the lighting unit 124 to emit light at different powers for each unit based on time.

[0394] For example, Fig.11 As shown in (a), the controller 180 can control the lighting unit to output light in sequence for each of the multiple units (segments) 900a, 900b, 900c, and 900d in a manner such that the light of the first power is output during the first sequence of light emission, the light of the second power is output during the second sequence of light emission, the light of the third power is output during the third sequence of light emission, and the light of the fourth power is output during the fourth sequence of light emission.

[0395] and Fig.10 The difference of the structure shown is that the whole area is divided into 4 sections (units). Similarly, the controller 180 can adjust the power of each section (unit) by changing the input. The number of sections (units) can be adjusted randomly.

[0396] On the other hand, Fig.12 As shown, the controller 180 may control the lighting unit so that the first light emitting elements 600a and the second light emitting elements 600b included in the plurality of groups 610-a, 610-b, ..., 610-n emit light with different beam profiles (light distributions), respectively.

[0397] For example, from Fig.13a The Gaussian beam output by the first light emitting element shown in (a) and the Gaussian beam output by the first light emitting element shown in (a) Fig.13a The beam profile (light distribution) of the annular light beam output by the second light-emitting element (or the third light-emitting element) shown in (b) may be different.

[0398] The controller 180 may control the lighting unit so that the first light emitting elements included in the plurality of groups output light having a first beam profile (eg, a Gaussian beam), and the second light emitting elements included in the plurality of groups output light having a second beam profile (eg, a ring beam).

[0399] like Fig.13b As shown in (a) to (d) of FIG. 1 , the beam profile may be different according to the type of material, and the controller 180 may determine / decide the material of the object based on the beam profile included in the image captured by the camera (sensor).

[0400] The controller 180 may determine the material of the object using different responses according to the beam profile of the light beam incident on the material.

[0401] The controller 180 may check the material / distance based on a beam profile of a light spot in which light reflected for each material is received in the sensor.

[0402] The controller 180 may sequentially irradiate light beams 1 to 4 (light output from the first to fourth light emitting elements) having different beam profiles onto the object and acquire lighting information accordingly.

[0403] Figure 12 to Figure 13b It is exemplarily shown that light beams 1 to 4 (light output from the first to fourth light emitting elements) are respectively Gaussian beam 1, Gaussian beam 2, ring beam 1 and ring beam 2. Other types of light beams may be selected.

[0404] The controller 180 may determine the material by checking the response of only one of the beams 1 to 4 or the responses of multiple beams 1 to 4.

[0405] In addition, the controller 180 may control the lighting unit 124 so that the first light emitting element and the second light emitting element included in each of the plurality of groups emit light having different patterns.

[0406] For example, Fig.15 As shown in (a), the controller 180 may control the first light emitting element to output light of a first pattern (cross pattern) to the object, and thereafter control the second light emitting element to output light of a second pattern (dot pattern) to the object.

[0407] Similarly, the controller 180 may control the first to fourth light emitting elements to emit light in sequence without temporal overlap.

[0408] The shapes of the above-mentioned beam profiles and patterns may be different from each other.

[0409] For example, even with the same pattern (circular / elliptical pattern), a beam profile having a difference in light intensity can be formed at the center and the periphery.

[0410] The controller 180 may determine the material of the object by using a response difference (illumination information) according to a specific pattern of the material.

[0411] The controller 180 may sequentially irradiate light of patterns 1 to 4 onto the object and check a response accordingly.

[0412] - The material can be determined by examining the response of only one of the patterns 1 to 4 or the responses of multiple patterns 1 to 4.

[0413] An example of determining the power of the entire area has been given, but the present disclosure is not limited thereto, and an example of segmenting the entire area and determining the power of each segmented area is also possible.

[0414] The controller 180 may use AI algorithms / data based on the RGB or IR raw images to determine the materials of objects that exhibit similar reflection / scattering responses but are made of different materials.

[0415] On the other hand, Fig.16 As shown, the controller 180 may also control the lighting unit so that the first light emitting elements and the second light emitting elements included in the plurality of groups irradiate the same pattern of light with light spots of different sizes.

[0416] For example, Fig.17 As shown in (a) of FIG. 1 , the controller 180 may control the lighting unit so that the first light emitting element emits light of the same pattern with light spots having a first size. Fig.17 As shown in (b), the controller 180 may control the lighting unit so that the second light emitting element emits light of the same pattern with a light spot having a second size.

[0417] The controller 180 may project patterns having the same circular shape and different diameters (sizes) on the object. For example, the patterns are classified into circular pattern 1 (large), circular pattern 2 (large-medium), circular pattern 3 (medium), and circular pattern 4 (small). Light having different sizes may also be generated in the VCSEL itself.

[0418] In this way, the controller 180 can control the lighting unit to irradiate light of different wavelengths, different powers, different beam profiles, light spots of different sizes, or different patterns to the object, and determine the material of the object based on information about the lighting (or light) reflected from the object and received by the sensor (camera).

[0419] In addition, if Fig.18 As shown, the first light source 124a may include an illumination lens 1800 configured to irradiate light of different patterns.

[0420] The mobile terminal of the present disclosure can adjust the size of the light pattern projected onto the object by using the lens 1800 (or / and the diffractive element) of the lighting unit. The passive optical element can be configured by combining the diffractive element (microlens array or DOE) with the lens, or can be configured independently. The passive optical element can be configured to adjust the size of the point projected on the object (reaching the object) differently according to the distance / material.

[0421] like Fig.19 As shown, the first light source 124a may include a variable aperture 1900 configured to irradiate different patterns of light.

[0422] In the mobile terminal of the present disclosure, a variable aperture 1900 may be provided to change the size of the pattern in an open / closed manner and irradiate the pattern on the object. Although light of the same size is emitted from the VCSEL, the size of the light pattern of the entire area can be adjusted by opening and closing the aperture and checking the response (illumination information) of the material.

[0423] The controller 180 can open or close the aperture through different aperture values ​​to adjust the size of the pattern. According to this method, the size of the entire pattern can be reduced or increased at the same time.

[0424] Therefore, the size of the pattern projected onto an object located at the same distance is similar, but the size / beam profile of the spot reaching the sensor may differ depending on the material.

[0425] The controller 180 may control the variable aperture to form a specific pattern size by using the fact that a specific material has an optimal response to a specific pattern size.

[0426] like Fig. 20 As shown in (a) to (c) of FIG. 1 , by changing the size of the light passing area (aperture value) of the variable aperture, the controller 180 can obtain lighting information having different reflectivities according to the material. Therefore, the controller 180 can determine the material of the object based on the lighting information.

[0427] In addition, if Fig.21 As shown, the first light source 124a may include a diffractive optical element (DOE) 2100 configured to irradiate light of different patterns.

[0428] The first light source 124a of the present disclosure may replicate an input light spot using a DOE, and sense an interference pattern generated due to interference between the light spot and a material through a sensor (camera) using coherence between the replicated light spots.

[0429] In the case of a VCSEL light source, light beams with different coherence are projected from multiple emitters. Since the light from each emitter has its own coherence, an interference pattern can be generated when responding to the material.

[0430] The controller 180 may detect the material by replicating a light spot passing through the same single aperture using a DOE to generate a plurality of light spot patterns, projecting the generated light spots on the material, and acquiring an optical pattern of the reflected / scattered light beam using a sensor.

[0431] For example, the DOE may be located between the VCSEL and the lens or on the front surface of the lens. The first light source 124a may be provided with only the VCSEL and the DOE without a lens.

[0432] In addition, if Figure 5 As described in, the mobile terminal of the present disclosure can control the uniform illumination light and the pattern auxiliary light to be alternately turned on / off without overlapping.

[0433] The patterned illumination light may be inspection light irradiated on the surface of the object, and uniform illumination light may be implemented to acquire texture information related to the object from a corresponding wavelength.

[0434] At this time, there may be three or more types of information for determining the material.

[0435] First, texture information about the object itself can be obtained from information obtained during operation with uniform illumination light.

[0436] Secondly, the pattern illumination light can determine the material based on the beam profile information (input light) called inspection light. (The pattern changes depending on the material)

[0437] Third, the brightness difference between the center and periphery of the sensor due to lens shading can be removed by using different images acquired through emission of uniform illumination light and pattern auxiliary light, respectively, thereby more robustly determining the material.

[0438] The present disclosure may provide a new mobile terminal capable of determining a material of an object to be photographed by using lighting.

[0439] The present disclosure may provide a new control method capable of determining the material of an object to be photographed in an optimized manner by varying the pattern of illumination.

[0440] The present disclosure can be implemented as a computer-readable code in a program recording medium. Computer-readable media may include recording devices of all types of data that can be read by a computer system. Examples of these computer-readable media may include hard disk drives (HDDs), solid state disks (SSDs), silicon disk drives (SDDs), ROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, optical data storage elements, etc. In addition, computer-readable media may also be implemented as the format of a carrier wave (e.g., transmission via the Internet). The computer may include a controller 180 of a terminal. Therefore, it should also be understood that, unless otherwise specified, the above-mentioned embodiments are not limited by any of the details described above, but should be interpreted broadly within the scope defined in the appended claims. Therefore, all changes and modifications falling within the scope of the claims or the equivalent scope of these scopes are therefore intended to be covered by the appended claims.

Claims

1. A mobile terminal, comprising: Lighting units; camera; as well as a controller configured to control the lighting unit to irradiate illumination light on an object to be photographed by the camera, and to control the camera to photograph the object irradiated with the illumination light, wherein the controller is configured to determine a material of the object based on information about the illumination light irradiated on the object photographed by the camera, The lighting unit includes a first light source configured to emit patterned light and a second light source configured to emit uniform light, the first light source includes a plurality of light emitting elements, and the plurality of light emitting elements are grouped into a plurality of groups. wherein the controller controls the lighting unit so that the first light emitting element and the second light emitting element included in each of the plurality of groups output light beams with different beam profiles, wherein the controller controls the lighting unit so that the first light emitting element has a first beam profile of a Gaussian beam and the second light emitting element has a second beam profile of a ring-shaped beam, wherein the controller acquires texture information related to the object from information obtained during operation of the uniform light, and wherein the controller controls the camera to sequentially acquire the object's response to the light beams with the different light beam profiles, and Therein, the material of the object is identified from responses to the light beams of different beam profiles.

2. The mobile terminal according to claim 1, wherein: The controller turns on the first light source and the second light source alternately.

3. The mobile terminal according to claim 2, wherein: When the first light source is turned on within a first time period, the controller turns off the second light source, and When the second light source is turned on within a second time period after the first time period has elapsed, the controller turns off the first light source.

4. The mobile terminal according to claim 1, wherein: Each of the plurality of groups includes the same number of light emitting elements.

5. The mobile terminal according to claim 4, wherein: The controller controls the light emitting elements included in the plurality of groups in a preset manner.

6. The mobile terminal according to claim 5, wherein: The controller groups the plurality of groups into a plurality of units, and The controller controls the lighting units to emit light for each unit without overlapping.

7. The mobile terminal according to claim 6, wherein: Each of the plurality of units includes the same number of groups, and Wherein, while controlling one of the plurality of units, the controller controls the lighting unit so that the light emitting elements included in the remaining units do not emit light.

8. The mobile terminal according to claim 1, wherein: The controller controls the lighting unit so that the first light emitting element and the second light emitting element included in each of the plurality of groups output light at different powers to determine the material and color of the object based on reflectivity of the object.

9. The mobile terminal according to claim 4, wherein: The controller controls all of the plurality of light emitting elements to emit light at different powers based on time.

10. The mobile terminal according to claim 9, wherein: The controller controls all of the plurality of light emitting elements to emit light at a first power within a first time period, and After the first time period elapses, the controller controls all of the plurality of light emitting elements to emit light at a second power different from the first power during a second time period.

11. The mobile terminal according to claim 4, wherein: The controller groups the plurality of light emitting elements into a plurality of units, and The controller controls the lighting unit to emit light at different powers based on time for each of the plurality of units.

12. The mobile terminal according to claim 5, wherein: The controller controls the lighting unit so that the first light emitting element and the second light emitting element included in each of the plurality of groups output light in different patterns.

13. The mobile terminal according to claim 5, wherein: The controller controls the lighting unit so that the first light emitting element and the second light emitting element included in each of the plurality of groups irradiate light of the same pattern with light spots of different sizes.

14. The mobile terminal according to claim 1, wherein: The first light source includes an illumination lens configured to illuminate different patterns of light.

15. The mobile terminal according to claim 1, wherein: The first light source includes a variable aperture configured to illuminate different patterns of light.

16. The mobile terminal according to claim 1, wherein: The first light source includes a diffractive optical element configured to irradiate different patterns of light.

17. A method for controlling a mobile terminal, the method comprising the following steps: controlling the illumination unit to irradiate illumination light toward an object to be photographed by the camera; Controlling a camera to photograph the object illuminated by the illumination light; determining a material of the object based on information about the illumination light irradiated to the object photographed by the camera, wherein the illumination unit includes a first light source configured to irradiate pattern light and a second light source configured to irradiate uniform light, the first light source includes a plurality of light emitting elements, and the plurality of light emitting elements are grouped into a plurality of groups; controlling the lighting unit so that the first light emitting element and the second light emitting element included in each of the plurality of groups output light beams having different beam profiles; controlling the lighting unit so that the first light emitting element has a first beam profile of a Gaussian beam and the second light emitting element has a second beam profile of a ring-shaped beam; and controlling the camera to sequentially acquire the object's response to the light beams with the different light beam profiles, and wherein texture information related to the object is obtained from information obtained during operation of the uniform light, and Therein, the material of the object is identified from responses to the light beams of different beam profiles.

Images