Electronic device and control method of electronic device

Abstract

The present invention provides an electronic device and a control method of the electronic device. The electronic device of the present invention projects an optical image to a projection area, outputs infrared light to the projection area, acquires an image of the projection area by using an image sensor provided with a dual band pass filter, acquires a visible light image and an infrared image on the basis of the image of the projection area, recognizes position information of a touch point by a subject in the projection area on the basis of the acquired visible light image and infrared image, and controls the operation of a projection part on the basis of the recognized position information of the touch point.

Description

Electronic device and method of controlling the same

[0001] The disclosed invention relates to an electronic device for acquiring an image and recognizing a subject in the acquired image, and a method for controlling the same.

[0002] Technological advancements have led to the development of a variety of electronic devices equipped with image sensors. Examples include display devices, vision systems, user authentication devices, mobile devices, and projectors.

[0003] Among them, a projector is an electronic device that enlarges and projects image light output from a light source onto a wall or screen through a projection lens.

[0004] Recently, the use of portable projectors has been increasing, and they are evolving from one-way projectors that simply project images (e.g., photos, movies, etc.) to interactive projectors that allow users to manipulate the displayed images by applying touch input on the display surface where the projected images (e.g., UI images) are displayed.

[0005] The most crucial part of these interactive projectors is the technology that recognizes the location of the touch point input by the user.

[0006] Currently, the technology for recognizing the location of touch points in two-way projectors is based on infrared (IR) recognition technology.

[0007] More specifically, the two-way projector recognizes an image projected by the projector using a first image sensor capable of recognizing visible light of the three primary colors (RGB), recognizes an object in the projection area using a second image sensor capable of recognizing infrared (IR), and recognizes the position of a touch point input by a user based on the recognized image and the recognized object.

[0008] One aspect of the disclosed invention provides an electronic device and a control method thereof for acquiring a visible light image and an infrared image using an image sensor equipped with a dual band pass filter and recognizing a subject based on the acquired visible light image and infrared image.

[0009] Another aspect of the disclosed invention provides an electronic device and a control method thereof for acquiring a visible light image and an infrared image by adjusting at least one of an exposure value and a gain value of an image sensor provided with a dual band pass filter.

[0010] An electronic device according to one aspect of the disclosed invention includes a projector that projects a light image onto a projection area; an emitter that outputs infrared light; an image sensor that is provided with a dual band pass filter and acquires an image of the projection area; and a processor that acquires a visible light image and an infrared image based on the image of the projection area received from the image sensor and recognizes positional information of a touch point by a subject within the projection area based on the acquired visible light image and infrared image.

[0011] A dual band pass filter of an electronic device according to one aspect blocks light in a wavelength band corresponding to red visible light and passes light in a wavelength band corresponding to blue and green visible light.

[0012] A dual band pass filter of an electronic device according to one aspect passes light in a wavelength band corresponding to infrared.

[0013] A processor of an electronic device according to one aspect recognizes a boundary of a projection area based on a visible light image, recognizes infrared coordinates and a touch point based on an infrared image, and recognizes position information of a touch point based on the boundary of the recognized projection area, the recognized infrared coordinates, and the touch point.

[0014] A processor of an electronic device according to one aspect obtains an infrared image by removing blue and green from an image of a projection area received from an image sensor.

[0015] A processor of an electronic device according to one aspect obtains a visible light image based on an image received from an image sensor during execution of a first sensing mode, and obtains an infrared image based on an image received from the image sensor during execution of a second sensing mode.

[0016] A processor of an electronic device according to one aspect adjusts an exposure value of an image sensor to a first exposure value during execution of a first sensing mode, and adjusts the exposure value of the image sensor to a second exposure value during execution of a second sensing mode. The first exposure value is an exposure value greater than the second exposure value.

[0017] A processor of an electronic device according to one aspect adjusts a gain value of an image sensor to a first gain value during execution of a first sensing mode, and adjusts a gain value of the image sensor to a second gain value during execution of a second sensing mode. The first gain value is a gain value greater than the second gain value.

[0018] A processor of an electronic device according to one aspect adjusts at least one of an exposure value and a gain value of an image sensor based on acquiring an image using the image sensor.

[0019] A processor of an electronic device according to one aspect recognizes a user input based on position information of a recognized touch point, and controls the operation of a projection unit based on the recognized user input.

[0020] An electronic device according to another aspect includes: an emitter for outputting infrared light; an image sensor having a dual band pass filter and obtaining an image of a projection area; a processor for obtaining a visible light image and an infrared image based on the image of the projection area received from the image sensor and recognizing an object based on the obtained visible light image and infrared image; and a display unit for displaying information about the recognized object.

[0021] A dual band pass filter of an electronic device according to another aspect blocks light in a wavelength band corresponding to red visible light, passes light in a wavelength band corresponding to blue and green visible light, and passes light in a wavelength band corresponding to infrared light.

[0022] A processor of an electronic device according to another aspect obtains an infrared image by removing blue and green from an image of a projection area received from an image sensor, and recognizes the shape of an object based on the obtained infrared image.

[0023] According to another aspect, a processor of an electronic device obtains a visible light image based on an image received from an image sensor during execution of a first sensing mode, and obtains an infrared image based on an image received from the image sensor during execution of a second sensing mode.

[0024] According to another aspect, the processor of the electronic device adjusts the exposure value of the image sensor to a first exposure value during execution of the first sensing mode, and adjusts the exposure value of the image sensor to a second exposure value during execution of the second sensing mode. The first exposure value is an exposure value greater than the second exposure value.

[0025] According to another aspect, a processor of an electronic device adjusts a gain value of an image sensor to a first gain value during execution of a first sensing mode, and adjusts a gain value of the image sensor to a second gain value during execution of a second sensing mode. The first gain value is a gain value greater than the second gain value.

[0026] A method for controlling an electronic device according to another aspect includes projecting a light image onto a projection area, outputting infrared light onto the projection area, acquiring an image of the projection area using an image sensor provided with a dual band pass filter, acquiring a visible light image and an infrared image based on the image of the projection area, recognizing position information of a touch point by a subject within the projection area based on the acquired visible light image and infrared image, and controlling an operation of a projection unit based on the position information of the recognized touch point.

[0027] In another aspect, a dual band pass filter in a control method of an electronic device is a filter that blocks light in a wavelength band corresponding to red visible light, passes light in a wavelength band corresponding to blue and green visible light, and passes light in a wavelength band corresponding to infrared light.

[0028] Recognizing the position information of the touch point includes obtaining an infrared image by removing blue and green from an image of the projection area, recognizing the border of the projection area based on the visible light image, recognizing infrared coordinates and the touch point based on the infrared image, and recognizing the position information of the touch point based on the border of the recognized projection area, the recognized infrared coordinates, and the touch point.

[0029] Obtaining a visible light image and an infrared image based on an image of a projection area includes adjusting an exposure value of an image sensor to a first exposure value and adjusting a gain value of the image sensor to the first gain value during execution of a first sensing mode, adjusting an exposure value of the image sensor to a second exposure value and adjusting a gain value of the image sensor to the second gain value during execution of a second sensing mode, obtaining a visible light image based on an image of a projection area acquired by the image sensor during execution of the first sensing mode, and obtaining an infrared image based on an image of a projection area acquired by the image sensor during execution of the second sensing mode.

[0030] The first exposure value is an exposure value greater than the second exposure value. The first gain value is a gain value greater than the second gain value.

[0031] According to the disclosed invention, the present invention can improve the recognition accuracy of visible light images and infrared images by acquiring visible light images and infrared images using an image sensor provided with a dual band pass filter.

[0032] The present invention can reduce the number of image sensors provided in an electronic device by recognizing visible light images and infrared images acquired by a single image sensor, thereby reducing manufacturing costs.

[0033] The present invention can improve the accuracy of recognizing the position of a touch point because the field of view of an infrared image and the field of view of a visible light image are the same when recognizing the position of a touch point.

[0034] Since the present invention projects images using a laser light source, it can project bright and clear images both during the day and at night, thereby improving the user's image recognition rate.

[0035] The present invention can improve the marketability of electronic devices, increase user satisfaction, enhance user reliability, and secure product competitiveness.

[0036] Figure 1 is an exemplary diagram of an electronic device according to one embodiment.

[0037] Figure 2 is an example diagram of a projection unit provided in an electronic device according to one embodiment.

[0038] FIG. 3 is an exemplary diagram of an image sensor provided in a projector according to one embodiment.

[0039] FIG. 4 is a graph of the quantum efficiency of the received light by wavelength band of an image sensor provided in a projector according to one embodiment.

[0040] FIG. 5 is a graph of transmittance by wavelength band for light passing through a dual band pass filter and a first color filter provided in an image sensor of a projector according to one embodiment.

[0041] Figure 6 is a control configuration diagram of an electronic device according to one embodiment.

[0042] Figures 7 to 9 are exemplary diagrams of image acquisition by a projector according to one embodiment.

[0043] Fig. 10 is an example of recognition of the position of a touch point of a projector according to one embodiment.

[0044] Figure 11 is a control flowchart of an electronic device according to one embodiment.

[0045] Figure 12 is a control configuration diagram of an electronic device according to another embodiment.

[0046] It should be understood that the various embodiments and terms used in this document are not intended to limit the technical features described in this document to specific embodiments, but rather to include various modifications, equivalents, or substitutes of the embodiments.

[0047] In connection with the description of the drawings, similar reference numerals may be used for similar or related components.

[0048] The singular form of a noun corresponding to an item may include one or more items, unless the context clearly indicates otherwise.

[0049] In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" may include any one of the items listed together in that phrase, or all possible combinations thereof.

[0050] Terms such as "first," "second," or "first" or "second" may be used simply to distinguish one component from another and do not qualify the components in any other respect (e.g., importance or order).

[0051] When a component (e.g., a first component) is referred to as being “coupled” or “connected” to another component (e.g., a second component), with or without the terms “functionally” or “communicatively,” it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0052] The terms “include” or “have” are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in this document, but do not preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0053] When a component is said to be “connected,” “coupled,” “supported,” or “in contact with” another component, this includes not only cases where the components are directly connected, coupled, supported, or in contact, but also cases where the components are indirectly connected, coupled, supported, or in contact through a third component.

[0054] When we say that a component is "on" another component, this includes not only cases where the component is in contact with the other component, but also cases where there is another component between the two components.

[0055] The term “and / or” includes any combination of a plurality of related described elements or any one of a plurality of related described elements.

[0056] The operating principle and embodiments of the present invention will be described with reference to the attached drawings below.

[0057] FIG. 1 is an exemplary diagram of an electronic device according to one embodiment, which will be described with reference to FIGS. 2 to 5. In this embodiment, a projector among the electronic devices will be described.

[0058] FIG. 2 is an exemplary diagram of a projection unit provided in a projector according to one embodiment, FIG. 3 is an exemplary diagram of an image sensor provided in a projector according to one embodiment, FIG. 4 is a graph of quantum efficiency by wavelength band of received light of an image sensor provided in a projector according to one embodiment, and FIG. 5 is a graph of transmittance by wavelength band for light transmitted through a dual band pass filter and a first color filter provided in an image sensor of a projector according to one embodiment.

[0059] A projector (1) is an optical device that projects content such as photos, videos, documents, etc. onto a screen, wall, or floor.

[0060] The projector (1) can project images without spatial restrictions, such as at home, at parks, campsites, or beaches. In other words, the projector (1) can be a portable and movable device.

[0061] The projector (1) can recognize touch input by the user and change the image of the content based on the recognized touch input.

[0062] The projector (1) may include a body (10) forming an exterior.

[0063] The main body (10) is provided with a projection unit (20), a light-emitting unit (30), and an image sensor (40), which may be provided to be exposed to the outside.

[0064] The projection part (20) can be provided at the top of the main body (10).

[0065] The projection unit (20) receives content from an external device, enlarges an image of the received content, and outputs an image of the enlarged content as light.

[0066] The projection unit (20) has a first field of view (A1) and can output an image corresponding to the first field of view (A1).

[0067] The image (PI) of the content output by the projection unit (20) is an optical image by laser light and can be projected onto a screen, wall, or floor.

[0068] External devices may include, but are not limited to, personal computers, laptops, tablet PCs, terminals, portable telephones, smart phones, handheld devices, wearable devices, and set-top boxes.

[0069] The external device may further include a removable storage device such as a USB memory stick or a hard disk, and may further include a server.

[0070] An external device can store content and transmit the stored content to the projector (1). The content can include at least one of image information and sound information.

[0071] The projection unit (20) can be classified into CRT, DLP, LCD, and LCoS types depending on the projection method for projecting the image.

[0072] The CRT method amplifies the image of content received from an external device using a cathode ray tube and projects the amplified image onto a screen through a lens.

[0073] The LCD method is a method of obtaining an image by transmitting light emitted from a light source to an LCD panel and projecting the obtained image through a lens.

[0074] LCoS (Liquid Crystal on Silicon) is a method that combines the LCD and DLP methods, generating images using LCD panels and DLP chips and projecting the generated images through a lens.

[0075] The DLP (Digital Light Processing) method processes light from a light source using a DMD (Digital Micromirror Device) chip to create an image and then projects the created image through a lens.

[0076] When the projection unit (20) is of the DLP type, as shown in FIG. 2, the projection unit (20) may include a light source (21), a lens (22), a color wheel (23), a DMD chip (24), and a projection lens (25).

[0077] The light source (21) may include a laser light source. The light source (21) may output light having a brightness higher than a preset reference brightness. Here, the preset reference brightness may be approximately 500 lux.

[0078] The lens (22) may include, but is not limited to, a condensing lens that focuses light emitted from a light source (21) and a shaping lens that transmits light passing through a color wheel (23) to a DMD chip (24).

[0079] The color wheel (23) can transmit the light of the three primary colors (RGB) among the light emitted from the light source (21). The light of the three primary colors transmitted from the color wheel (23) can be incident on the DMD chip (24) after passing through the shape lens.

[0080] The DMD chip (24) includes a plurality of mirrors, and mixes three primary colors of light by turning the plurality of mirrors on and off at high speed to create an image of the content.

[0081] The projection lens (25) enlarges and outputs the image of the content generated from the DMD chip (24). The image (PI) of the content output through the projection lens (25) can be projected onto a screen, wall, or floor.

[0082] The image (PI) of the content generated from the DMD chip (24) may be an optical image formed by laser light.

[0083] Fig. 2 is only an example for explaining the basic operation of the DLP type projection unit (20), and the structure of the DLP type projection unit (2) is not limited thereto.

[0084] The projection method of the projector of this embodiment is not limited to the CRT method, DLP method, LCD method, and LCoS method.

[0085] The light emitting part (30) can be provided at the lower part of the main body (10).

[0086] The light emitting unit (30) emits infrared light (L1) to generate infrared coordinates on a screen, wall, or floor.

[0087] A screen, wall, or floor surface may be a projection area on which an image output from a projection unit (20) is projected. That is, infrared coordinates (20) may be formed in a projection area on which an image output from a projection unit is projected.

[0088] Below, the screen, wall or floor surface is combined and described as the projection area.

[0089] Infrared coordinates may be coordinates for recognizing a touch point by a user's touch input.

[0090] The light emitting unit (30) may include a plurality of infrared emitters (IR Emitters) arranged at regular intervals.

[0091] The image sensor (40) can be provided at the top of the main body (10).

[0092] The image sensor (40) may be provided adjacent to the projection unit (20).

[0093] The image sensor (40) has a second field of view (A2) and can acquire an image corresponding to the second field of view (A2). The second field of view (A2) may be a wider field of view than the first field of view (A1) of the projection unit (20). In other words, the image sensor (40) can acquire an image of an area wider than the projection area on which the image of the content is projected.

[0094] The image sensor (40) acquires images of content and subjects within the second field of view (A2). The subjects may include the user's hand performing touch input, the user's foot, a pen, a pointer, etc.

[0095] The image acquired by the image sensor (40) may include a visible light image.

[0096] The image acquired by the image sensor (40) may include an infrared image.

[0097] More specifically, when a subject (T) exists on the image (PI) of the projection area, infrared light emitted from the light emitting unit (30) is scattered (Scattered or Diffusing) by the subject (T). In this case, the image sensor (40) can obtain an image of infrared light (L2) scattered by the subject (T) and a coordinate image formed by infrared light emitted by the light emitting unit (30).

[0098] The image sensor (40) may include, but is not limited to, a CMOS (Complementary Metal Oxide Semiconductor).

[0099] As illustrated in Fig. 3, the image sensor (40) includes a micro lens unit (41), a color filter unit (42), a photodiode unit (43), a circuit board (44), and a dual band pass filter (45). CMOS is used as an example for the image sensor.

[0100] The micro lens unit (41) collects incident light. The incident light is light incident from the projection area, and may include visible light corresponding to the image of the content, natural light, and infrared light emitted by the light emitting unit (30).

[0101] The micro lens unit (41) may include a plurality of micro lenses.

[0102] The number of microlenses can vary depending on the application and needs. Furthermore, multiple microlenses can be arranged in various configurations on the same plane. For example, multiple microlenses can be arranged in a horizontal or vertical row, or in a horizontal and vertical matrix.

[0103] In this embodiment, three micro lenses will be described. That is, the micro lens section (41) may include a first micro lens (41a), a second micro lens (41b), and a third micro lens (41c).

[0104] The color filter section (42) filters the light collected by the micro lens section (41).

[0105] The color filter section (42) can only pass light of a predetermined wavelength band among the light collected by the micro lens section (41).

[0106] The predetermined wavelength bands may include a first wavelength band, a second wavelength band, and a third wavelength band. The first, second, and third wavelength bands may be different wavelength bands.

[0107] The first wavelength band may include a wavelength band corresponding to infrared light, the second wavelength band may include a wavelength band corresponding to green light, and the third wavelength band may include a wavelength band corresponding to blue light.

[0108] For example, the first wavelength band may include a wavelength band from 850 nm to 950 nm. The second wavelength band may include a wavelength band from 500 nm to 565 nm. The third wavelength band may include a wavelength band from 400 nm to 485 nm.

[0109] The color filter unit (42) may include a first color filter (42a), a second color filter (42b), and a third color filter (42c).

[0110] The first color filter (42a) can be provided at a position corresponding to the first micro lens (41a).

[0111] The first color filter (42a) can pass light of the first wavelength band among the light transmitted through the dual band pass filter (45).

[0112] More specifically, since light in the wavelength band of red visible light is blocked by the dual band pass filter (45) and light in the wavelength band of red visible light is not incident, the first color filter (42a) can pass light in the first wavelength band corresponding to infrared light.

[0113] The wavelength band of red visible light can include a wavelength band from 625 nm to 750 nm.

[0114] The second color filter (42b) can be provided at a position corresponding to the second micro lens (41b).

[0115] The second color filter (42b) can pass light of the second wavelength band among the light transmitted through the dual band pass filter (45).

[0116] The third color filter (42c) can be provided at a position corresponding to the third micro lens (41c).

[0117] The third color filter (42c) can pass light of the third wavelength band among the light transmitted through the dual band pass filter (45).

[0118] The photodiode section (43) can be provided adjacent to the color filter section (42).

[0119] The photodiode section (43) converts the received light into electrons.

[0120] The photodiode section (43) may include a first photodiode (43a) that is provided adjacent to the first color filter (42a) and converts light of a first wavelength band that has passed through the first color filter (42a) into electrons, a second photodiode (43b) that is provided adjacent to the second color filter (42b) and converts light of a second wavelength band that has passed through the second color filter (42b) into electrons, and a third photodiode (43c) that is provided adjacent to the third color filter (42c) and converts light of a third wavelength band that has passed through the third color filter (42c) into electrons.

[0121] The circuit board (44) may include a plurality of analog / digital circuits (44a, 44b, 44c) that convert electrons into digital signals, and may include an image signal processor that performs image signal processing based on digital signals converted by the plurality of analog / digital circuits.

[0122] More specifically, the first analog-digital circuit (44a) is provided adjacent to the first photodiode (43a) and converts electrons converted by the first photodiode (43a) into a digital signal.

[0123] The second analog-digital circuit (44b) is provided adjacent to the second photodiode (43b) and converts electrons converted by the second photodiode (43b) into a digital signal.

[0124] The third analog-digital circuit (44c) is provided adjacent to the third photodiode (43c) and converts electrons converted by the third photodiode (43c) into a digital signal.

[0125] The image signal processor can acquire an image by performing image processing based on the digital signal converted by the first, second, and third analog-digital circuits (44a, 44b, 44c).

[0126] A dual band pass filter (45) can be provided adjacent to the micro lens section (41).

[0127] A dual band pass filter (45) blocks light of a reference wavelength band among the light incident on the dual band pass filter (45) and passes light of the remaining wavelength bands.

[0128] The reference wavelength band may include a wavelength band from 625 nm to 750 nm. Preferably, the reference wavelength band may include a wavelength band of approximately 640 nm.

[0129] The remaining wavelength bands may include the first, second, and third wavelength bands. For example, the remaining wavelength bands may include a wavelength band from 400 nm to 565 nm, and a wavelength band from 850 nm to 950 nm.

[0130] A dual band pass filter (45) can block red visible light and pass green and blue visible light. The dual band pass filter (45) can pass infrared (IR) light.

[0131] This is explained with reference to FIGS. 4 and 5.

[0132] Fig. 4 is a graph of the quantum efficiency (QE) by wavelength band of light received by the image sensor (40), and Fig. 5 is a graph of the transmittance by wavelength band for light transmitted through the dual band pass filter (45) and the first color filter (42a).

[0133] As shown in Fig. 4, it can be seen that the wavelength band corresponding to the red visible light received by a general image sensor (i.e., the reference wavelength band) is from 625 nm to 750 nm.

[0134] Based on this, the present embodiment blocks light of a reference wavelength band using a dual band pass filter (45) to prevent red visible light from entering the image sensor (40).

[0135] As illustrated in Fig. 4, it can be seen that visible light in the second and third wavelength bands is received by the image sensor 43a. In other words, it can be seen that visible light in the second and third wavelength bands is received by the first color filter (42a).

[0136] Based on this, the image sensor of the present embodiment reduces the transmittance of the dual band pass filter for the second and third wavelength bands. That is, the transmittance of light of the dual band pass filter (45) of the present embodiment for the second and third wavelength bands may be lower than the first transmittance.

[0137] The light transmittance of the first wavelength band of the dual band pass filter (45) may be greater than or equal to the second transmittance.

[0138] The red visible light transmittance of the dual band pass filter (45) can be approximately 0%.

[0139] Here, the second transmittance may be a higher transmittance than the first transmittance.

[0140] For example, the first transmittance of light in the second and third wavelength bands may be approximately 5%, and the second transmittance of light in the first wavelength band may be approximately 95%.

[0141] A dual band pass filter (45) can pass light of the second and third wavelength bands with a first transmittance, pass light of the first wavelength band with a second transmittance, and pass red visible light with a transmittance of approximately 0%. In other words, the dual band pass filter (45) can block red visible light from passing through.

[0142] As shown in FIG. 5, the image sensor (40) can allow light of a wavelength band corresponding to infrared rays to pass through a dual band pass filter (45) and a first color filter (42a).

[0143] Fig. 6 is a control configuration diagram of an electronic device according to one embodiment, which will be described with reference to Figs. 7 to 10. The control configuration of a projector among electronic devices will be described.

[0144] Figures 7 to 10 are exemplary diagrams of image acquisition and touch point position recognition of a projector according to one embodiment.

[0145] The projector (1) may include a projection unit (20), a light emitting unit (30), an image sensor (40), an input unit (51), a communication unit (52), a speaker (53), a processor (54), and a memory (55).

[0146] The projection unit (20) can adjust the brightness of the image, the size of the image, and the projection direction of the image based on the control command of the processor (54).

[0147] The projection unit (20) can output an image of content based on a control command of the processor (54), and can change the image of content output based on the control command of the processor (54).

[0148] Here, the content may include content selected by the user.

[0149] The projection unit (20) can also output a preset basic image based on receipt of a power-on command for the projector.

[0150] The projection unit (20) can output an image of the content using a light source that emits light higher than a preset reference brightness. Here, the preset reference brightness can be approximately 500 lux.

[0151] The light source provided in the projection unit (20) may include a laser light source.

[0152] The light emitting unit (30) can emit infrared light (L1) to the projection area based on a control command of the processor (54).

[0153] The image sensor (40) can acquire an image of the projection area and transmit the acquired image to the processor (54).

[0154] As illustrated in FIG. 7, when a user's hand is present at any point within an image (PI) of content projected onto a floor surface, the image sensor (40) can acquire an image (PI) of the content displayed in the projection area and an image of a subject (T). Here, the subject (T) may include a user's hand performing a touch input, a user's foot, and a pointing means such as a pen and a pointer.

[0155] The image sensor (40) may include a dual band pass filter (45) that blocks the passage of light in a reference wavelength band. Only light in the remaining wavelength bands may be incident on the image sensor (40). The image sensor (40) may acquire first and second images using the incident light in the remaining wavelength bands.

[0156] The reference wavelength band may include a wavelength band corresponding to red visible light.

[0157] The remaining wavelength bands may include a wavelength band corresponding to green visible light, a wavelength band corresponding to blue visible light, and a wavelength band corresponding to infrared light.

[0158] A dual band pass filter (45) can only pass light in the wavelength band of green visible light and light in the wavelength band of blue visible light. The dual band pass filter may be a dual band color pass filter.

[0159] The image sensor (40) can acquire an image during execution of the first sensing mode and acquire an image during execution of the second sensing mode based on a control command of the processor (54).

[0160] The projector (1) may further include an exposure control unit (46) for controlling the exposure value.

[0161] The exposure control unit (46) may further include at least one of a shutter and an aperture provided adjacent to the image sensor (40).

[0162] The shutter allows light to pass through for a set amount of time, thereby controlling the amount of light that reaches the image sensor.

[0163] The aperture adjusts the opening to control the amount of light entering the image sensor (40).

[0164] The input unit (51) receives user input.

[0165] User input may include a power on command and a power off command.

[0166] User input may include configuration information such as resolution, focus, contrast, brightness, sharpness, and screen size (i.e., the size of the image).

[0167] User input may include selection information about the content.

[0168] The input unit (51) can also receive a touch function on command and a touch function off command.

[0169] The input unit (51) may include hardware devices such as various buttons, switches, a keyboard, a mouse, a trackball, various levers, handles, or sticks.

[0170] Additionally, the input unit (51) may include a GUI (Graphical User Interface), i.e., a software device, such as a touch pad.

[0171] The communication unit (52) performs communication with a remote control device (not shown) and an external device. The communication unit (52) can receive content from an external device.

[0172] The communication unit (52) can receive user input from a remote control device and transmit the received user input to the processor.

[0173] The communication unit (52) can communicate with home appliances and can also communicate with a server.

[0174] The communication unit (52) can receive operation information of home appliances or transmit control commands to home appliances. The communication unit (52) can receive content from a server.

[0175] The communication unit (52) may include at least one of a short-range communication module or a long-range communication module.

[0176] The communication unit (52) can transmit data to an external device or receive data from an external device. For example, the communication unit (52) can establish communication with an external device and transmit and receive various types of data.

[0177] To this end, the communication unit (52) can support the establishment of a direct (e.g. wired) communication channel or wireless communication channel with an external device, and the performance of communication through the established communication channel.

[0178] According to one embodiment, the communication unit (52) may include a wireless communication module (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (e.g., a local area network (LAN) communication module, or a power line communication module).

[0179] These communication units (52) can communicate with external devices via a first network (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as a plurality of separate components (e.g., multiple chips).

[0180] The short-range wireless communication module may include, but is not limited to, a Bluetooth communication module, a BLE (Bluetooth Low Energy) communication module, a near field communication module, a WLAN (Wi-Fi) communication module, a Zigbee communication module, an infrared (IrDA, infrared Data Association) communication module, a WFD (Wi-Fi Direct) communication module, an UWB (ultrawideband) communication module, an Ant+ communication module, a microwave (uWave) communication module, etc.

[0181] The speaker (53) can output audio information of the content.

[0182] The speaker (53) can turn audio on or off and adjust the volume of audio based on the control command of the processor (54).

[0183] The processor (54) controls the overall operation of the projector (1).

[0184] The processor (54) can control the activation of various components provided in the projector (1) based on a power-on command received through the input unit (51), and can control the deactivation of various components provided in the projector (1) based on a power-off command received through the input unit (51).

[0185] The processor (54) can control the operation of the projection unit (20), the light emitting unit (30), the image sensor (40), the communication unit (52), and the speaker (53) based on user input received through the input unit (51).

[0186] The processor (54) can control communication with an external device based on user input received through the input unit (51).

[0187] The processor (54) can receive content from an external device based on user input received through the input unit (51) and control the output of the received content.

[0188] Content received from an external device may include content selected by the user.

[0189] The processor (54) can control the deactivation of the light emitting unit (30) and the image sensor (40) based on receiving an off command of the touch function, and can control the activation of the light emitting unit (30) and the image sensor (40) based on receiving an on command of the touch function.

[0190] The processor (54) can control at least one of the projection unit (20) and the speaker (53) based on the content output command being received.

[0191] The processor (54) controls the operation of the projector (20) based on the fact that the content includes image information, controls the operation of the speaker (53) based on the fact that the content includes audio information, and can control the operation of the projector (20) and the speaker (53) based on the fact that the content includes image information and audio information.

[0192] Let's describe the behavior of the processor when a touch-on command is received.

[0193] The processor (54) can control the projection unit (20) to project an image of the content.

[0194] The processor (54) can control the image sensor (40) while projecting an image of the content, acquire first and second images based on the images received from the image sensor (40), and recognize position information of the touch point based on the acquired first and second images.

[0195] An image for a projection area may contain an image of the content.

[0196] The image for the projection area may include an image of the content and an image of the subject for touch input.

[0197] The processor (54) can control the light emitting unit (30) to output infrared light while projecting an image of the content. Through this, infrared coordinates can be formed in the projection area.

[0198] The processor (54) can control the performance of the first sensing mode and the second sensing mode while the image of the content is projected.

[0199] The processor (54) can control the performance of the second sensing mode after performing the first sensing mode.

[0200] In order to minimize the amount of light of the second and third wavelength bands visible light incident on the 42a image sensor, the processor (54) can adjust the amount of light incident on the image sensor (40) based on controlling the performance of the first and second sensing modes.

[0201] The processor (54) can adjust the amount of light incident on the image sensor (40) differently during the execution of the first and second sensing modes.

[0202] The first sensing mode is a mode for acquiring visible light images, and the second sensing mode is a mode for acquiring infrared images.

[0203] Based on this, the processor (54) can control the amount of light incident on the image sensor (40) so that the amount of light incident on the image sensor (40) during the execution of the first sensing mode is greater than the amount of light incident on the image sensor (40) during the execution of the second sensing mode.

[0204] The processor (54) can adjust at least one of the exposure value and gain value of the image sensor (40) to control the amount of light incident on the image sensor (40).

[0205] That is, the processor (54) can adjust at least one of the exposure value and the gain value of the image sensor based on controlling the performance of the first sensing mode, and can adjust at least one of the exposure value and the gain value of the image sensor based on controlling the performance of the second sensing mode.

[0206] The processor (54) can adjust the exposure value of the image sensor (40) to the first exposure value during execution of the first sensing mode, and can adjust the gain value of the image sensor to the first gain value.

[0207] The processor (54) can adjust the exposure value of the image sensor (40) to the second exposure value during execution of the second sensing mode, and can adjust the gain value of the image sensor (40) to the second gain value.

[0208] The first exposure value may be greater than the second exposure value. For example, the first exposure value may be approximately 62.5 msec, and the second exposure value may be approximately 7.8 msec.

[0209] The first gain value may be greater than the second gain value. For example, the first gain value may be approximately four times greater than the second gain value.

[0210] Adjusting the gain value may include adjusting the intensity of the electrical signal of the image of the image sensor (40).

[0211] The processor (54) can also control the sensitivity (ISO: International Standard Organization) to adjust the exposure value of the image sensor (40).

[0212] That is, the processor (54) can adjust at least one of the exposure value and gain value of the image sensor (40) so that the amount of light incident through the image sensor (40) during the execution of the first sensing mode is greater than the amount of light incident through the image sensor (40) during the execution of the second sensing mode.

[0213] The projector (1) may further include an exposure control unit (46) that controls the exposure value of the image sensor (40) based on a control command of the processor (54).

[0214] The exposure control unit (46) may include at least one of a shutter and an aperture.

[0215] That is, the processor (54) can control at least one of the shutter speed and the aperture opening to adjust the exposure value of the image sensor.

[0216] The processor (54) can control the shutter speed during execution of the first sensing mode to be faster than the shutter speed during execution of the second sensing mode.

[0217] The processor (54) can control the opening of the aperture during the execution of the first sensing mode to be greater than the opening of the aperture during the execution of the second sensing mode.

[0218] The processor (54) can adjust the exposure value of the image sensor (40) to the first exposure value and adjust the gain value to the first gain value so that the amount of light received through the image sensor (40) becomes greater than the amount of light received through the image sensor during the execution of the second sensing mode.

[0219] The processor (54) adjusts the exposure value of the image sensor (40) to the first exposure value and adjusts the gain value to the first gain value, thereby allowing only light of the first, second, and third wavelength bands to be incident through the dual band pass filter (45) and color filter unit (22) of the image sensor, but allowing the amount of light of the second and third wavelength bands to be incident to be greater than when the second sensing mode is performed.

[0220] Through this, the processor (54) can increase the amount of blue visible light and the amount of green visible light passing through the dual band pass filter (45) and the color filter unit (22) of the image sensor (40) during the execution of the first sensing mode.

[0221] The processor (54) can receive an image from the image sensor (40) while performing the first sensing mode and obtain a first image based on the received image.

[0222] The first image may be a blue and green visible light image. If the red visible light is not 100% blocked by the dual band pass filter and the color filter unit, the first image may be a red, blue, and green visible light image.

[0223] The processor (54) can recognize the border of the projection area based on the acquired first image.

[0224] The processor (54) can adjust the exposure value of the image sensor (40) to a second exposure value and adjust the gain value to a second gain value so that the amount of light received through the image sensor (40) is less than the amount of light received through the image sensor (40) during the execution of the first sensing mode.

[0225] Through this, the processor (54) can reduce the amount of blue visible light and the amount of green visible light passing through the dual band pass filter (45) and the color filter unit (22) of the image sensor (40) during the execution of the second sensing mode, and increase the amount of infrared light.

[0226] The processor (54) adjusts the exposure value of the image sensor (40) to the second exposure value and adjusts the gain value to the second gain value, thereby allowing only light of the first, second, and third wavelength bands to be incident through the dual band pass filter (45) and the color filter unit (22) of the image sensor (40), but allowing the amount of light of the second and third wavelength bands to be incident to be less than in the first sensing mode.

[0227] The processor (54) can receive an image from the image sensor (40) while performing the second sensing mode and obtain a second image based on the received image.

[0228] The processor (54) removes green and blue from the image received from the image sensor (40) while performing the second sensing mode, leaving only red. The processor (54) acquires a second image using the remaining red from the received image, and can recognize infrared coordinates and touch points based on the acquired second image.

[0229] The light that passes through the dual band pass filter (45) and color filter section (22) of the image sensor (40) is infrared, and when green and blue are removed from the image received from the image sensor (40) during the execution of the second sensing mode, the infrared that passes through the first color filter remains.

[0230] The second image is an image acquired by infrared rays, and may include infrared coordinates and an image of infrared rays scattered by the subject.

[0231] As illustrated in FIG. 8, the processor (54) can recognize the border (B) of the projection area based on the first image (IM1).

[0232] As illustrated in FIG. 9, the processor (54) can recognize a subject touched within the projection area based on the second image (IM2) and recognize a touch point (Po) by the subject. Although not illustrated, the processor (54) can recognize coordinates within the projection area based on the second image (IM2).

[0233] The processor (54) can recognize the point where infrared light is scattered based on the second image (IM2) and recognize the recognized scattering point as a touch point.

[0234] As illustrated in FIG. 10, the processor (54) can obtain location information of the touch point based on the recognized border (B), the recognized touch point (Po), and the coordinates.

[0235] The processor (54) can recognize user input based on the location information of the touch point and control the operation of the projection unit (20) and speaker (53) or the operation of the communication unit (52) based on the recognized user input.

[0236] The processor (54) can recognize user input based on the location information of the touch point and change the image of the content to be projected based on the recognized user input.

[0237] The processor (54) can also recognize whether the user input is a tap, double tap, press, flick, swipe, drag, pinch or rotate based on the location information of the touch point.

[0238] The processor (54) can also control the image sensor (40) to acquire an image of the projection area after adjusting the exposure value of the image sensor (40) to the first exposure value and adjusting the gain value to the first gain value.

[0239] The processor (54) obtains a first image based on an image received from an image sensor (40), removes green and blue from the received image, leaves only red, and obtains a second image using the remaining red, recognizes the border of the projection area based on the obtained first image, recognizes infrared coordinates and a touch point based on the obtained second image, and recognizes location information of the touch point based on the recognized border, infrared coordinates, and touch point.

[0240] The border of the projection area may be the border of an image of content projected onto a screen, floor, or wall.

[0241] The processor (54) may include hardware such as a central processing unit (CPU) or memory, and software such as a control program. For example, the processor (54) may include one or more processor chips that perform the above-described operations using an algorithm for controlling the operations of components within the projector (1), at least one memory that stores data in the form of a program, and data stored in the at least one memory, or may include one or more processing cores.

[0242] The processor (54) may include a separate NPU that performs the operation of the artificial intelligence model, and may include a graphics-only processor (GPU), etc.

[0243] The memory (55) can store coordinate information by infrared rays emitted by the light emitting unit.

[0244] The memory (55) can store a first exposure value and a first gain value corresponding to a first sensing mode, and a second exposure value and a second gain value corresponding to a second sensing mode.

[0245] The memory (55) can store data for an algorithm for controlling the operation of components within the projector (1) or a program that reproduces the algorithm.

[0246] The memory (55) and the processor (54) may be implemented as separate chips. Alternatively, the memory (55) and the processor (54) may be implemented as a single chip.

[0247] The memory (55) may be implemented as at least one of a non-volatile memory element such as a cache, a ROM (Read Only Memory), a PROM (Programmable ROM), an EPROM (Erasable Programmable ROM), an EEPROM (Electrically Erasable Programmable ROM), and a flash memory, a volatile memory element such as a RAM (Random Access Memory), or a storage medium such as a hard disk drive (HDD) or a CD-ROM, but is not limited thereto.

[0248] At least one component may be added or deleted to correspond to the performance of the components of the projector illustrated in FIG. 6. Furthermore, it will be readily apparent to those skilled in the art that the relative positions of the components may be altered to correspond to the performance or structure of the system.

[0249] Meanwhile, each component illustrated in FIG. 6 represents software and / or hardware components such as a Field Programmable Gate Array (FPGA) and an Application Specific Integrated Circuit (ASIC).

[0250] Figure 11 is a control flowchart of an electronic device according to one embodiment.

[0251] Let's use a projector as an example of an electronic device.

[0252] The projector controls the projection unit based on the content output command received so that the image of the content is projected on the projection area, and controls the light-emitting unit to emit infrared light so that infrared coordinates are formed on the projection area (101).

[0253] The projector performs a first sensing mode and adjusts the amount of light incident on the image sensor (40) based on the performance of the first sensing mode.

[0254] Controlling the amount of light incident on the image sensor (40) is to minimize the amount of red visible light incident on the image sensor.

[0255] Controlling the amount of light incident on the image sensor (40) may include controlling at least one of the exposure value and the gain value.

[0256] The projector can adjust the exposure value of the image sensor (40) to the first exposure value while performing the first sensing mode, and can adjust the gain value of the image sensor to the first gain value (102).

[0257] The projector can acquire an image of a projection area using an image sensor (40) while performing the first sensing mode, and acquire a first image based on the acquired image (103).

[0258] By adjusting the exposure value of the image sensor (40) to the first exposure value and adjusting the gain value to the first gain value during execution of the first sensing mode, the projector can allow only light of the first, second, and third wavelength bands to be incident through the dual band pass filter (45) and the color filter unit (22) of the image sensor, but can allow green visible light and blue visible light to be incident, and infrared light to be incident. In other words, the projector can minimize the incident of red visible light by adjusting the amount of light incident on the image sensor.

[0259] The first image may be a visible light image in blue and green.

[0260] If the red visible light is not 100% blocked by the dual band pass filter and color filter section, the first image may be a red, blue and green visible light image.

[0261] The projector can recognize the border of the projection area based on the acquired first image.

[0262] When the projector completes performing the first sensing mode, it performs the second sensing mode.

[0263] The projector can adjust at least one of an exposure value and a gain value of the image sensor so that the amount of light incident on the image sensor during the performance of the second sensing mode is less than the amount of light incident on the image sensor during the performance of the first sensing mode.

[0264] That is, the projector can adjust the exposure value of the image sensor (40) to the second exposure value while performing the second sensing mode, and can adjust the gain value of the image sensor to the second gain value (104).

[0265] The projector can acquire an image of the projection area using the image sensor (40) while performing the second sensing mode, and acquire a second image based on the acquired image (104).

[0266] Infrared rays that pass through the dual band pass filter (45) and the first color filter can be recognized as red because they pass through the first color filter.

[0267] Based on this, the projector can remove green and blue from the image received from the image sensor (40) while performing the second sensing mode, leaving only red, and then acquire a second image using the remaining red, and recognize infrared coordinates and touch points based on the acquired second image.

[0268] The red color left here may be perceived by infrared.

[0269] That is, the light that passes through the dual band pass filter (45) and color filter section (22) of the image sensor (40) is infrared, and when green and blue are removed from the image received from the image sensor (40) during the execution of the second sensing mode, the infrared that passes through the first color filter remains.

[0270] The projector acquires a second image based on the remaining infrared light.

[0271] The second image is an infrared image, which may include infrared coordinates and an image of infrared light scattered by the subject.

[0272] The projector can recognize the location information of the touch point based on the recognized border, recognized touch point, and infrared coordinates (106).

[0273] The projector can recognize user input based on the location information of the touch point and control the operation of the projection unit (20) based on the recognized user input (107).

[0274] The projector can recognize user input based on the location information of the touch point and control the operation of the speaker (53) or the operation of the communication unit (52) based on the recognized user input.

[0275] Since the present embodiment acquires the first and second images using a single image sensor, the field of view of the image sensor for acquiring the first image and the field of view of the image for acquiring the second image may be identical. This means that the boundary of the projection area and the acquisition location of the infrared coordinates are identical. Since the acquisition location of the boundary of the projection area and the acquisition location of the infrared coordinates are not different in the present embodiment, the accuracy of recognizing the touch location can be improved.

[0276] Fig. 12 is a control flowchart of an electronic device according to another embodiment.

[0277] Let's illustrate a vision system with an electronic device.

[0278] The vision system (100) may include a light emitting unit (130), an image sensor (140), an exposure control unit (146), an input unit (151), a communication unit (152), a processor (154), a memory (155), and a display unit (156).

[0279] The light emitting unit (130) can emit infrared light. The direction in which the infrared light is emitted may be the same as the direction in which an image is acquired through the image sensor (140).

[0280] The image sensor (140) can acquire an image and transmit the acquired image to the processor (154).

[0281] The image sensor (140) includes a micro lens unit, a color filter unit, a photodiode unit, a circuit board, and a dual band pass filter (see FIG. 3). The image sensor (140) according to another embodiment is identical to the image sensor (40) according to one embodiment, and thus, a description thereof is omitted.

[0282] The vision system (100) may further include an exposure control unit (146) for controlling the exposure value of the image sensor (140).

[0283] The exposure control unit (146) may further include at least one of a shutter and an aperture provided adjacent to the image sensor (140).

[0284] The shutter allows light to pass through for a set amount of time, thereby controlling the amount of light that reaches the image sensor.

[0285] The aperture adjusts the opening to control the amount of light entering the image sensor (140).

[0286] The input unit (151) receives user input.

[0287] User input may include a power on command and a power off command.

[0288] User input may include a shooting command for performing a vision function.

[0289] The input unit (151) may include hardware devices such as various buttons, switches, a keyboard, a mouse, a trackball, various levers, handles, or sticks.

[0290] Additionally, the input unit (151) may include a GUI (Graphical User Interface), i.e., a software device, such as a touch pad.

[0291] The communication unit (152) performs communication between components within the vision system and with external devices. This communication unit (152) is identical to the communication unit (52) of one embodiment, and thus a description thereof is omitted.

[0292] The display unit (156) can display an image based on a control command of the processor (154).

[0293] The display unit (156) can display an image acquired by the image sensor (140) and can display information on a subject within the image acquired by the image sensor (140).

[0294] In the case of a vision system for object recognition, the display unit (156) can display information about the recognized object.

[0295] In the case of a vision system for user authentication, the display unit (156) can display recognition result information corresponding to user authentication success or user authentication failure.

[0296] The display unit (156) may be provided as a cathode ray tube (CRT), a digital light processing (DLP) panel, a plasma display panel, a liquid crystal display (LCD) panel, an electroluminescence (EL) panel, an electrophoretic display (EPD) panel, an electrochromic display (ECD) panel, a light emitting diode (LED) panel, or an organic light emitting diode (OLED) panel, but is not limited thereto.

[0297] The processor (154) controls the overall operation of the vision system (100).

[0298] The processor (154) can control the activation of various components provided in the vision system (100) based on a power-on command received through the input unit (151), and can control the deactivation of various components provided in the vision system (100) based on a power-off command received through the input unit (151).

[0299] The processor (154) can control the operation of the light emitting unit (130), the image sensor (140), and the exposure control unit (146) based on user input received through the input unit (151).

[0300] The processor (154) can control the image sensor (140) during the performance of the vision function, acquire first and second images based on images received from the image sensor (140), recognize a subject based on the acquired first and second images, and control the display unit (156) to display information about the recognized subject.

[0301] The processor (154) can control the light emitting unit (130) to output infrared light while performing a vision function. Through this, the infrared light can be scattered by the subject.

[0302] The processor (154) can control the performance of the first sensing mode and the second sensing mode while performing the vision function.

[0303] The processor (154) can control the performance of the second sensing mode after performing the first sensing mode.

[0304] To minimize the amount of red visible light incident on the image sensor (140), the processor (154) can adjust the amount of light incident on the image sensor (140) based on controlling the performance of the first and second sensing modes.

[0305] The processor (154) can adjust the amount of light incident on the image sensor (140) differently during the execution of the first and second sensing modes.

[0306] The first sensing mode is a mode for acquiring visible light images, and the second sensing mode is a mode for acquiring infrared images.

[0307] Based on this, the processor (154) can control the amount of light incident on the image sensor (140) so that the amount of light incident on the image sensor (140) during the execution of the first sensing mode is greater than the amount of light incident on the image sensor (140) during the execution of the second sensing mode.

[0308] The processor (154) can adjust at least one of the exposure value and gain value of the image sensor (140) to control the amount of light incident on the image sensor (140).

[0309] The processor (154) can adjust the exposure value of the image sensor (140) to the first exposure value during execution of the first sensing mode, and can adjust the gain value of the image sensor to the first gain value.

[0310] The processor (154) can adjust the exposure value of the image sensor (140) to the second exposure value during execution of the second sensing mode, and can adjust the gain value of the image sensor (140) to the second gain value.

[0311] The first exposure value may be greater than the second exposure value. The first gain value may be greater than the second gain value.

[0312] Adjusting the gain value may include adjusting the intensity of an electrical signal of an image of the image sensor (140).

[0313] The processor (154) can control the exposure control unit (146) to adjust the exposure value of the image sensor (140). That is, the processor (154) can control at least one of the shutter speed and the aperture opening to adjust the exposure value of the image sensor.

[0314] The processor (154) can control the shutter speed during execution of the first sensing mode to be faster than the shutter speed during execution of the second sensing mode.

[0315] The processor (154) can control the opening of the aperture during the execution of the first sensing mode to be greater than the opening of the aperture during the execution of the second sensing mode.

[0316] The processor (154) can receive an image from the image sensor (140) while performing the first sensing mode and obtain a first image based on the received image.

[0317] The first image may be a blue and green visible light image. If the red visible light is not 100% blocked by the dual band pass filter and the color filter unit, the first image may be a red, blue, and green visible light image.

[0318] The processor (154) can recognize the subject based on the acquired first image.

[0319] The processor (154) can adjust the exposure value of the image sensor (140) to a second exposure value and adjust the gain value to a second gain value so that the amount of light received through the image sensor (140) is less than the amount of light received through the image sensor (140) during the execution of the first sensing mode.

[0320] The processor (154) can receive an image from the image sensor (140) while performing the second sensing mode and obtain a second image based on the received image.

[0321] The processor (154) removes green and blue from the image received from the image sensor (140) while performing the second sensing mode, leaving only red. The processor (154) acquires a second image using the remaining red from the received image, and can recognize the shape of the subject based on the acquired second image.

[0322] The light that passes through the dual band pass filter and color filter of the image sensor (140) is infrared, and when green and blue are removed from the image received from the image sensor (140) during the execution of the second sensing mode, the infrared light that passes through the first color filter remains.

[0323] The second image is an image acquired by infrared rays, and may include an image of infrared rays scattered by the subject.

[0324] The processor (154) can recognize the subject based on the first image and the second image.

[0325] When the vision function is a function for user authentication, the processor (154) can recognize whether the user is a pre-registered user based on information about the recognized subject and information about the subject stored in the memory (155).

[0326] That is, if the similarity between the information on the recognized subject and the information on the subject stored in the memory (155) is greater than or equal to the standard similarity, the processor (154) recognizes that the user corresponding to the subject is a pre-registered user and determines that user authentication is successful, and if the similarity between the information on the recognized subject and the information on the subject stored in the memory (155) is less than the standard similarity, the processor (154) recognizes that the user corresponding to the subject is not a pre-registered user and determines that user authentication is unsuccessful.

[0327] When the vision function is a function for object recognition, the processor (154) can recognize an object corresponding to the subject based on information about the recognized subject and information about the subject stored in the memory (155), and control the display unit (156) to display information corresponding to the recognized object.

[0328]

[0329] The processor (154) may include hardware such as a central processing unit (CPU) or memory, and software such as a control program. For example, the processor (154) may include one or more processor chips that perform the aforementioned operations using an algorithm for controlling the operations of components within the vision system (100), at least one memory that stores data in the form of a program, and data stored in the at least one memory, or may include one or more processing cores.

[0330] The processor (154) may include a separate NPU that performs the operation of the artificial intelligence model, and may include a graphics-only processor (GPU), etc.

[0331] The memory (155) can store shape information for each subject.

[0332] The memory (155) can store a first exposure value and a first gain value corresponding to a first sensing mode, and can store a second exposure value and a second gain value corresponding to a second sensing mode.

[0333] The memory (155) can store data for an algorithm for controlling the operation of components within the vision system (100) or a program that reproduces the algorithm.

[0334] The memory (155) and the processor (154) may be implemented as separate chips. Alternatively, the memory (155) and the processor (154) may be implemented as a single chip.

[0335] The memory (155) may be implemented as at least one of a non-volatile memory element such as a cache, a ROM (Read Only Memory), a PROM (Programmable ROM), an EPROM (Erasable Programmable ROM), an EEPROM (Electrically Erasable Programmable ROM), and a flash memory, a volatile memory element such as a RAM (Random Access Memory), or a storage medium such as a hard disk drive (HDD) or a CD-ROM, but is not limited thereto.

[0336] At least one component may be added or deleted to correspond to the performance of the components of the vision system illustrated in Fig. 12. Furthermore, it will be readily apparent to those skilled in the art that the relative positions of the components may be altered to correspond to the performance or structure of the system.

[0337] Meanwhile, each component illustrated in FIG. 12 represents software and / or hardware components such as a Field Programmable Gate Array (FPGA) and an Application Specific Integrated Circuit (ASIC).

[0338] Meanwhile, the disclosed embodiments may be implemented in the form of a recording medium storing computer-executable instructions. The instructions may be stored in the form of program code, and when executed by a processor, may generate program modules to perform the operations of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

[0339] Computer-readable storage media include all types of storage media that store instructions that can be deciphered by a computer. Examples include read-only memory (ROM), random access memory (RAM), magnetic tape, magnetic disks, flash memory, and optical data storage devices.

[0340] The disclosed embodiments have been described with reference to the attached drawings as described above. Those skilled in the art will understand that the present invention can be implemented in forms other than the disclosed embodiments without altering the technical spirit or essential features of the present invention. The disclosed embodiments are illustrative and should not be construed as limiting.

Claims

1. A projection unit that projects a light image onto a projection area; An emitter that emits infrared light; An image sensor having a dual band pass filter and acquiring an image of the projection area; and An electronic device including a processor that acquires a visible light image and an infrared image based on an image of a projection area received from the image sensor, and recognizes location information of a touch point by a subject within the projection area based on the acquired visible light image and infrared image.

2. In paragraph 1, The above dual band pass filter blocks light in a wavelength band corresponding to red visible light, passes light in a wavelength band corresponding to blue and green visible light, and passes light in a wavelength band corresponding to infrared light. An electronic device in which the processor recognizes the boundary of the projection area based on the visible light image, recognizes infrared coordinates and a touch point based on the infrared image, and recognizes location information of the touch point based on the boundary of the recognized projection area, the recognized infrared coordinates, and the touch point.

3. In paragraph 1, The above dual band pass filter blocks light in a wavelength band corresponding to red visible light, passes light in a wavelength band corresponding to blue and green visible light, and passes light in a wavelength band corresponding to infrared light. The above processor is an electronic device that obtains the infrared image by removing blue and green from an image of a projection area received from the image sensor.

4. In the first paragraph, the processor, An electronic device that acquires the visible light image based on an image received from the image sensor during execution of the first sensing mode, and acquires the infrared image based on an image received from the image sensor during execution of the second sensing mode.

5. In paragraph 4, The processor adjusts the exposure value of the image sensor to a first exposure value during execution of the first sensing mode, and adjusts the exposure value of the image sensor to a second exposure value during execution of the second sensing mode. An electronic device wherein the first exposure value is an exposure value greater than the second exposure value.

6. In paragraph 4, The processor adjusts the gain value of the image sensor to a first gain value during execution of the first sensing mode, and adjusts the gain value of the image sensor to a second gain value during execution of the second sensing mode. An electronic device wherein the first gain value is a gain value greater than the second gain value.

7. In the first paragraph, the processor, An electronic device that adjusts at least one of an exposure value and a gain value of the image sensor based on acquiring an image using the image sensor.

8. In the first paragraph, the processor, An electronic device that recognizes a user input based on position information of the recognized touch point and controls the operation of the projection unit based on the recognized user input.

9. An emitter that emits infrared light; An image sensor having a dual band pass filter and acquiring an image of the projection area; A processor that acquires a visible light image and an infrared image based on an image of a projection area received from the image sensor and recognizes a subject based on the acquired visible light image and infrared image; and An electronic device including a display unit that displays information about the recognized subject.

10. In paragraph 9, The above dual band pass filter blocks light in a wavelength band corresponding to red visible light, passes light in a wavelength band corresponding to blue and green visible light, and passes light in a wavelength band corresponding to infrared light. The processor is an electronic device that obtains the infrared image by removing blue and green from an image of a projection area received from the image sensor, and recognizes the shape of the subject based on the obtained infrared image.

11. In the 9th paragraph, the processor, An electronic device that acquires the visible light image based on an image received from the image sensor during execution of the first sensing mode, and acquires the infrared image based on an image received from the image sensor during execution of the second sensing mode.

12. In paragraph 11, The processor adjusts the exposure value of the image sensor to a first exposure value and adjusts the gain value of the image sensor to the first gain value during execution of the first sensing mode, and adjusts the exposure value of the image sensor to a second exposure value and adjusts the gain value of the image sensor to the second gain value during execution of the second sensing mode. The above first exposure value is an exposure value greater than the above second exposure value, An electronic device wherein the first gain value is a gain value greater than the second gain value.

13. Project a light image onto the projection area, Outputting infrared light to the above projection area, An image of the projection area is acquired using an image sensor equipped with a dual band pass filter, Obtaining visible light images and infrared images based on the images of the above projection area, Recognize the location information of the touch point by the subject within the projection area based on the above-mentioned acquired visible light image and infrared image, Controlling the movement of the projection part based on the position information of the above recognized touch point, A control method of an electronic device, wherein the above dual band pass filter is a filter that blocks light in a wavelength band corresponding to red visible light, passes light in a wavelength band corresponding to blue and green visible light, and passes light in a wavelength band corresponding to infrared light.

14. In the 13th paragraph, recognizing the location information of the touch point, By removing blue and green from the image of the above projection area, the above infrared image is obtained, Recognize the border of the projection area based on the above visible light image, Recognize infrared coordinates and touch points based on the above infrared image, A control method of an electronic device, comprising recognizing position information of the touch point based on the boundary of the recognized projection area, the recognized infrared coordinates, and the touch point.

15. In the 13th paragraph, obtaining a visible light image and an infrared image based on the image of the projection area, During the execution of the first sensing mode, the exposure value of the image sensor is adjusted to the first exposure value, and the gain value of the image sensor is adjusted to the first gain value. During the execution of the second sensing mode, the exposure value of the image sensor is adjusted to the second exposure value, and the gain value of the image sensor is adjusted to the second gain value. Obtaining a visible light image based on an image of a projection area acquired by the image sensor during execution of the first sensing mode; Including obtaining the infrared image based on the image of the projection area obtained by the image sensor during the execution of the second sensing mode, The above first exposure value is an exposure value greater than the above second exposure value, A control method for an electronic device, wherein the first gain value is a gain value greater than the second gain value.

Images