Interaction system for setting user recognition criteria on basis of fingernail of user

Abstract

Disclosed is an interaction system for setting user recognition criteria on the basis of a fingernail of a user. The interaction system for setting user recognition criteria on the basis of a fingernail of a user, according to the present invention, may comprise: a display device for outputting content including at least one object; and a content providing device for determining an interaction between the object and the user, and providing, to the display device, the content reflecting a preset event effect in response to the interaction. The content providing device may include: an image capturing module for capturing an image of a content area including the content outputted via the display device; and a content control module for detecting an optical effect from an image acquired by the image capturing module on the basis of physical information provided from the fingernail of the user, and setting the optical effect as user recognition criteria for determining whether the user is present in the content area. The physical information may include biophoton information provided from the fingernail of the user.

Description

An interaction system that sets user recognition criteria based on the user's fingernails.

[0001] The present invention relates to an interaction system for interaction between content and a user. In particular, the present invention relates to an interaction system that sets user recognition criteria based on the user's fingernails.

[0002] An interaction device is being developed that uses a beam projector to project content such as educational and game videos onto indoor walls, and detects the user's operations or movements using a joystick and reflects them in the content, allowing the user to enjoy a variety of content indoors.

[0003] However, these types of interaction devices require specialized and complex installation, as they require various sensors to detect user movements. Furthermore, joystick-based interaction systems are complex to operate, making them suitable only for users over a certain age, and infants and children may struggle to use them.

[0004] The present invention is intended to solve the above problems, and provides an interaction system for interaction between users and content that can be easily installed and used, can be constructed at a low cost, and can enhance users' concentration and immersion and stimulate their imagination and creativity.

[0005] The problems to be solved by the present invention are not limited to those mentioned above, and other problems to be solved that are not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the description below.

[0006] An interaction system for setting a user recognition criterion based on a user's fingernail according to the present invention may include a display device for outputting content including at least one object; and a content providing device for determining an interaction between the object and a user and providing the content, which reflects a preset event effect in response to the interaction, to the display device. The content providing device may include a photographing module for photographing a content area including at least the content output through the display device; and a content control module for detecting a light effect based on physical information provided from the user's fingernail from an image acquired from the photographing module, and setting the light effect as a user recognition criterion for determining whether the user is present within the content area. The physical information may include biophoton information provided from the user's fingernail.

[0007] The content providing device may further include a lighting module that irradiates light toward the user's fingernails. The content control module may additionally detect the light effect by inducing light reflection by the user's fingernails through the light irradiated from the lighting module.

[0008] The content control module may acquire first and second images acquired at different points in time from the photographing module, and detect the presence of the user's fingernails based on the difference in the light effect detected in the first and second images. The lighting module may be driven in conjunction with the detection operation of the light effect by the content control module.

[0009] According to the present invention, by setting the light effect caused by light reflected from the fingernail and light generated from the nail as a user recognition criterion to determine the interaction between the user's hand and the content, the position of the user's hand can be detected more easily and quickly than a structure that detects the position of the user's hand for the entire content area.

[0010] According to the present invention, the system can be implemented using a display device such as a beam projector and a user terminal without the need to install a separate sensor, so it can be easily used even in ordinary homes.

[0011] The effects that can be obtained from the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by a person having ordinary skill in the art to which the present invention belongs from the description below.

[0012] FIG. 1 is a diagram illustrating an interaction system for interaction between content and a user according to an embodiment of the present invention.

[0013] FIG. 2 is a block diagram schematically illustrating the configuration of an interaction system for interaction between content and a user according to an embodiment of the present invention.

[0014] FIG. 3 is a drawing for explaining a light effect by a nail according to an embodiment of the present invention.

[0015] FIG. 4 is a drawing showing an example of an operation for detecting a light effect by a fingernail and an operation for detecting noise according to one embodiment of the present invention.

[0016] FIG. 5 and FIG. 6 are drawings showing examples of operations for detecting and determining a gesture according to another embodiment of the present invention.

[0017] FIG. 7 is a drawing showing an example of a light effect detection operation using a guide image according to another embodiment of the present invention.

[0018] Hereinafter, the embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Regardless of the drawing numbers, identical or similar components will be given the same reference numbers, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably only for the convenience of writing the specification, and do not in themselves have distinct meanings or roles. In addition, when describing the embodiments disclosed in this specification, if it is determined that a specific description of a related known technology may obscure the gist of the embodiments disclosed in this specification, a detailed description thereof will be omitted. In addition, the attached drawings are only intended to facilitate easy understanding of the embodiments disclosed in this specification, and the technical ideas disclosed in this specification are not limited by the attached drawings, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and technical scope of the present invention.

[0019] Terms that include ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another.

[0020] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0021] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0022] In this application, terms such as “include” or “have” are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but should be understood not to exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0023] FIG. 1 is a diagram illustrating an interaction system for interaction between content and a user according to an embodiment of the present invention. FIG. 2 is a block diagram schematically illustrating the configuration of an interaction system for interaction between content and a user according to an embodiment of the present invention. FIG. 3 is a diagram illustrating a light effect by a fingernail according to an embodiment of the present invention.

[0024] Referring to FIGS. 1 and 2, an interaction system (10) for interaction between content and a user according to a preferred embodiment of the present invention may include a display device (100) that outputs predetermined content, and a content providing device (200) that detects interaction between a user (30) and at least one object (25) included in the content and generates a preset event effect in response to the detected interaction.

[0025] The display device (100) is provided from the content providing device (200) and can output content predetermined by the user (30) and event effects corresponding to interactions between the content and the user (30). The display device (100) is placed at a predetermined distance from a projection body (20) predetermined by the user (30), such as a wall, a fixture, or a screen, and can be a beam projector that outputs content to a preset area (hereinafter, referred to as “content area (21)”) of the projection body (20). The content can include at least one object (25), and the object (25) can be, but is not limited to, a person, an animal, an object, a number, a character, etc. in an image.

[0026] Although not shown, the display device (100) may be implemented as a display device (100) such as a liquid crystal display (LCD), a field emission display (FED), a plasma display panel (PDP), an organic light emitting diode display (OLED), an electrophoretic display (EPD), or a quantum dot display (QDD) capable of outputting content. Hereinafter, for convenience of explanation, an example in which the display device (100) is a beam projector will be described.

[0027] The display device (100) may include an image processing unit that outputs an input image. The display device (100) may further include image signal input terminals for inputting image signals from a content providing device (200), and / or a communication module for transmitting and receiving data with the content providing device (200).

[0028] The content providing device (200) may include a shooting module (210) and a content control module (230). The content providing device (200) may be implemented in the form of a single user terminal including any one of a smartphone, a tablet PC, a laptop, a personal digital assistant (PDA), and a mobile communication terminal. However, the present invention is not limited thereto, and the modules constituting the content providing device (200) may be provided as separate devices or as a device in which two or more modules are combined to transmit and receive data via a wired / wireless network.

[0029] The shooting module (210) includes at least one image sensor and can acquire an image of content output through the display device (100) and a specific body part of the user (30) located on the content. The image sensor captures an image of the content area (21) (or a preset area including the content area (21)) to acquire an image. For example, the image sensor may be implemented as a semiconductor device capable of converting an optical signal into an electrical signal, such as a CCD (Charge-Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), but is not limited thereto. The shooting module (210) may include an encoder (not shown), and the encoder may perform an operation of encoding an image acquired through the image sensor into a digital signal. The image sensor may acquire an image in real time or at a preset cycle. The shooting cycle of the image sensor may be predetermined or may be adjusted by a control signal of the content control module (230).

[0030] Content may be provided in the form of a video, and the video may consist of multiple video frames. Here, one video frame may be expressed as a still image. The capturing module (210) may capture content output through the display device (100) in real time or at a predetermined cycle to obtain a still image.

[0031] The content control module (230) can provide content selected by a user (30) and content with an event effect corresponding to the interaction to the display device (100). The content control module (230) can include a memory (not shown) in which various contents are stored. And / or, the content control module (230) can receive the selected contents from a network-based contents provision server (not shown) and provide the content to the display device (100). The contents provision server can be, but is not limited to, a server provided by a company or public institution, a contents open market server that provides various contents such as education and games in an integrated manner and to which a plurality of users (30) can access, or an email or web hard drive that can be accessed by individual users (30).

[0032] When the content control module (230) detects user information of the user (30) from an image acquired from the shooting module (210), it can determine an interaction with at least one object (25) in the content according to the detected user information, and can reflect a preset event effect corresponding to the interaction in the content output through the display device (100). The user information may include at least one of the position, size, and / or motion (gesture) information of a specific body part (e.g., nail, hand, finger, etc.) of the user (30). For example, the content control module (230) can detect the position of the hand of the user (30) through the image acquired from the shooting module (210), and when it is determined that the position of at least one object (25) included in the content overlaps with the hand of the user (30), it can generate a corresponding preset event effect and reflect it in the content output through the display device (100).

[0033] The event effect may be a change in the shape or size of an object (25) that is the target of interaction with a user (30), or may be an additional effect reflected in the content, but is not limited thereto. The event effect may be an effect visually provided through a display device (100), and may further include an auditory effect provided audibly through a speaker or the like, if necessary. For example, the event effect may include a visual effect in which the shape of a specific object (25) changes when it overlaps with a specific body part of a user (30), and may further include an auditory effect that provides a sound indicating that an interaction has occurred.

[0034] The content control module (230) can select a content area (21) within the field of view of the shooting module (210) and determine whether an interaction has occurred within the content area (21). Alternatively, the content control module (230) can determine a predetermined area of ​​interest among the projected content areas (21) and determine whether an interaction has occurred within the area of ​​interest. Alternatively, the content control module (230) can control the shooting area by controlling the shooting module and determine whether an interaction has occurred within the shooting area.

[0035] The content control module (230) can detect and identify the fingernail (FN) of the user (30) through image analysis from the image acquired from the shooting module (210). In addition, the content control module (230) can determine an interaction with at least one object (25) within the content based on the detected fingernail (FN) information, and can generate a preset event effect in response to the interaction based on the interaction determination result. To this end, the content control module (230) can include an identification unit (231), a determination unit (233), and a generation unit (235).

[0036] Referring further to FIG. 3, the identification unit (231) can detect the fingernail (FN) of the user (30) in the content area (21) based on physical information provided by at least one of light reflected from the fingernail (FN) of the user (30) and light generated (hereinafter, referred to as 'light effect (LE)'), and can set this as a criterion for recognizing the user (30) in the content area (21). The cycle of the fingernail (FN) detection operation through the identification unit (231) can be predetermined. The physical information may be, but is not limited to, information on reflectance of light and change in reflected color. In addition, the physical information may include biophoton information provided from the fingernail (FN). The light reflected from the fingernail (FN) may be, but is not limited to, reflected light provided from the display device (100) or reflected light provided from an external environment.

[0037] Deep learning technology (e.g., learning and detecting a pattern of occurrence of light effects (LE) using an architecture such as a CNN (Convolutional Neural Network)) may be employed to detect light effects (LE) caused by fingernails (FN), but is not limited thereto, and a known algorithm may be used to detect light effects (LE) caused by fingernails (FN).

[0038] The identification unit (231) can detect a fingernail (FN) based on a light effect (LE), and can detect at least one of the outline (or, outer line) of the fingernail (FN) and the size, shape, and position of the area formed by the outline of the fingernail (FN). And / or, the identification unit (231) can obtain and detect the outline of a hand (or, finger) including the fingernail (FN) based on the fingernail (FN) detected based on the light effect (LE), and can detect at least one of the size, shape, and position of the area formed by the outline of the finger. An algorithm for recognizing an object such as a fingernail (FN) or a finger can be implemented as a known algorithm.

[0039] The identification unit (231) determines that the user's (30) fingernail (FN) is located at the corresponding location within the content area (21) when a light effect (LE) due to the fingernail (FN) is detected on the image acquired from the photographing module (210), and can detect the presence, outline (shape, size, etc.) and location information of the fingernail (FN) and / or finger. The determination unit (233) can determine or estimate the movement and gesture information (hereinafter referred to as "user action information") of the fingernail (FN) and / or finger based on the detected fingernail (FN) and / or finger information. The determination unit (233) can determine whether there is an interaction with at least one object (25) within the content based on the estimated user action information. The determination of whether there is an interaction can be performed under a preset condition, and the condition can be predetermined and stored in a memory. For example, the condition can be whether there is an overlap between the fingernail (FN) or the finger and the object (25) within the content. That is, the determination unit (233) can determine that an interaction has occurred when a fingernail (FN) or hand overlaps with an object (25) within the content.

[0040] The determination unit (233) can track the movement of the fingernail (FN) and / or hand detected by the identification unit (231). For example, the determination unit (233) can track the movement of the hand and / or hand of the user (30) based on a plurality of time-series images acquired by the photographing module (210). As an example, the movement of the fingernail (FN) and / or hand can be tracked using an optical flow algorithm. As another example, the determination unit (233) can track the movement of the fingernail (FN) and / or hand of the user (30) by expressing a change in the position of the fingernail (FN) and / or hand detected for a plurality of time-series acquired images as a motion vector.

[0041] If the determination unit (233) determines that the interaction condition between the user (30) and the object (25) within the content is satisfied, the generation unit (235) can generate a corresponding event effect and provide content reflecting the same to the display device (100). The generation unit (235) can selectively activate preset event effects in response to various interactions. To this end, event effects matched to each user behavior information or each object (25) within the content can be stored in advance in memory. And / or, the event effects can be provided and activated from a content provision server.

[0042] The content providing device (200) may further include a lighting module (250) that irradiates light toward the projector (20) or the user (the user's fingernail (FN)). The lighting module (250) may be used as a means for effectively detecting a light effect (LE) by inducing light reflection by the fingernail (FN). The light irradiation by the lighting module (250) may be controlled to be linked to the light effect (LE) detection operation by the identification unit (231). However, the present invention is not limited thereto, and the light irradiation by the lighting module (250) may be controlled at a preset cycle.

[0043] FIG. 4 is a drawing showing an example of an operation for detecting a light effect by a fingernail and an operation for detecting noise according to one embodiment of the present invention.

[0044] Referring to FIG. 4, the content control module (230) can compare and analyze n (n is a natural number greater than or equal to 2) frame images acquired in time series to detect a light effect (LE) caused by a fingernail (FN).

[0045] The identification unit (231) can acquire first and second images at different points in time from the photographing module (210), and detect the presence of a fingernail (FN) from the difference between the first and second images. The first image may be an image that does not include a light effect (LE), and the second image may be an image that includes a light effect (LE).

[0046] The identification unit (231) can perform an operation of detecting noise (NI) other than the light effect (LE) caused by the fingernail (FN) based on images acquired from the photographing module (210). The detected noise (NI) may be an effect (hereinafter referred to as a light reflection effect) that is irradiated from a fixed external light or the like and reflected from the projector (20).

[0047] For example, the identification unit (231) can obtain third and fourth images at different points in time from the photographing module (210). The third and fourth images may include a light effect (LE) and a light reflection effect determined as noise (NI). The identification unit (231) can detect noise (NI) by comparing the difference in the location where the 'light effect (LE)' and the 'light reflection effect' occur in the third and fourth images, respectively.

[0048] The 'light reflection effect' determined as noise (NI) can be detected at a fixed position in the third and fourth images without any change in position. In contrast, the 'light effect (LE)' caused by the fingernail (FN) can be detected at different positions in the third and fourth images as the position and direction of the fingernail (FN) change due to a certain movement of the user (30). The identification unit (231) can compare the third and fourth images to distinguish between the 'light effect (LE)' caused by the fingernail (FN) and the 'light reflection effect' caused by external lighting, and classify the 'light reflection effect' as noise (NI) so as not to identify it as the fingernail (FN) of the user (30).

[0049] FIG. 5 and FIG. 6 are drawings showing examples of operations for detecting and determining a gesture according to another embodiment of the present invention.

[0050] Referring to FIGS. 5 and 6, the content control module (230) can compare and analyze frame images to detect and determine a gesture based on the light effect (LE) by the fingernail (FN).

[0051] The identification unit (231) can detect the number of fingernails (FN) according to the change in hand gesture by counting the number of light effects (LE) detected on the image. In addition, the identification unit (231) can detect the position of the fingernails (FN) and / or the mutual distance between neighboring fingernails (FN). The number of light effects (LE) can correspond to the number of fingernails (FN) of the user (30). The determination unit (233) can estimate the gesture of the user (30) based on the detection information through the identification unit (231). The determination unit (233) can additionally extract outline information of the finger (or hand) including the fingernails (FN) and use this to effectively estimate the gesture of the user (30).

[0052] More specifically, as illustrated, the user (30) can perform a gesture action of clenching and unclenching a finger. The identification unit (231) can detect a light effect (LE) from images acquired through the photographing module (210) in response to the gesture action of the user (30), and based on this, can detect information such as the number of fingernails (FN) and the location of the fingernails (FN).

[0053] The determination unit (233) can estimate a gesture action based on the detection information by the identification unit (231). Referring to FIG. 5, when the determination unit (233) obtains information on one fingernail (FN) from the identification unit (231), it can estimate it as a first gesture, and can determine the interaction between the estimated first gesture and the object (25). When the determination unit (233) obtains information on k (k is a natural number greater than or equal to 2 and less than 5) fingernails (FN) from the identification unit (231), it can estimate it as a k-th gesture, and can determine the interaction between the estimated k-th gesture and the object (25).

[0054] Alternatively, referring to FIG. 6, the determination unit (233) can first estimate the k-th gesture by obtaining k (k is a natural number greater than or equal to 2 and less than 5) fingernail (FN) information from the identification unit (231), and secondarily estimate the k-th gesture as one of a plurality of sub-gestures by obtaining information on the mutual spacing between neighboring fingernails (FN) from the identification unit (231) (including hand outline information, if necessary), and can determine the interaction between the estimated sub-gesture and the object (25). A gesture includes a plurality of sub-gestures that are allowable due to the body structure, and can be clustered and stored in a memory. For example, a second gesture may be a gesture including two fingernail information and may include a plurality of sub-gestures that can be implemented with two fingers, and a third gesture may be a gesture including three fingernail information and may include a plurality of sub-gestures that can be implemented with three fingers.

[0055] If the generation unit (235) determines that the interaction condition with the object (25) is satisfied in response to the user's (30) gesture (Fig. 5) or sub-gesture (Fig. 6), the generation unit (235) can generate a corresponding event effect and provide it to the display device (100). The generation unit (235) can activate a pre-matched event effect in response to the interaction with the object (25) for each gesture or sub-gesture. To this end, gesture information corresponding to the number of nails (FN), sub-gesture information corresponding to the mutual spacing between neighboring nails (FN), and event effects corresponding to the interaction for each gesture / sub-gesture may be determined in advance and stored in memory. And / or, the event effect may be provided from a content provision server and activated.

[0056] FIG. 7 is a drawing showing an example of a light effect detection operation using a guide image according to another embodiment of the present invention.

[0057] Referring to FIG. 7, the generation unit (235) of the content control module (230) can transmit a guide image (GI) to the display device (100), and the display device (100) can output the received guide image (GI). The guide image (GI) is intended to facilitate detection of a light effect (LE) by a fingernail (FN), and may include a guide marker (GM) to guide the user (30) to position the hand or fingernail. The guide marker (GM) may have a shape of the hand or fingernail of the user (30), but is not limited thereto.

[0058] The content control module (230) can generate a guide image (GI) at a predetermined time, for example, at the start time of the content, and output it through the display device (100), thereby guiding the user (30) to place a hand or fingernail on a guide marker (GM) within the guide image (GI). The guide image (GI) can further include visual and / or auditory notification information that guides the user (30) to place a finger or hand on the guide marker (GM).

[0059] The identification unit (231) of the content control module (230) can designate a guide marker (GM) or a preset range of areas based on the guide marker (GM) as an area of ​​interest, and can detect only light effects (LE) within the area of ​​interest as valid values. That is, the identification unit (231) can determine light reflection effects outside the area of ​​interest as noise. Accordingly, the light effect (LE) detection performance of the content control module (230) can be improved.

[0060] The present invention described above can be created as a program to be executed on a computer and stored in a computer-readable recording medium. Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. In addition, the computer-readable recording medium can be distributed to computer systems connected to a network, so that the computer-readable code can be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the above method can be easily inferred by programmers in the technical field to which the present invention pertains. In addition, in describing the present invention, '~ module' can be implemented in various ways, for example, by a processor, program instructions executed by a processor, a software module, microcode, a computer program product, a logic circuit, an application-specific integrated circuit, firmware, etc.

[0061] It will be apparent to those skilled in the art that the present invention can be embodied in other specific forms without departing from the essential characteristics thereof. Therefore, the above detailed description should not be construed as limiting in any respect, but rather as illustrative. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the scope of equivalents of the present invention are intended to be included within the scope of the present invention.

Claims

1. A display device that outputs content including at least one object; and A content providing device that determines an interaction between the object and the user and provides the content reflecting a preset event effect in response to the interaction to the display device, The above content providing device, A shooting module for capturing a content area including at least the content output through the display device; and A content control module that detects a light effect based on physical information provided from the user's fingernails from an image obtained from the photographing module and sets the light effect as a user recognition criterion for determining the presence of the user within the content area, The above physical information is, Contains biophoton information provided from the user's fingernails, An interaction system that sets user recognition criteria based on the user's fingernails.

2. In paragraph 1, The above content providing device, Further comprising a lighting module that irradiates light toward the user's fingernails, The above content control module, By inducing light reflection by the user's fingernails through the light irradiated from the above lighting module, the above light effect is additionally detected. An interaction system that sets user recognition criteria based on the user's fingernails.

3. In paragraph 2, The above content control module, Obtaining first and second images acquired at different times from the above-described shooting module, and detecting the presence of the user's fingernails based on the difference in the light effect detected on the first and second images, The above lighting module, Driven in conjunction with the detection operation of the light effect by the above content control module, An interaction system that sets user recognition criteria based on the user's fingernails.

Images