Operation recognition apparatus and operation recognition method

Abstract

To provide an operation recognition apparatus that is capable of determining an operation by a hand movement even if the contrast of an image is low.SOLUTION: An imaging unit 3a captures an image of a hand and outputs image data. An operation determination unit 3b determines an operation by a hand movement based on the image data. A DMS 2 further includes a storage unit 2c for storing hand shape information that associates a hand with a face. The operation determination unit 3b estimates the fingertip positions of the hand based on the hand shape information.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an operation recognition device and an operation recognition method for recognizing an operation by a hand movement based on image data. [Background technology]

[0002] Conventionally, various operation recognition devices that recognize operations by hand movements have been proposed, and are disclosed, for example, in Patent Document 1. Such operation recognition devices use a camera to capture hand gestures in an area where an image is projected, and recognize the gestures. In the image captured by the camera, a hand area is determined using a color threshold value of the image data, and hand gestures are recognized. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2006-18276 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, if the background of the gesturing hand (e.g., the right hand) is a part of the body (e.g., the left hand), there is a risk of erroneously determining the hand region. In other words, if the colors of the hand region and the background region in the image data are similar, it is difficult to determine the hand region based on the contrast of the image, and there is a problem that the hand region is erroneously determined and the hand gesture cannot be recognized. [Means for solving the problem]

[0005] The operation recognition device disclosed herein is an operation recognition device 1 including an imaging unit 3a that outputs imaging data by imaging a hand and a face, and an operation determination unit 3b that determines an operation by hand movement based on the imaging data, and further includes a memory unit 2c that stores hand shape information that associates the hand and the face, and the operation determination unit 3b estimates the fingertip position of the hand based on the hand shape information. The operation recognition method disclosed herein is an operation recognition method in an operation recognition device having a memory unit that stores hand shape information associated with a hand and a face, and includes a step of inputting imaging data from an imaging unit that outputs imaging data by imaging the hand and the face, and an operation determination step of determining an operation performed by a hand movement based on the imaging data, wherein the operation determination step estimates the fingertip position of the hand based on the hand shape information.

[0006] Moreover, the operation recognition device disclosed herein is characterized in that the hand shape information stored in the storage unit 2c includes image data of a left hand and a right hand.

[0007] Moreover, the operation recognition device disclosed herein is characterized in that the hand shape information stored in the storage unit 2c includes image data of the hand in an open state and in a clenched state.

[0008] Moreover, the operation recognition device disclosed herein is characterized in that the hand shape information stored in the storage unit 2c includes image data of the back of the hand and the palm of the hand.

[0009] Moreover, the manipulation recognition device of the present disclosure is characterized by further comprising a projection unit 5 that projects an image onto the hand. Effect of the Invention

[0010] Since the positions of the fingertips can be estimated based on the hand shape information, operations based on hand movements can be determined even when the contrast of the image is low. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing an embodiment of the present invention. [Diagram 2] FIG. [Diagram 3] FIG. [Figure 4] FIG. 4 is a diagram of a touch operation assistance image according to the embodiment. [Diagram 5] 4A to 4C are explanatory diagrams of a touch operation in the embodiment; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, with reference to the accompanying drawings, a description will be given of an embodiment in which the present invention is applied to an operation recognition device 1. The operation recognition device 1 includes a DMS (Driver Monitoring System) 2, a hand recognition unit 3, a display control unit 4, a projection unit 5, and a control unit 6.

[0013] The DMS2 has an imaging camera 2a, a determination unit 2b, and a storage unit 2c. The imaging camera 2a of the DMS2 captures an image of the face and hands of the vehicle driver, and outputs the image data to the determination unit 2b. The determination unit 2b of the DMS2 identifies the vehicle driver based on the image data output by the imaging camera 2a. The storage unit 2c of the DMS2 stores data on the face of the vehicle driver, hand shape data, etc.

[0014] The hand recognition unit 3 has an imaging camera 3a and an operation determination unit 3b. The imaging camera 3a of the hand recognition unit 3 is composed of a stereo camera, captures an image of the vehicle driver's hand, and outputs the image data to the operation determination unit 3b. The operation determination unit 3b determines the operation of the vehicle driver based on the image data output from the imaging camera 3a.

[0015] The display control unit 4 outputs image data to the projection unit 5 based on a control signal from the control unit 6. The projection unit 5 is composed of a projector installed inside the vehicle cabin, and projects an image onto the hand of the vehicle driver.

[0016] 2 is a flow diagram for registering personal information of a vehicle driver. In step S01, the image of the vehicle driver is captured by the imaging camera 2a of the DMS 2, and the face of the vehicle driver is recognized. In step S02, it is determined whether the face of the driver recognized in step S01 is registered in the storage unit 2c. If the face of the driver is already registered in the storage unit 2c, the process proceeds to step S07, and if the face is not registered in the storage unit 2c, the process proceeds to step S03.

[0017] In step S03, a facial image of the vehicle driver is registered in the storage unit 2c. In step S04, an image of the vehicle driver's hands is captured by the imaging camera 2a of the DMS 2. In step S05, a touch operation assist image, which will be described later, is generated based on the image of the vehicle driver's hands captured in step S4.

[0018] In step S6, the driver individual information that associates the face image of the vehicle driver registered in the storage unit 2c in step S03, the hand image of the vehicle driver captured in step S04, and the touch operation assist image generated in step S05 is registered in the storage unit 2c. In step S07, the personal settings registered in the storage unit 2c are read.

[0019] 3 shows an example of a hand image of a vehicle driver captured in step S04. In step S04, it is desirable to capture at least eight images of the left hand, right hand, closed hand, open hand, palm side, and back side of the hand, but this is not a limitation.

[0020] Fig. 4 shows an example of a touch operation assistance image. Although the length of a finger varies depending on the vehicle driver, in this embodiment, the fingertip position can be estimated based on the vehicle driver's individual information. In other words, even if the fingertip position cannot be recognized from image data alone due to low image contrast, the operation by the movement of the hand can be determined.

[0021] The control unit 6 causes the projection unit 5 to project an image displaying entertainment and air conditioning adjustment functions using voice recognition and steering switches onto one hand (e.g., the left hand) of the vehicle driver via the display control unit 4. The vehicle driver operates the entertainment and air conditioning by touching / swiping the image projected onto one hand (e.g., the left hand) with the other hand (e.g., the right hand). The control unit 6 captures images of both hands of the vehicle driver with the imaging camera 3a of the hand recognition unit 3, and recognizes the operation of the vehicle driver with the operation determination unit 3b of the hand recognition unit 3.

[0022] 5(a) and 5(b) are examples of images of both hands of a vehicle driver. As in Fig. 5(a), when the index finger of the right hand overlaps the palm of the left hand of the vehicle driver in the image, it is difficult to determine the fingertip position of the index finger of the right hand only from the image data because of low contrast, but as in Fig. 5(b), the operation recognition device 1 can recognize the operation by the right hand by estimating the fingertip position of the index finger of the right hand based on the touch operation assist image.

[0023] The present invention is not limited to the present embodiment, and various modifications are possible. For example, the present embodiment includes the imaging camera 2a and the imaging camera 3a, but the number of imaging cameras may be one as long as the face and hands of the vehicle driver can be captured. [Explanation of symbols]

[0024] 1 Operation recognition device 2. DMS 2b Judgment part 2c storage section 3 Hand recognition section 3a Imaging section 3b Operation judgment section 4. Display control section 5 Projection part 6 Control section

Claims

1. An operation recognition device including an operation determination unit that receives image data from an imaging unit that outputs image data by imaging a hand and a face, and determines an operation by a hand movement based on the image data, A storage unit that stores hand shape information that associates the hand and the face, The operation determination unit estimates a fingertip position of the hand based on the hand shape information.

2. The operation recognition device according to claim 1 , wherein the hand shape information stored in the storage unit includes image data of a left hand and a right hand.

3. 3. The operation recognition device according to claim 1, wherein the hand shape information stored in the storage unit includes image data of the hand in an open state and in a clenched state.

4. 4. The operation recognition device according to claim 1, wherein the hand shape information stored in the storage unit includes image data of the back of the hand and the palm of the hand.

5. An operation recognition device described in any one of claims 1 to 4, further comprising a projection unit that projects an image onto the hand.

6. A method for recognizing an operation in an operation recognition device having a memory unit that stores hand shape information associated with a hand and a face, comprising: inputting the imaging data from an imaging unit that outputs imaging data by imaging the hand and the face; and determining an operation by a hand movement based on the imaging data, The operation recognition method is characterized in that, in the operation determination step, a fingertip position of the hand is estimated based on the hand shape information.

Images