Gesture detection device, passenger monitoring system, and gesture detection method

Abstract

A gesture detection device (1) comprises: a hand candidate detection unit (12) that, on the basis of a captured image obtained by imaging a person, detects a hand candidate that is a candidate for a hand of the person; a gesture detection unit (13) that detects a gesture of the person on the basis of the hand candidate detected by the hand candidate detection unit; a face mask determination unit (14) that, on the basis of the captured image, determines whether or not the person is wearing a face mask; a shielding detection unit (15) that, in a case where the face mask determination unit has determined that the person is not wearing a face mask, determines whether or not the person's mouth is shielded, on the basis of information on the person's face obtained on the basis of the captured image; and a determination unit (16) that, in a case where the shielding detection unit has determined that the mouth of the person is shielded, rejects the gesture detected by the gesture detection unit.

Description

[Technical Field]

[0001] The present disclosure relates to a gesture detection device, an occupant monitoring system, and a gesture detection method. [Background technology]

[0002] Conventionally, there is known a technique for detecting a gesture of a person from an image captured by an imaging device such as a camera. For example, Patent Document 1 describes an information processing system including an authentication unit that performs biometric authentication using biometric information of a first user, an acquisition unit that acquires information about the first user's behavioral history if the biometric authentication is successful, a storage unit that stores the information about the behavioral history, a generation unit that generates risk information indicating risks related to the first user based on the stored information about the behavioral history, and an output unit that outputs the risk information of the first user in response to a request from a second user different from the first user. The authentication unit (biometric authentication unit) particularly includes a motion information acquisition unit that is configured to detect gestures around an obstruction (e.g., a mask) that partially covers the first user's face when performing biometric authentication of the first user and acquire motion information corresponding to the gesture. Patent Document 1 also describes that the motion information acquisition unit may include, for example, a camera, and may detect a gesture such as bringing a finger to a position overlapping the mask as a gesture. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2022 / 264206 Summary of the Invention [Problem to be solved by the invention]

[0004] As described above, Patent Document 1 describes gesture detection when a person (first user) has their face covered with a mask, i.e., when they are wearing a mask, but does not describe gesture detection when they are not wearing a mask. On the other hand, when a person eats or drinks, they remove their mask and bring their hand holding food or drink to their mouth. In this case, the action of bringing their hand to their mouth is not intended to be any kind of gesture, so it is essentially undesirable for this action to be detected as a gesture. However, as described above, Patent Document 1 does not describe gesture detection when a person is not wearing a mask, so there is a risk that the system described in Patent Document 1 may erroneously detect actions taken when a person is eating or drinking as gestures.

[0005] The present disclosure has been made to solve the above-mentioned problems, and aims to provide a gesture detection device that can prevent erroneous detection of actions taken when a person is eating or drinking as a gesture. [Means for solving the problem]

[0006] The gesture detection device according to the present disclosure includes a hand candidate detection unit that detects hand candidates that are candidates for the hands of a person based on an image of the person; a gesture detection unit that detects gestures of the person based on the hand candidates detected by the hand candidate detection unit; a mask determination unit that determines whether the person is wearing a mask based on the image; an occlusion detection unit that, when the mask determination unit determines that the person is not wearing a mask, determines whether the person's mouth is occluded based on information about the person's face obtained from the image; and a determination unit that, when the occlusion detection unit determines that the person's mouth is occluded, rejects the gesture detected by the gesture detection unit. [Effects of the Invention]

[0007] According to the present disclosure, it is possible to prevent a person's actions when eating or drinking from being erroneously detected as a gesture. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram showing an example of the configuration of an occupant monitoring system including a gesture detection device according to a first embodiment. [Figure 2] 4 is a flowchart illustrating an example of the operation of the gesture detection device according to the first embodiment. [Figure 3] 3A and 3B are diagrams illustrating an example of a hardware configuration of the gesture detection device according to the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Embodiment 1

[0010] 1 is a diagram showing a configuration example of an occupant monitoring system 100 including a gesture detection device 1 according to embodiment 1. In the following description, a case where the gesture detection device 1 according to embodiment 1 is mounted in the occupant monitoring system 100 will be described as an example.

[0011] The occupant monitoring system 100 includes an imaging device 110, a gesture detection device 1, and a control device 120, as shown in FIG.

[0012] The imaging device 110 is configured by, for example, a camera installed in a vehicle (not shown) and captures images of the interior of the vehicle including the vehicle occupants in a time series. The imaging device 110 sequentially outputs images obtained by capturing images (hereinafter also referred to as "captured images") to the gesture detection device 1.

[0013] The gesture detection device 1 detects (recognizes) a gesture made by a person (here, a vehicle occupant) based on a captured image output from the imaging device 110, and outputs information indicating the detected gesture to the control device 120.

[0014] The control device 120 executes predetermined control based on the gesture indicated by the information output from the gesture detection device 1. For example, if the gesture indicated by the information output from the gesture detection device 1 is a gesture for operating an in-vehicle device such as an air conditioner or audio, the control device 120 executes the temperature adjustment of the air conditioner, the volume adjustment of the audio, etc. based on the gesture. However, the in-vehicle device is not limited to the air conditioner and audio.

[0015] <Gesture detection device 1> As shown in FIG. 1, the gesture detection device 1 includes an image acquisition unit 10, a face information acquisition unit 11, a hand candidate detection unit 12, a gesture detection unit 13, a mask determination unit 14, an occlusion detection unit 15, and a determination unit 16.

[0016] The image acquisition unit 10 acquires a captured image output from the imaging device 110. The image acquisition unit 10 outputs the acquired captured image to the face information acquisition unit 11, the hand candidate detection unit 12, and the mask determination unit 14.

[0017] The face information acquisition unit 11 acquires information about the face of the occupant (hereinafter also referred to as “face information”) based on the captured image output from the image acquisition unit 10. The face information acquisition unit 11 outputs the acquired face information to the occlusion detection unit 15.

[0018] The facial information of the occupant is, for example, information indicating an image obtained by cutting out the area in which the occupant's face is captured from the captured image output from the image acquisition unit 10. The facial information of the occupant may include information indicating parts of the face such as the eyes, eyebrows, nose, mouth, forehead, cheeks, or chin, and information indicating the positions of those parts in the captured image.

[0019] The hand candidate detection unit 12 detects hand candidates that are candidates for the occupant's hands based on the captured image output from the image acquisition unit 10. The detected hand candidates also include the position and shape of the hand candidates. The hand candidate detection unit 12 outputs information indicating the detected hand candidates to the gesture detection unit 13.

[0020] The hand candidate detection unit 12 detects hand candidates of the occupant by, for example, matching the object shape pattern (luminance distribution information) in the captured image output from the image acquisition unit 10 with a predetermined hand shape pattern, that is, by pattern matching processing. The hand shape to be detected may be either an open hand shape or a closed hand shape. The hand shape to be detected may also be, for example, a hand shape indicating a number, a hand shape indicating a direction, a hand shape indicating the occupant's intention (e.g., OK or Good), etc.

[0021] Furthermore, the hand candidate detection unit 12 may detect hand candidates using, for example, a machine learning model. In this case, the machine learning model may be a trained model that has been trained to output a result of inferring hand candidates of an occupant in response to an input of a captured image of the occupant.

[0022] The gesture detection unit 13 detects the gesture of the occupant based on the position and shape of the hand candidate indicated by the information output from the hand candidate detection unit 12. The gesture detection unit 13 outputs information indicating the detected gesture of the occupant to the determination unit 16.

[0023] The gesture detection unit 13 detects the gesture of the occupant using, for example, a machine learning model. In this case, the machine learning model is a trained model that has been trained to output, in response to input of information indicating hand candidates of the occupant, a result of inferring a gesture corresponding to the hand candidate indicated by the information.

[0024] The mask determination unit 14 determines whether or not the occupant is wearing a mask based on the captured image output from the image acquisition unit 10. The mask determination unit 14 outputs information indicating the determination result to the occlusion detection unit 15 and the determination unit 16.

[0025] The mask determination unit 14 determines whether or not an occupant is wearing a mask by using, for example, a machine learning model. In this case, the machine learning model is a trained model that has been trained to output an inference result as to whether or not an occupant is wearing a mask in response to an input of a captured image of the occupant.

[0026] The mask determination unit 14 may determine whether or not the occupant is wearing a mask based on the facial information of the occupant acquired by the facial information acquisition unit 11. In this case, the mask determination unit 14 may use, as the trained model, a trained model that has been trained to output an inference result as to whether or not the occupant is wearing a mask in response to input of facial information of the occupant.

[0027] When the information output from the mask determination unit 14 indicates that the occupant is not wearing a mask, the occlusion detection unit 15 determines whether the occupant's mouth is covered or not, based on the occupant's face information output from the face information acquisition unit 11. The occlusion detection unit 15 outputs information indicating the determination result to the determination unit 16. Note that the occlusion detection unit 15 is only required to make the above determination when the information output from the mask determination unit 14 indicates that the occupant is not wearing a mask, and is not required to make the above determination when the information output from the mask determination unit 14 indicates that the occupant is wearing a mask.

[0028] The occlusion detection unit 15 determines whether the occupant's mouth is occluded by using, for example, a machine learning model. In this case, the machine learning model is a trained model that has been trained to output an inference result as to whether the occupant's mouth is occluded in response to input of, for example, facial information of the occupant.

[0029] In the above description, the occlusion detection unit 15 has been described as determining whether the mouth of an occupant is occluded based on the facial information of the occupant output from the facial information acquisition unit 11. However, the occlusion detection unit 15 is not limited to this, and may determine whether part or all of the face of the occupant is occluded based on the facial information of the occupant output from the facial information acquisition unit 11. In this case, it is sufficient for the occlusion detection unit 15 to be able to determine whether at least the mouth of the occupant is occluded based on the facial information of the occupant output from the facial information acquisition unit 11, for example.

[0030] When the information output from the mask determination unit 14 indicates that the occupant is wearing a mask, the determination unit 16 recognizes the gesture indicated by the information output from the gesture detection unit 13 as the gesture of the occupant.

[0031] In addition, when the information output from the mask determination unit 14 indicates that the occupant is not wearing a mask and the information output from the occupancy detection unit 15 indicates that the occupant's mouth is not occupying a mask, the determination unit 16 recognizes the gesture indicated by the information output from the gesture detection unit 13 as the gesture of the occupant.

[0032] On the other hand, if the information output from the mask determination unit 14 indicates that the occupant is not wearing a mask and the information output from the occupancy detection unit 15 indicates that the occupant's mouth is occupant's mouth is occupant's mouth, the determination unit 16 rejects the gesture indicated by the information output from the gesture detection unit 13 without recognizing it as the occupant's gesture.

[0033] In addition, when the information output from the mask determination unit 14 indicates that the occupant is not wearing a mask, the determination unit 16 may calculate the distance between the hand candidate and the occupant's mouth based on the information indicating the hand candidate detected by the hand candidate detection unit 12 and the occupant's face information acquired by the face information acquisition unit 11 before determining whether to reject or recognize the gesture based on the information output from the occlusion detection unit 15.

[0034] For example, when the information output from the mask determination unit 14 indicates that the occupant is not wearing a mask, the determination unit 16 acquires a detection box for hand candidates based on the information indicating hand candidates detected by the hand candidate detection unit 12, and acquires a detection box for the occupant's mouth based on the occupant's face information acquired by the face information acquisition unit 11. The determination unit 16 can acquire the detection box for the occupant's hand candidates and the detection box for the occupant's mouth using a known method. Then, the determination unit 16 calculates the minimum distance between the acquired detection box for the occupant's hand candidate and the detection box for the occupant's mouth as the distance between the hand candidate and the occupant's mouth.

[0035] Then, when the calculated distance exceeds a predetermined threshold, the determination unit 16 may recognize the gesture detected by the gesture detection unit 13 as a gesture of the occupant, regardless of the information output from the occlusion detection unit 15. Furthermore, the determination unit 16 may determine to reject or recognize the gesture based on the information output from the occlusion detection unit 15 only when the calculated distance is equal to or less than the predetermined threshold. In this way, the determination unit 16 makes a determination based on the distance between the hand candidate and the occupant's mouth prior to determining to reject or recognize the gesture based on the information output from the occlusion detection unit 15, thereby increasing the opportunities to determine whether to reject or recognize the gesture and improving the accuracy of the determination regarding the rejection or recognition of the gesture.

[0036] Next, an operation example of the gesture detection device 1 according to the first embodiment will be described with reference to the flowchart shown in Fig. 2. In the following description, the determination unit 16 performs a determination based on the distance between a hand candidate and the occupant's mouth before determining whether to reject or recognize a gesture based on information output from the occlusion detection unit 15.

[0037] First, the image acquisition unit 10 acquires a captured image output from the imaging device 110 (step ST1). The image acquisition unit 10 outputs the acquired captured image to the face information acquisition unit 11, the hand candidate detection unit 12, and the mask determination unit 14.

[0038] Next, the face information acquisition unit 11 acquires information about the face of the occupant (face information) based on the captured image output from the image acquisition unit 10 (step ST2). The face information acquisition unit 11 outputs the acquired face information to the occlusion detection unit 15 and the determination unit 16.

[0039] Next, the hand candidate detection unit 12 detects hand candidates that are candidates for the occupant's hands based on the captured image output from the image acquisition unit 10 (step ST3). The hand candidate detection unit 12 outputs information indicating the detected hand candidates to the gesture detection unit 13 and the determination unit 16.

[0040] Next, the gesture detection unit 13 detects a gesture of the occupant based on the position and shape of the hand candidate indicated by the information output from the hand candidate detection unit 12 (step ST4). The gesture detection unit 13 outputs information indicating the detected gesture of the occupant to the determination unit 16.

[0041] Next, the mask determination unit 14 determines whether or not the occupant is wearing a mask based on the captured image output from the image acquisition unit 10 (step ST5). The mask determination unit 14 outputs information indicating the determination result to the occlusion detection unit 15 and the determination unit 16.

[0042] Next, the determination unit 16 checks whether the information output from the mask determination unit 14 indicates that the occupant is wearing a mask, i.e., whether the mask determination unit 14 has determined that the occupant is wearing a mask (step ST6). As a result, when it is confirmed that the mask determination unit 14 has determined that the occupant is wearing a mask (step ST6; YES), the determination unit 16 does not reject the gesture detected by the gesture detection unit 13 in step ST4, but recognizes it as a gesture of the occupant (step ST7).

[0043] On the other hand, if the judgment unit 16 confirms that the mask judgment unit 14 has determined that the occupant is not wearing a mask (step ST6; NO), it calculates the distance between the hand candidate and the occupant's mouth based on the information indicating the hand candidate detected by the hand candidate detection unit 12 and the occupant's face information acquired by the face information acquisition unit 11 (step ST8).

[0044] Then, the determination unit 16 checks whether the calculated distance is equal to or less than a predetermined threshold (step ST9). As a result, if the calculated distance is not equal to or less than the threshold (step ST9; NO), the process proceeds to step ST7, and the determination unit 16 does not reject the gesture detected by the gesture detection unit 13 in step ST4, but recognizes it as a gesture by an occupant. On the other hand, if the calculated distance is equal to or less than the threshold (step ST9; YES), the process proceeds to step ST10.

[0045] In step ST10, the occlusion detection unit 15 determines whether or not the occupant's mouth is occluded based on the face information of the occupant output from the face information acquisition unit 11, and outputs information indicating the determination result to the determination unit 16. Then, the determination unit 16 checks whether or not the information output from the occlusion detection unit 15 indicates that the occupant's mouth is occluded (step ST10).

[0046] As a result, if it is confirmed that the information output from the occlusion detection unit 15 indicates that the occupant's mouth is occluded (step ST10; YES), the judgment unit 16 rejects the gesture indicated by the information output from the gesture detection unit 13 in step ST4 without recognizing it as a gesture of the occupant (step ST11).

[0047] On the other hand, if the judgment unit 16 confirms that the information output from the occlusion detection unit 15 indicates that the occupant's mouth is not occluded (step ST10; NO), it does not reject the gesture detected by the gesture detection unit 13 in step ST4, but recognizes it as a gesture by the occupant (step ST7).

[0048] In the above description, the determination unit 16 calculates the distance between the hand candidate and the occupant's mouth in steps ST8 to ST9 before determining whether to reject or recognize the gesture based on the information output from the occlusion detection unit 15 in step ST10, and performs a determination based on the calculated distance. However, the processing of steps ST8 to ST9 is not essential and may be omitted. In that case, if it is confirmed in step ST6 that the occupant is determined not to be wearing a mask, the processing may proceed to step ST10. However, if the determination unit 16 performs the processing of steps ST8 to ST9, the number of opportunities to determine whether to reject or recognize the gesture can be increased, and the accuracy of the determination regarding the rejection or recognition of the gesture can be improved.

[0049] As described above, in the gesture detection device 1 according to the first embodiment, when the mask determination unit 14 determines that the occupant is not wearing a mask, the occlusion detection unit 15 determines whether the occupant's mouth is occluded. Then, in the gesture detection device 1, when the occlusion detection unit 15 determines that the occupant's mouth is occluded, the determination unit 16 rejects the gesture detected by the gesture detection unit 13. This enables the gesture detection device 1 to prevent erroneous detection of the occupant's actions when eating or drinking as a gesture.

[0050] In the above description, an example was described in which the target of gesture detection by the gesture detection device 1 was a vehicle occupant, but the target of gesture detection is not limited to a vehicle occupant, and may be any person who can be imaged by an imaging device.

[0051] Next, an example of a hardware configuration of the gesture detection device 1 according to the first embodiment will be described with reference to Fig. 3. The functions of the image acquisition unit 10, face information acquisition unit 11, hand candidate detection unit 12, gesture detection unit 13, mask determination unit 14, occlusion detection unit 15, and determination unit 16 in the gesture detection device 1 are realized by processing circuits. The processing circuit may be dedicated hardware as shown in Fig. 3A, or may be a CPU (also referred to as a central processing unit, processing unit, arithmetic unit, microprocessor, microcomputer, processor, or DSP (Digital Signal Processor)) 52 that executes a program stored in memory 53 as shown in Fig. 3B.

[0052] When the processing circuit is dedicated hardware, the processing circuit 51 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. The functions of the image acquisition unit 10, the face information acquisition unit 11, the hand candidate detection unit 12, the gesture detection unit 13, the mask determination unit 14, the occlusion detection unit 15, and the determination unit 16 may be realized individually by the processing circuit 51, or the functions of each unit may be realized collectively by the processing circuit 51.

[0053] When the processing circuit is a CPU 52, the functions of the image acquisition unit 10, face information acquisition unit 11, hand candidate detection unit 12, gesture detection unit 13, mask determination unit 14, occlusion detection unit 15, and determination unit 16 are realized by software, firmware, or a combination of software and firmware. The software and firmware are written as programs and stored in memory 53. The processing circuit realizes the functions of each unit by reading and executing the programs recorded in memory 53. That is, the gesture detection device 1 includes a memory for storing programs that, when executed by the processing circuit, result in the execution of, for example, each step shown in FIG. 2 . Furthermore, these programs can also be said to cause a computer to execute the procedures and methods of the image acquisition unit 10, face information acquisition unit 11, hand candidate detection unit 12, gesture detection unit 13, mask determination unit 14, occlusion detection unit 15, and determination unit 16. Here, examples of memory 53 include non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable ROM), EEPROM (Electrically EPROM), magnetic disk, flexible disk, optical disk, compact disk, mini disk, or DVD (Digital Versatile Disc).

[0054] Note that the functions of the image acquisition unit 10, the face information acquisition unit 11, the hand candidate detection unit 12, the gesture detection unit 13, the mask determination unit 14, the occlusion detection unit 15, and the determination unit 16 may be partially realized by dedicated hardware and partially realized by software or firmware. For example, the function of the image acquisition unit 10 may be realized by a processing circuit as dedicated hardware, and the functions of the face information acquisition unit 11, the hand candidate detection unit 12, the gesture detection unit 13, the mask determination unit 14, the occlusion detection unit 15, and the determination unit 16 may be realized by the processing circuit reading and executing a program stored in the memory 53.

[0055] Thus, the processing circuitry can implement each of the above-described functions by hardware, software, firmware, or a combination thereof.

[0056] As described above, according to the first embodiment, the gesture detection device 1 includes a hand candidate detection unit 12 that detects hand candidates that are candidates for the hand of a person based on a captured image of the person, a gesture detection unit 13 that detects a gesture of the person based on the hand candidates detected by the hand candidate detection unit 12, a mask determination unit 14 that determines whether the person is wearing a mask based on the captured image, an occlusion detection unit 15 that determines whether the person's mouth is covered based on information about the person's face obtained from the captured image when the mask determination unit 14 determines that the person is not wearing a mask, and a determination unit 16 that rejects the gesture detected by the gesture detection unit 13 when the occlusion detection unit 15 determines that the person's mouth is covered. This makes it possible for the gesture detection device 1 according to the first embodiment to prevent erroneous detection of a person's actions when eating or drinking as a gesture.

[0057] Furthermore, when the mask determination unit 14 determines that the person is wearing a mask, the determination unit 16 recognizes the gesture detected by the gesture detection unit 13 as a human gesture. This allows the gesture detection device 1 according to the first embodiment to detect gestures made by a person wearing a mask.

[0058] Furthermore, when the mask determination unit 14 determines that the person is not wearing a mask and the occlusion detection unit 15 determines that the person's mouth is not occluded, the determination unit 16 recognizes the gesture detected by the gesture detection unit 13 as a human gesture. This allows the gesture detection device 1 according to the first embodiment to detect, as a gesture, actions other than eating and drinking actions when the person is not wearing a mask.

[0059] Furthermore, when mask determination unit 14 determines that the person is not wearing a mask, determination unit 16 calculates the distance between the hand candidate and the person's mouth based on the position of the hand candidate detected by hand candidate detection unit 12 and information about the person's face, and when the calculated distance is equal to or less than a threshold and occlusion detection unit 15 determines that the person's mouth is occluded, determination unit 16 rejects the gesture detected by gesture detection unit 13, and when the calculated distance exceeds the threshold and when occlusion detection unit 15 determines that the person's mouth is not occluded even if the calculated distance is equal to or less than the threshold, determination unit 16 recognizes the gesture detected by gesture detection unit 13 as a human gesture. This allows the gesture detection device 1 according to the first embodiment to increase the opportunities to determine whether to reject or recognize a gesture, thereby improving determination accuracy.

[0060] Furthermore, the mask determination unit 14 determines whether a person is wearing a mask using a trained model that outputs whether or not a person is wearing a mask in response to an input captured image of the person. This allows the gesture detection device 1 according to the first embodiment to accurately determine whether or not a person is wearing a mask, thereby improving the accuracy of determination regarding rejection or recognition of gestures.

[0061] Furthermore, the occlusion detection unit 15 determines whether the person's mouth is occluded or not by using a trained model that outputs whether the person's mouth is occluded or not in response to input of information about the person's face. This allows the gesture detection device 1 according to the first embodiment to accurately determine whether the person's mouth is occluded or not, thereby improving the accuracy of determination regarding rejection or recognition of gestures.

[0062] The gesture detection unit 13 receives information indicating the positions and shapes of hand candidates as input, and detects human gestures using a trained model that outputs gestures corresponding to the positions and shapes of the hand candidates indicated by the information. This allows the gesture detection device 1 according to the first embodiment to accurately detect human gestures.

[0063] Moreover, the occupant monitoring system 100 according to the first embodiment is configured to include the gesture detection device 1, and is provided with an imaging device 110 that images a vehicle occupant as a person, and a control device 120 that executes predetermined control based on the gesture recognized by the determination unit 16. As a result, the occupant monitoring system 100 according to the first embodiment can suppress erroneous control caused by erroneously detecting the actions of the vehicle occupant when eating or drinking as a gesture.

[0064] In addition, in the present disclosure, any component of the embodiments may be modified or any component of the embodiments may be omitted. [Industrial Applicability]

[0065] The present disclosure can prevent erroneous detection of actions taken when a person is eating or drinking as a gesture, and is suitable for use in gesture detection devices, occupant monitoring systems, and gesture detection methods. [Explanation of symbols]

[0066] 1 gesture detection device, 10 image acquisition unit, 11 face information acquisition unit, 12 hand candidate detection unit, 13 gesture detection unit, 14 mask determination unit, 15 occlusion detection unit, 16 determination unit, 51 processing circuit, 52 CPU, 53 memory, 100 occupant monitoring system, 110 imaging device, 120 control device.

Claims

1. a hand candidate detection unit that detects hand candidates that are candidates for the hands of a person based on a captured image of the person; a gesture detection unit that detects a gesture of the person based on the hand candidates detected by the hand candidate detection unit; a mask determination unit that determines whether the person is wearing a mask based on the captured image; an occlusion detection unit that, when it is determined by the mask determination unit that the person is not wearing a mask, determines whether the person's mouth is occluded based on facial information of the person obtained from the captured image; and a determination unit that rejects the gesture detected by the gesture detection unit when the occlusion detection unit determines that the person's mouth is occluded; A gesture detection device comprising:

2. The determination unit If the mask determination unit determines that the person is wearing a mask, the gesture detected by the gesture detection unit is recognized as a gesture of the person.

2. The gesture detection device according to claim 1, wherein:

3. The determination unit When the mask determination unit determines that the person is not wearing a mask, and when the occlusion detection unit determines that the person's mouth is not occluded, the gesture detected by the gesture detection unit is recognized as a gesture of the person.

3. The gesture detection device according to claim 1, wherein the gesture detection device further comprises: a first detecting section;

4. The determination unit when the mask determination unit determines that the person is not wearing a mask, calculates a distance between the hand candidate and the person's mouth based on the position of the hand candidate detected by the hand candidate detection unit and information about the person's face; If the calculated distance is equal to or less than a threshold and the occlusion detection unit determines that the person's mouth is occluded, reject the gesture detected by the gesture detection unit; When the calculated distance exceeds a threshold, and when the occlusion detection unit determines that the person's mouth is not occluded even if the calculated distance is equal to or less than the threshold, the gesture detected by the gesture detection unit is recognized as a gesture of the person.

4. The gesture detection device according to claim 3, wherein:

5. The mask determination unit In response to an input of a captured image of the person, a trained model is used to output whether the person is wearing a mask or not, and the trained model determines whether the person is wearing a mask or not.

3. The gesture detection device according to claim 1, wherein the gesture detection device further comprises: a first detecting section;

6. The occlusion detection unit In response to input of the person's face information, a trained model is used to output whether the person's mouth is occluded or not, and the trained model determines whether the person's mouth is occluded or not.

3. The gesture detection device according to claim 1, wherein the gesture detection device further comprises: a first detecting section;

7. The gesture detection unit The gesture of the person is detected using a trained model that receives information indicating the position and shape of the hand candidate as input and outputs a gesture corresponding to the position and shape of the hand candidate indicated by the information.

3. The gesture detection device according to claim 1, wherein the gesture detection device further comprises: a first detecting section;

8. An occupant monitoring system including the gesture detection device according to claim 1 or 2, an imaging device that images a vehicle occupant as the person; a control device that executes a predetermined control based on the gesture recognized by the determination unit; An occupant monitoring system with

9. A gesture detection method using a gesture detection device, comprising: a step in which a hand candidate detection unit detects hand candidates that are candidates for the hands of a person based on a captured image of the person; a gesture detection unit detecting a gesture of the person based on the hand candidates detected by the hand candidate detection unit; a step of a mask determination unit determining whether or not the person is wearing a mask based on the captured image; a step of, when the mask determination unit determines that the person is not wearing a mask, using an occlusion detection unit to determine whether or not the person's mouth is occluded based on face information of the person obtained from the captured image; a determination unit rejecting the gesture detected by the gesture detection unit when the occlusion detection unit determines that the person's mouth is occluded; A gesture detection method comprising:

Images