Gesture detection device, occupant monitoring system, and gesture detection method

Abstract

A gesture detection device (1) comprises: a hand candidate detection unit (12) that, on the basis of a captured image obtained by imaging a person, detects a hand candidate that is a candidate for a hand of the person; a gesture detection unit (13) that detects a gesture of the person on the basis of the hand candidate detected by the hand candidate detection unit; a face mask determination unit (14) that, on the basis of the captured image, determines whether or not the person is wearing a face mask; a shielding detection unit (15) that, in a case where the face mask determination unit has determined that the person is not wearing a face mask, determines whether or not the person's mouth is shielded, on the basis of information on the person's face obtained on the basis of the captured image; and a determination unit (16) that, in a case where the shielding detection unit has determined that the mouth of the person is shielded, rejects the gesture detected by the gesture detection unit.

Description

Gesture detection device, passenger monitoring system, and gesture detection method

[0001] The present disclosure relates to a gesture detection device, an occupant monitoring system, and a gesture detection method.

[0002] Conventionally, technologies for detecting a person's gestures from an image captured by an imaging device such as a camera have been known. For example, Patent Document 1 describes an information processing system including an authentication unit that performs biometric authentication using biometric information of a first user, an acquisition unit that acquires information about the first user's behavioral history if the biometric authentication is successful, a storage unit that accumulates information about the behavioral history, a generation unit that generates risk information indicating a risk related to the first user based on the accumulated information about the behavioral history, and an output unit that outputs the risk information of the first user in response to a request from a second user different from the first user. The authentication unit (biometric authentication unit) particularly includes a motion information acquisition unit that is configured to detect gestures around an obstruction (e.g., a mask) that partially covers the first user's face when performing biometric authentication of the first user and acquire motion information corresponding to the gesture. Patent Document 1 also describes that the motion information acquisition unit may include, for example, a camera, and may detect a gesture such as bringing a finger to a position overlapping the mask as a gesture.

[0003] International Publication No. 2022 / 264206

[0004] As described above, Patent Document 1 describes gesture detection when a person (first user) has a mask covering their face, i.e., when they are wearing a mask, but does not describe gesture detection when they are not wearing a mask. Meanwhile, when a person eats or drinks, they remove their mask and bring their hand holding food or drink to their mouth. In this case, the person's hand-to-mouth movement is not intended to be a gesture, so it is not desirable for this movement to be detected as a gesture. However, as described above, Patent Document 1 does not describe gesture detection when a person is not wearing a mask, and therefore the system described in Patent Document 1 may erroneously detect a person's eating or drinking movements as a gesture.

[0005] The present disclosure has been made to solve the above-mentioned problems, and aims to provide a gesture detection device that can prevent erroneous detection of actions taken when a person is eating or drinking as a gesture.

[0006] The gesture detection device according to the present disclosure includes a hand candidate detection unit that detects hand candidates that are candidates for the hands of a person based on an image of the person; a gesture detection unit that detects gestures of the person based on the hand candidates detected by the hand candidate detection unit; a mask determination unit that determines whether the person is wearing a mask based on the image; an occlusion detection unit that, when the mask determination unit determines that the person is not wearing a mask, determines whether the person's mouth is occluded based on information about the person's face obtained from the image; and a determination unit that, when the occlusion detection unit determines that the person's mouth is occluded, rejects the gesture detected by the gesture detection unit.

[0007] According to the present disclosure, it is possible to prevent a person's actions when eating or drinking from being erroneously detected as a gesture.

[0008] 3A and 3B are diagrams illustrating an example of the configuration of an occupant monitoring system including a gesture detection device according to Embodiment 1. FIG. 3A is a flowchart for explaining an example of the operation of the gesture detection device according to Embodiment 1. FIG. 3B is a diagram illustrating an example of the hardware configuration of the gesture detection device according to Embodiment 1.

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0010] 1 is a diagram showing a configuration example of an occupant monitoring system 100 including a gesture detection device 1 according to embodiment 1. In the following description, a case where the gesture detection device 1 according to embodiment 1 is mounted in the occupant monitoring system 100 will be described as an example.

[0011] The occupant monitoring system 100 includes an imaging device 110, a gesture detection device 1, and a control device 120, as shown in FIG.

[0012] The imaging device 110 is configured by, for example, a camera installed in a vehicle (not shown) and captures images of the interior of the vehicle, including the vehicle occupants, in a time-series manner. The imaging device 110 sequentially outputs images obtained by capturing images (hereinafter also referred to as “captured images”) to the gesture detection device 1.

[0013] The gesture detection device 1 detects (recognizes) a gesture made by a person (here, a vehicle occupant) based on the captured image output from the imaging device 110, and outputs information indicating the detected gesture to the control device 120.

[0014] The control device 120 executes predetermined control based on the gesture indicated by the information output from the gesture detection device 1. For example, if the gesture indicated by the information output from the gesture detection device 1 is a gesture for operating an in-vehicle device such as an air conditioner or audio system, the control device 120 executes the temperature adjustment of the air conditioner, the volume adjustment of the audio system, etc. based on the gesture. However, the in-vehicle device is not limited to the air conditioner and audio system.

[0015] <Gesture detection device 1> As shown in FIG. 1 , the gesture detection device 1 includes an image acquisition unit 10, a face information acquisition unit 11, a hand candidate detection unit 12, a gesture detection unit 13, a mask determination unit 14, an occlusion detection unit 15, and a determination unit 16.

[0016] The image acquisition unit 10 acquires a captured image output from the imaging device 110. The image acquisition unit 10 outputs the acquired captured image to the face information acquisition unit 11, the hand candidate detection unit 12, and the mask determination unit 14.

[0017] The face information acquisition unit 11 acquires information about the face of the occupant (hereinafter also referred to as “face information”) based on the captured image output from the image acquisition unit 10. The face information acquisition unit 11 outputs the acquired face information to the occlusion detection unit 15.

[0018] The facial information of the occupant is, for example, information indicating an image obtained by cutting out the area in which the occupant's face is captured from the captured image output from the image acquisition unit 10. The facial information of the occupant may include information indicating parts of the face, such as the eyes, eyebrows, nose, mouth, forehead, cheeks, or chin, and information indicating the positions of those parts in the captured image.

[0019] The hand candidate detection unit 12 detects hand candidates that are candidates for the occupant's hands based on the captured image output from the image acquisition unit 10. The detected hand candidates also include the position and shape of the hand candidates. The hand candidate detection unit 12 outputs information indicating the detected hand candidates to the gesture detection unit 13.

[0020] The hand candidate detection unit 12 detects hand candidates of the occupant by, for example, matching an object shape pattern (luminance distribution information) in the captured image output from the image acquisition unit 10 with a predetermined hand shape pattern, i.e., by pattern matching processing. The hand shape to be detected may be either an open hand shape or a closed hand shape. Furthermore, the hand shape to be detected may be, for example, a hand shape indicating a number, a hand shape indicating a direction, a hand shape indicating the occupant's intention (e.g., OK or Good), etc.

[0021] The hand candidate detection unit 12 may also detect hand candidates using, for example, a machine learning model. In this case, the machine learning model may be a trained model that has been trained to output a result of inferring hand candidates of an occupant in response to an input of a captured image of the occupant.

[0022] The gesture detection unit 13 detects a gesture of the occupant based on the position and shape of the hand candidate indicated by the information output from the hand candidate detection unit 12. The gesture detection unit 13 outputs information indicating the detected gesture of the occupant to the determination unit 16.

[0023] The gesture detection unit 13 detects the gesture of the occupant using, for example, a machine learning model. In this case, the machine learning model is a trained model that has been trained to output, in response to input of information indicating hand candidates of the occupant, a result of inferring a gesture corresponding to the hand candidate indicated by the information.

[0024] The mask determination unit 14 determines whether or not the occupant is wearing a mask based on the captured image output from the image acquisition unit 10. The mask determination unit 14 outputs information indicating the determination result to the occlusion detection unit 15 and the determination unit 16.

[0025] The mask determination unit 14 determines whether or not the occupant is wearing a mask by using, for example, a machine learning model. In this case, the machine learning model is a trained model that has been trained to output an inference result as to whether or not the occupant is wearing a mask in response to an input of a captured image of the occupant.

[0026] The mask determination unit 14 may determine whether or not the occupant is wearing a mask based on the facial information of the occupant acquired by the facial information acquisition unit 11. In this case, the mask determination unit 14 may use, as the trained model, a trained model that has been trained to output an inference result as to whether or not the occupant is wearing a mask in response to input of facial information of the occupant.

[0027] When the information output from the mask determination unit 14 indicates that the occupant is not wearing a mask, the occlusion detection unit 15 determines whether the occupant's mouth is covered or not, based on the occupant's face information output from the face information acquisition unit 11. The occlusion detection unit 15 outputs information indicating the determination result to the determination unit 16. Note that the occlusion detection unit 15 is only required to make the above determination when the information output from the mask determination unit 14 indicates that the occupant is not wearing a mask, and is not required to make the above determination when the information output from the mask determination unit 14 indicates that the occupant is wearing a mask.

[0028] The occlusion detection unit 15 determines whether the occupant's mouth is occluded by using, for example, a machine learning model. In this case, the machine learning model is a trained model that has been trained to output an inference result as to whether the occupant's mouth is occluded in response to input of, for example, facial information of the occupant.

[0029] In the above description, the occlusion detection unit 15 has been described as determining whether the mouth of an occupant is occluded based on the facial information of the occupant output from the facial information acquisition unit 11. However, the occlusion detection unit 15 is not limited to this, and may determine whether part or all of the face of the occupant is occluded based on the facial information of the occupant output from the facial information acquisition unit 11. In this case, it is sufficient for the occlusion detection unit 15 to be able to determine whether at least the mouth of the occupant is occluded based on the facial information of the occupant output from the facial information acquisition unit 11, for example.

[0030] When the information output from the mask determination unit 14 indicates that the occupant is wearing a mask, the determination unit 16 recognizes the gesture indicated by the information output from the gesture detection unit 13 as the gesture of the occupant.

[0031] In addition, if the information output from the mask determination unit 14 indicates that the occupant is not wearing a mask and the information output from the occupancy detection unit 15 indicates that the occupant's mouth is not occupant's mouth, the determination unit 16 recognizes the gesture indicated by the information output from the gesture detection unit 13 as the gesture of the occupant.

[0032] On the other hand, if the information output from the mask determination unit 14 indicates that the occupant is not wearing a mask and the information output from the occupancy detection unit 15 indicates that the occupant's mouth is occupant's mouth is occupant's mouth, the determination unit 16 will reject the gesture indicated by the information output from the gesture detection unit 13 without recognizing it as the occupant's gesture.

[0033] In addition, when the information output from the mask determination unit 14 indicates that the occupant is not wearing a mask, the determination unit 16 may calculate the distance between the hand candidate and the occupant's mouth based on the information indicating the hand candidate detected by the hand candidate detection unit 12 and the occupant's face information acquired by the face information acquisition unit 11 before determining whether to reject or recognize the gesture based on the information output from the occlusion detection unit 15.

[0034] For example, when the information output from the mask determination unit 14 indicates that the occupant is not wearing a mask, the determination unit 16 acquires a detection box for hand candidates based on the information indicating hand candidates detected by the hand candidate detection unit 12, and acquires a detection box for the occupant's mouth based on the occupant's face information acquired by the face information acquisition unit 11. The determination unit 16 can acquire the detection box for the occupant's hand candidates and the detection box for the occupant's mouth using a known method. Then, the determination unit 16 calculates the minimum distance between the acquired detection box for the occupant's hand candidate and the detection box for the occupant's mouth as the distance between the hand candidate and the occupant's mouth.

[0035] Then, when the calculated distance exceeds a predetermined threshold, the determination unit 16 may recognize the gesture detected by the gesture detection unit 13 as a gesture of the occupant, regardless of the information output from the occlusion detection unit 15. Furthermore, the determination unit 16 may determine to reject or recognize the gesture based on the information output from the occlusion detection unit 15 only when the calculated distance is equal to or less than the predetermined threshold. In this way, the determination unit 16 makes a determination based on the distance between the hand candidate and the occupant's mouth prior to determining to reject or recognize the gesture based on the information output from the occlusion detection unit 15, thereby increasing the opportunities to determine whether to reject or recognize the gesture and improving the accuracy of the determination regarding the rejection or recognition of the gesture.

[0036] Next, an operation example of the gesture detection device 1 according to the first embodiment will be described with reference to the flowchart shown in Fig. 2. In the following description, the determination unit 16 performs a determination based on the distance between a hand candidate and the occupant's mouth before determining whether to reject or recognize a gesture based on information output from the occlusion detection unit 15.

[0037] First, the image acquisition unit 10 acquires a captured image output from the imaging device 110 (step ST1). The image acquisition unit 10 outputs the acquired captured image to the face information acquisition unit 11, the hand candidate detection unit 12, and the mask determination unit 14.

[0038] Next, the face information acquisition unit 11 acquires information about the face of the occupant (face information) based on the captured image output from the image acquisition unit 10 (step ST2). The face information acquisition unit 11 outputs the acquired face information to the occlusion detection unit 15 and the determination unit 16.

[0039] Next, the hand candidate detection unit 12 detects hand candidates that are candidates for the occupant's hands based on the captured image output from the image acquisition unit 10 (step ST3). The hand candidate detection unit 12 outputs information indicating the detected hand candidates to the gesture detection unit 13 and the determination unit 16.

[0040] Next, the gesture detection unit 13 detects a gesture of the occupant based on the position and shape of the hand candidate indicated by the information output from the hand candidate detection unit 12 (step ST4). The gesture detection unit 13 outputs information indicating the detected gesture of the occupant to the determination unit 16.

[0041] Next, the mask determination unit 14 determines whether or not the occupant is wearing a mask based on the captured image output from the image acquisition unit 10 (step ST5). The mask determination unit 14 outputs information indicating the determination result to the occlusion detection unit 15 and the determination unit 16.

[0042] Next, the determination unit 16 checks whether the information output from the mask determination unit 14 indicates that the occupant is wearing a mask, i.e., whether the mask determination unit 14 has determined that the occupant is wearing a mask (step ST6). As a result, if it is confirmed that the mask determination unit 14 has determined that the occupant is wearing a mask (step ST6; YES), the determination unit 16 does not reject the gesture detected by the gesture detection unit 13 in step ST4, but recognizes it as a gesture made by the occupant (step ST7).

[0043] On the other hand, if the judgment unit 16 confirms that the mask judgment unit 14 has determined that the occupant is not wearing a mask (step ST6; NO), it calculates the distance between the hand candidate and the occupant's mouth based on the information indicating the hand candidate detected by the hand candidate detection unit 12 and the occupant's face information acquired by the face information acquisition unit 11 (step ST8).

[0044] The determination unit 16 then checks whether the calculated distance is equal to or less than a predetermined threshold (step ST9). If the calculated distance is not equal to or less than the threshold (step ST9; NO), the process proceeds to step ST7, where the determination unit 16 does not reject the gesture detected by the gesture detection unit 13 in step ST4, but recognizes it as a gesture made by an occupant. On the other hand, if the calculated distance is equal to or less than the threshold (step ST9; YES), the process proceeds to step ST10.

[0045] In step ST10, the occlusion detection unit 15 determines whether the occupant's mouth is occluded or not based on the occupant's face information output from the face information acquisition unit 11, and outputs information indicating the determination result to the determination unit 16. Then, the determination unit 16 checks whether the information output from the occlusion detection unit 15 indicates that the occupant's mouth is occluded or not (step ST10).

[0046] As a result, if it is confirmed that the information output from the occlusion detection unit 15 indicates that the occupant's mouth is occluded (step ST10; YES), the judgment unit 16 rejects the gesture indicated by the information output from the gesture detection unit 13 in step ST4 without recognizing it as a gesture of the occupant (step ST11).

[0047] On the other hand, if the judgment unit 16 confirms that the information output from the occlusion detection unit 15 indicates that the occupant's mouth is not occluded (step ST10; NO), it does not reject the gesture detected by the gesture detection unit 13 in step ST4, but recognizes it as an occupant's gesture (step ST7).

[0048] In the above description, the determination unit 16 calculates the distance between the hand candidate and the occupant's mouth in steps ST8 to ST9 before determining whether to reject or recognize the gesture based on the information output from the occlusion detection unit 15 in step ST10, and performs a determination based on the calculated distance. However, the processing of steps ST8 to ST9 is not essential and may be omitted. In this case, once it is confirmed in step ST6 that the occupant is not wearing a mask, the processing may proceed to step ST10. However, if the determination unit 16 performs the processing of steps ST8 to ST9, the number of opportunities to determine whether to reject or recognize the gesture can be increased, thereby improving the accuracy of the determination regarding whether to reject or recognize the gesture.

[0049] As described above, in the gesture detection device 1 according to the first embodiment, when the mask determination unit 14 determines that the occupant is not wearing a mask, the occlusion detection unit 15 determines whether the occupant's mouth is occluded. In the gesture detection device 1, when the occlusion detection unit 15 determines that the occupant's mouth is occluded, the determination unit 16 rejects the gesture detected by the gesture detection unit 13. This enables the gesture detection device 1 to prevent erroneous detection of the occupant's actions when eating or drinking as a gesture.

[0050] In the above explanation, an example was described in which the target of gesture detection by the gesture detection device 1 is a vehicle occupant, but the target of gesture detection is not limited to a vehicle occupant, and may be any person who can be imaged by an imaging device.

[0051] Next, an example of a hardware configuration of the gesture detection device 1 according to the first embodiment will be described with reference to Fig. 3. The functions of the image acquisition unit 10, face information acquisition unit 11, hand candidate detection unit 12, gesture detection unit 13, mask determination unit 14, occlusion detection unit 15, and determination unit 16 in the gesture detection device 1 are realized by processing circuits. The processing circuit may be dedicated hardware as shown in Fig. 3A, or may be a CPU (also referred to as a central processing unit, processing unit, arithmetic unit, microprocessor, microcomputer, processor, or DSP (Digital Signal Processor)) 52 that executes a program stored in a memory 53 as shown in Fig. 3B.

[0052] When the processing circuit is dedicated hardware, the processing circuit 51 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. The functions of each of the image acquisition unit 10, the face information acquisition unit 11, the hand candidate detection unit 12, the gesture detection unit 13, the mask determination unit 14, the occlusion detection unit 15, and the determination unit 16 may be realized individually by the processing circuit 51, or the functions of each unit may be realized collectively by the processing circuit 51.

[0053] When the processing circuit is a CPU 52, the functions of the image acquisition unit 10, face information acquisition unit 11, hand candidate detection unit 12, gesture detection unit 13, mask determination unit 14, occlusion detection unit 15, and determination unit 16 are realized by software, firmware, or a combination of software and firmware. The software and firmware are written as programs and stored in the memory 53. The processing circuit realizes the functions of each unit by reading and executing the programs recorded in the memory 53. That is, the gesture detection device 1 includes a memory for storing programs that, when executed by the processing circuit, result in the execution of, for example, each step shown in FIG. 2 . Furthermore, these programs can also be said to cause a computer to execute the procedures and methods of the image acquisition unit 10, face information acquisition unit 11, hand candidate detection unit 12, gesture detection unit 13, mask determination unit 14, occlusion detection unit 15, and determination unit 16. Here, examples of the memory 53 include non-volatile or volatile semiconductor memories such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable ROM), and EEPROM (Electrically EPROM), magnetic disks, flexible disks, optical disks, compact disks, minidisks, and DVDs (Digital Versatile Discs).

[0054] Note that the functions of the image acquisition unit 10, the face information acquisition unit 11, the hand candidate detection unit 12, the gesture detection unit 13, the mask determination unit 14, the occlusion detection unit 15, and the determination unit 16 may be partially realized by dedicated hardware and partially realized by software or firmware. For example, the function of the image acquisition unit 10 may be realized by a processing circuit as dedicated hardware, and the functions of the face information acquisition unit 11, the hand candidate detection unit 12, the gesture detection unit 13, the mask determination unit 14, the occlusion detection unit 15, and the determination unit 16 may be realized by the processing circuit reading and executing a program stored in the memory 53.

[0055] Thus, the processing circuitry can implement each of the above-described functions by hardware, software, firmware, or a combination thereof.

[0056] As described above, according to the first embodiment, the gesture detection device 1 includes the hand candidate detection unit 12 that detects hand candidates that are candidates for the hand of a person based on a captured image of the person, the gesture detection unit 13 that detects a gesture of the person based on the hand candidates detected by the hand candidate detection unit 12, the mask determination unit 14 that determines whether the person is wearing a mask based on the captured image, the occlusion detection unit 15 that, when the mask determination unit 14 determines that the person is not wearing a mask, determines whether the person's mouth is covered based on information about the person's face obtained from the captured image, and the determination unit 16 that, when the occlusion detection unit 15 determines that the person's mouth is covered, rejects the gesture detected by the gesture detection unit 13. This makes it possible for the gesture detection device 1 according to the first embodiment to prevent erroneous detection of actions taken when a person is eating or drinking as a gesture.

[0057] Furthermore, when the mask determination unit 14 determines that the person is wearing a mask, the determination unit 16 recognizes the gesture detected by the gesture detection unit 13 as a human gesture. This allows the gesture detection device 1 according to the first embodiment to detect gestures made by a person wearing a mask.

[0058] Furthermore, when the mask determination unit 14 determines that the person is not wearing a mask and the occlusion detection unit 15 determines that the person's mouth is not occluded, the determination unit 16 recognizes the gesture detected by the gesture detection unit 13 as a human gesture. This allows the gesture detection device 1 according to the first embodiment to detect, as a gesture, actions other than eating and drinking actions when the person is not wearing a mask.

[0059] Furthermore, when the mask determination unit 14 determines that the person is not wearing a mask, the determination unit 16 calculates the distance between the hand candidate and the person's mouth based on the position of the hand candidate detected by the hand candidate detection unit 12 and information about the person's face, and when the calculated distance is equal to or less than a threshold and the occlusion detection unit 15 determines that the person's mouth is occluded, the determination unit 16 rejects the gesture detected by the gesture detection unit 13, and when the calculated distance exceeds the threshold and when the occlusion detection unit 15 determines that the person's mouth is not occluded even when the calculated distance is equal to or less than the threshold, the determination unit 16 recognizes the gesture detected by the gesture detection unit 13 as a human gesture. This allows the gesture detection device 1 according to the first embodiment to increase the opportunities for determining whether to reject or recognize a gesture, thereby improving the determination accuracy.

[0060] Furthermore, the mask determination unit 14 determines whether a person is wearing a mask using a trained model that outputs whether a person is wearing a mask in response to an input captured image of the person. This allows the gesture detection device 1 according to the first embodiment to accurately determine whether a person is wearing a mask, thereby improving the accuracy of determination regarding rejection or recognition of gestures.

[0061] Furthermore, the occlusion detection unit 15 determines whether a person's mouth is occluded or not using a trained model that outputs whether a person's mouth is occluded or not in response to input of information about the person's face. This allows the gesture detection device 1 according to the first embodiment to accurately determine whether a person's mouth is occluded or not, thereby improving the accuracy of determination regarding rejection or recognition of gestures.

[0062] The gesture detection unit 13 receives information indicating the positions and shapes of hand candidates as input, and detects human gestures using a trained model that outputs gestures corresponding to the positions and shapes of hand candidates indicated by the input information. This enables the gesture detection device 1 according to the first embodiment to accurately detect human gestures.

[0063] The occupant monitoring system 100 according to the first embodiment is configured to include the gesture detection device 1, and is equipped with an imaging device 110 that images a vehicle occupant as a person, and a control device 120 that executes predetermined control based on the gesture recognized by the determination unit 16. As a result, the occupant monitoring system 100 according to the first embodiment can suppress erroneous control caused by erroneously detecting the actions of the vehicle occupant when eating or drinking as a gesture.

[0064] In addition, in the present disclosure, any component of the embodiments may be modified or any component of the embodiments may be omitted.

[0065] The present disclosure can prevent erroneous detection of actions taken when a person is eating or drinking as a gesture, and is suitable for use in gesture detection devices, occupant monitoring systems, and gesture detection methods.

[0066] REFERENCE SIGNS LIST 1 gesture detection device, 10 image acquisition unit, 11 face information acquisition unit, 12 hand candidate detection unit, 13 gesture detection unit, 14 mask determination unit, 15 occlusion detection unit, 16 determination unit, 51 processing circuit, 52 CPU, 53 memory, 100 occupant monitoring system, 110 imaging device, 120 control device.

Claims

1. A hand candidate detection unit that detects a hand candidate, which is a candidate for the hand of the person, based on a captured image of the person; A gesture detection unit that detects a gesture of the person based on the hand candidate detected by the hand candidate detection unit; A mask determination unit that determines whether or not the person is wearing a mask based on the captured image; A shielding detection unit that determines whether or not the person's mouth is shielded based on the face information of the person obtained based on the captured image when the mask determination unit determines that the person is not wearing a mask; A determination unit that rejects the gesture detected by the gesture detection unit when the shielding detection unit determines that the person's mouth is shielded; A gesture detection device comprising the above.

2. The determination unit: when the mask determination unit determines that the person is wearing a mask, recognizes the gesture detected by the gesture detection unit as the gesture of the person; The gesture detection device according to claim 1, characterized in that.

3. The determination unit: when the mask determination unit determines that the person is not wearing a mask and the shielding detection unit determines that the person's mouth is not shielded, recognizes the gesture detected by the gesture detection unit as the gesture of the person; The gesture detection device according to claim 1 or claim 2, characterized in that.

4. The determination unit: when the mask determination unit determines that the person is not wearing a mask, calculates the distance between the hand candidate and the person's mouth based on the position of the hand candidate detected by the hand candidate detection unit and the face information of the person; when the calculated distance is less than or equal to a threshold value and the shielding detection unit determines that the person's mouth is shielded, rejects the gesture detected by the gesture detection unit; when the calculated distance exceeds the threshold value, and even when the calculated distance is less than or equal to the threshold value and the shielding detection unit determines that the person's mouth is not shielded, recognizes the gesture detected by the gesture detection unit as the gesture of the person; The gesture detection device according to claim 3, characterized in that.

5. The mask determination unit: uses a learned model that outputs whether or not the person is wearing a mask in response to an input of a captured image of the person to determine whether or not the person is wearing a mask. The gesture detection device according to claim 1 or claim 2, characterized in that...

6. The occlusion detection unit determines whether or not the person's mouth is occluded by using a learned model that outputs whether or not the person's mouth is occluded in response to the input of information on the person's face. The gesture detection device according to claim 1 or claim 2, characterized in that...

7. The gesture detection unit detects the person's gesture by using a learned model that takes, as input, information indicating the position and shape of the hand candidate and outputs a gesture corresponding to the position and shape of the hand candidate indicated by the information. The gesture detection device according to claim 1 or claim 2, characterized in that...

8. A passenger monitoring system configured to include the gesture detection device according to claim 1 or claim 2, an imaging device that images a passenger of a vehicle as the person, and a control device that executes predetermined control based on the gesture recognized by the determination unit. A passenger monitoring system comprising the above.

9. A gesture detection method using a gesture detection device, wherein a hand candidate detection unit detects a hand candidate that is a candidate for the person's hand based on a captured image of the person; a gesture detection unit detects the person's gesture based on the hand candidate detected by the hand candidate detection unit; a mask determination unit determines whether or not the person is wearing a mask based on the captured image; when the mask determination unit determines that the person is not wearing a mask, an occlusion detection unit determines whether or not the person's mouth is occluded based on the information on the person's face obtained from the captured image; when the occlusion detection unit determines that the person's mouth is occluded, a determination unit rejects the gesture detected by the gesture detection unit; A gesture detection method having the above steps.