Determination device and determination method

What is AI technical title?
AI technical title is built by Patsnap AI team. It summarizes the technical point description of the patent document.
The determination device uses machine learning for action and mask detection to prevent false alarms by controlling warnings based on mask presence, addressing the challenge of accurate mouth movement detection in masked subjects.

JP7721045B2Active Publication Date: 2025-08-08MITSUBISHI ELECTRIC MOBILITY CORP

View PDF 4 Cites 0 Cited by

Patent Information

Application Number: JP2025527207
Authority / Receiving Office: JP · JP
Patent Type: Patents
Current Assignee / Owner
Filing Date: 2023-06-19
Publication Date: 2025-08-08
Estimated Expiration: 2043-06-19

AI Technical Summary

Technical Problem

Conventional methods struggle to accurately determine whether a subject is performing actions involving mouth movements, such as smoking or eating, without issuing false alarms, especially when the subject is wearing a mask, due to the challenges of object size and facial orientation, leading to unnecessary warnings.

Method used

A determination device that includes a target movement determination unit, a mask wearing determination unit, and a determination control unit to prevent false alarms by controlling the output of warnings when a mask is detected, using machine learning models for both actions and mask detection, and integrating these units to manage alarm necessity.

Benefits of technology

Prevents false alarms by accurately distinguishing between actions like smoking or eating when a mask is worn, ensuring appropriate warnings are issued only when necessary, thus reducing user inconvenience.

✦ Generated by Eureka AI based on patent content.

Smart Images

Figure 0007721045000001
Figure 0007721045000002
Figure 0007721045000003

Patent Text Reader

Abstract

The present invention is provided with: a target motion determination unit (11) that, on the basis of a captured image obtained by capturing the face of a target person who is a target for determining whether or not a target motion involving mouth movement has been performed, executes target motion determination for determining whether or not the target person is performing the target motion; a mask wearing determination unit (12) that, on the basis of the captured image, executes mask wearing determination for determining whether or not the target person is wearing a mask; an alarm necessity determination unit (14) for determining that an output of an alarm is necessary when a motion determination result indicating that the target motion is being performed by the target person is outputted from the target motion determination unit (11); and a determination control unit (13) for performing control so that the alarm necessity determination unit (14) does not determine that the output of the alarm is necessary when the mask wearing determination unit (12) has determined that the target person is wearing a mask.

Need to check novelty before this filing date? Find Prior Art

Description

[Technical Field]

[0001] The present disclosure relates to a determination device and a determination method for determining whether a person is performing an action involving mouth movement that is subject to an alarm. [Background technology]

[0002] Conventionally, a technology is known that determines, based on an image of a person's face, whether the person is performing an action involving mouth movement that is subject to an alarm (hereinafter referred to as a "target action"), such as smoking or eating or drinking in an inappropriate place, and issues an alarm if the person is performing the target action. For example, Patent Document 1 discloses a technology that outputs warning information when a distracting behavior that may distract the driver, such as smoking, drinking water, eating, talking on the phone, entertainment, putting on makeup, etc., is detected based on an image of the driver captured in a vehicle. Incidentally, there are cases where a person (hereinafter referred to as "subject") who is to be judged as to whether or not he or she is performing a target motion is wearing a mask. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2021-516829 Summary of the Invention [Problem to be solved by the invention]

[0004] When determining whether a subject is performing a target action from a captured image, it is technically difficult to eliminate false positives. For example, when determining whether a subject is performing a target action based on the fact that the subject is holding a specific object near their mouth, depending on the size of the object, it may be erroneously determined that the subject is performing the target action even though they are not. Furthermore, depending on the subject's facial orientation, it may be erroneously determined that the subject is performing the target action even though they are not. If it is erroneously determined that the subject is performing the target action even though they are not, a false alarm will be issued based on the erroneous determination. In conventional technology, when the subject is wearing a mask and is clearly unable to perform the target action, a false alarm, which is technically unavoidable as described above, may be issued, which may cause inconvenience to the subject. Patent Document 1 discloses a method for determining whether a predetermined distraction has occurred by determining a motion target frame based on the characteristics of a driver image and classifying and detecting a predetermined distraction based on the motion target frame, and disclosing whether a predetermined distraction has occurred. It also discloses that the motion target frame includes a local region of the face and motion-related objects, and the motion-related objects include a mask. However, the technology disclosed in Patent Document 1 does not specifically describe how the mask is used when disclosing whether a predetermined distraction has occurred. Therefore, the technology disclosed in Patent Document 1 still does not solve the above-mentioned problem.

[0005] The present disclosure has been made to solve the above-mentioned problems, and aims to provide a determination device that prevents false alarms due to an obvious erroneous determination that a subject is performing a target action involving mouth movement when the subject is wearing a mask and is to be determined as to whether or not the subject is performing a target action involving mouth movement. [Means for solving the problem]

[0006] The determination device according to the present disclosure includes a target movement determination unit that performs a target movement determination to determine whether or not a target person is performing a target movement based on a captured image of the face of the person to be determined to be performing a target movement involving mouth movement; a mask wearing determination unit that performs a mask wearing determination to determine whether or not the subject is wearing a mask based on the captured image; an alarm necessity determination unit that determines that an alarm needs to be output when the target movement determination unit outputs a movement determination result indicating that the subject is performing the target movement; and a determination control unit that controls the alarm necessity determination unit not to determine that an alarm needs to be output when the mask wearing determination unit determines that the subject is wearing a mask. The determination control unit rejects the action determination result output from the target action determination unit when the target action determination unit determines that the target person is performing the target action and the mask wearing determination unit determines that the target person is wearing a mask, and the warning necessity determination unit determines that a warning is unnecessary when the determination control unit rejects the action determination result. It is something. [Effects of the Invention]

[0007] According to the present disclosure, when a subject who is to be judged as to whether or not he or she is performing a target action involving mouth movement is wearing a mask, it is possible to prevent a false alarm due to an obvious erroneous judgment that the subject is performing the target action. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of the configuration of a determination device according to a first embodiment. [Figure 2] 4 is a flowchart for explaining the operation of the determination device according to the first embodiment. [Figure 3] 3 is a flowchart for explaining the detailed operation of step ST3 in FIG. 2. [Figure 4] 10 is a flowchart illustrating the operation of the determination device in the first embodiment, in which the determination device stops execution of the target behavior determination when it determines that the driver is wearing a mask, thereby controlling the device so that it does not determine that an alarm needs to be output. [Figure 5] 5 is a flowchart for explaining the detailed operation of step ST12 in FIG. 4. [Figure 6] 6A and 6B are diagrams illustrating an example of a hardware configuration of the determining device according to the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Embodiment 1 FIG. 1 is a diagram illustrating an example of the configuration of a determination device 1 according to the first embodiment. The determination device 1 according to the first embodiment determines whether a person (hereinafter referred to as a "subject") is performing an action involving mouth movement (hereinafter referred to as a "target action") that is the subject of an alert. In the first embodiment, the target action is assumed to be, for example, an action involving mouth movement in an inappropriate place, etc., and an action that requires an alarm to be output when the action is performed. Specifically, in the first embodiment, the target action is assumed to be smoking or eating or drinking in a vehicle. In the following first embodiment, as an example, the determination device 1 is mounted on a vehicle (not shown) and determines whether the driver of the vehicle is performing a target action, specifically, whether the driver is smoking or eating or drinking. That is, in the following first embodiment, the target person is the driver of the vehicle. The driver of the vehicle may be wearing a mask. Note that in the first embodiment, a "mask" refers to a mask made of nonwoven fabric, gauze, or the like that covers the nose or mouth.

[0010] The determination device 1 is connected to a camera 2 and an alarm device 3 mounted on a vehicle. For example, the determination device 1, the camera 2, and the alarm device 3 may constitute a so-called DMS (Driver Monitoring System). The determination device 1 determines whether or not the driver is performing a target action based on an image (hereinafter referred to as a "captured image") of the driver's face captured by the camera 2. When the determination device 1 determines that the driver is performing a target action, it outputs information to the warning device 3 to cause it to output a warning.

[0011] The camera 2 is installed, for example, in the center of the vehicle's instrument panel, meter panel, steering column, A-pillar, or around the center console. The camera 2 is installed so as to be able to capture an image of at least the face of the occupant. Camera 2 is a visible light camera or an infrared camera. 1, only one camera 2 is installed on the vehicle, but this is merely an example. A plurality of cameras 2 may be installed on the vehicle. The camera 2 outputs the captured image to the determination device 1.

[0012] The warning device 3 is, for example, an audio output device installed in a vehicle, and outputs, for example, an alarm sound or a warning voice. The warning device 3 is installed, for example, in the center of the vehicle's instrument panel, meter panel, steering column, A-pillar, or around the center console. The warning device 3 may also be mounted on, for example, a navigation device (not shown). The warning device 3 is not limited to an audio output device, but may be, for example, a display device that displays a warning message or a warning icon, or a vibration generating device that generates vibrations. 1, only one alarm device 3 is connected to the determination device 1, but this is merely an example. A plurality of alarm devices 3 may be connected to the determination device 1.

[0013] The determination device 1 according to the first embodiment will be described in detail. The determination device 1 includes a target behavior determination unit 11, a mask wearing determination unit 12, a determination control unit 13, a warning necessity determination unit 14, and a wearing notification control unit 15.

[0014] The target action determination unit 11 acquires an image of the face of the vehicle driver from the camera 2, and performs action determination based on the acquired image to determine whether the driver is smoking or eating or drinking.

[0015] The target behavior determination unit 11 determines whether the driver is smoking or eating or drinking, for example, based on the captured image acquired from the camera 2 and a trained model (hereinafter referred to as "machine learning model"). In the following description, the machine learning model used by the target behavior determination unit 11 when determining whether the driver is smoking or eating or drinking is also referred to as "first machine learning model." The first machine learning model has been trained to receive an image of a person's face as input and output information indicating whether the person is performing a target action (hereinafter referred to as "target action information"). The target action information includes, for example, information indicating whether the person is smoking and information indicating whether the person is eating or drinking. Note that this is merely an example, and the target action information may also include, for example, information indicating whether the person is performing an action other than smoking or eating or drinking. The target action information only needs to include information indicating at least whether the person is smoking and whether the person is eating or drinking. In other words, the first machine learning model only needs to be a model that has been trained to receive, at least, an image of a person's face as input and output target action information including information indicating whether the person is smoking or eating or drinking.

[0016] When inputting a captured image into the first machine learning model and obtaining target action information indicating that the driver is smoking, the target action determination unit 11 determines that the driver is smoking. Furthermore, when inputting a captured image into the first machine learning model and obtaining target action information indicating that the driver is eating or drinking, the target action determination unit 11 determines that the driver is eating or drinking. On the other hand, when inputting a captured image into the first machine learning model and obtaining target action information indicating that the driver is not smoking, the target action determination unit 11 determines that the driver is not smoking. Furthermore, when inputting a captured image into the first machine learning model and obtaining target action information indicating that the driver is not eating or drinking, the target action determination unit 11 determines that the driver is not eating or drinking.

[0017] It should be noted that the above-described method is merely an example, and the target behavior determining unit 11 may determine whether the driver is smoking or eating or drinking using other methods based on the captured image. For example, the target operation determination unit 11 may acquire a heat source image from a heat source camera (not shown) mounted on the vehicle, and determine whether the driver is smoking or not depending on whether a heat source is detected around the driver's mouth based on the heat source image and the captured image acquired from the camera 2. The heat source camera is a so-called thermography that generates a heat source image according to the amount of infrared rays emitted from an object. The target action determination unit 11 performs known image recognition processing on the captured image and the heat source image to identify which area on the heat source image corresponds to the area around the driver's mouth. Note that the installation position and angle of view of the camera 2 and the installation position and angle of view of the heat source camera are known in advance, so the target action determination unit 11 can associate the area on the captured image with the area on the heat source image. Furthermore, for example, the target behavior determination unit 11 may determine whether the driver is eating or drinking by performing pattern matching between the captured image obtained from the camera 2 and an image of a beverage container such as a plastic bottle or tableware that has been prepared in advance.

[0018] When the target behavior determination unit 11 performs a target behavior determination as to whether the driver is smoking or eating or drinking, the target behavior determination unit 11 outputs the result (hereinafter referred to as “behavior determination result”) to the determination control unit 13. The movement determination result includes information indicating whether the driver is smoking or not, and information indicating whether the driver is eating or drinking or not.

[0019] The mask wearing determination unit 12 acquires an image of the face of the vehicle driver from the camera 2, and performs a mask wearing determination to determine whether the driver is wearing a mask based on the acquired image.

[0020] The mask wearing determination unit 12 determines whether or not the driver is wearing a mask based on, for example, the captured image acquired from the camera 2 and a machine learning model. In the following description, the machine learning model used by the mask wearing determination unit 12 to determine whether or not the driver is wearing a mask is also referred to as a "second machine learning model." The second machine learning model has been trained to receive an image of a person's face as input and output information indicating whether the person is wearing a mask (hereinafter referred to as "mask wearing information"). The mask wearing information includes, for example, information indicating whether the person is wearing a mask. Note that this is merely an example, and the mask wearing information may include, for example, information indicating the color, pattern, or type of mask (nonwoven fabric, gauze, etc.) in addition to the information indicating whether the person is wearing a mask. The mask wearing information only needs to include information indicating at least whether the person is wearing a mask. In other words, the second machine learning model only needs to be a model that has been trained to receive, at least, an image of a person's face as input and output mask wearing information including information indicating whether the person is wearing a mask.

[0021] When the captured image is input to the second machine learning model and mask wearing information indicating that a mask is being worn is obtained, the mask wearing determination unit 12 determines that the driver is wearing a mask. On the other hand, when the captured image is input to the second machine learning model and mask wearing information indicating that a mask is not being worn is obtained, the mask wearing determination unit 12 determines that the driver is not wearing a mask.

[0022] The above-described method is merely an example, and the mask wearing determiner 12 may determine whether or not the driver is wearing a mask using other methods based on the captured image. For example, the mask wearing determination unit 12 may determine whether the driver is wearing a mask based on whether or not a characteristic point of the nose can be extracted based on the captured image. If the mask wearing determination unit 12 cannot extract a characteristic point of the nose, it determines that the driver is wearing a mask. The mask wearing determination unit 12 may extract feature points representing parts of the driver's face (specifically, the nose) in the captured image using a known technique such as a method using Haar-Like features or pattern matching.

[0023] When the mask wearing determination unit 12 determines whether the driver is wearing a mask or not, it outputs the result (hereinafter referred to as the “mask determination result”) to the determination control unit 13. The mask determination result includes information indicating whether the driver is wearing a mask.

[0024] Furthermore, if the mask wearing determination unit 12 determines that the driver is wearing a mask as a result of performing the mask wearing determination, it outputs information indicating that the driver is wearing a mask (hereinafter referred to as "mask present information") to the wearing notification control unit 15.

[0025] When the mask wearing determination unit 12 determines that the driver is wearing a mask, the determination control unit 13 controls the warning necessity determination unit 14 so that output of a warning is not necessary.

[0026] Here, the warning necessity determining unit 14 will be described. The warning necessity determining unit 14 executes a warning necessity determination to determine whether a warning needs to be output. The alarm necessity determination unit 14 determines that an alarm needs to be output when the target behavior determination unit 11 outputs a behavior determination result indicating that the driver is smoking or eating or drinking in the alarm necessity determination. The alarm necessity determination unit 14 acquires the behavior determination result output from the target behavior determination unit 11 via the determination control unit 13. When it is determined that an alarm needs to be output, the alarm necessity determining unit 14 outputs, to the alarm device 3, information for causing the alarm to be output (hereinafter referred to as "alarm output control information"). For example, if the alarm device 3 is an audio output device, the alarm output control information is information for causing the audio output device to output an alarm sound or an alarm message warning against smoking or drinking. For example, if the alarm device 3 is a display device, the alarm output control information is information for causing the display device to display an alarm message warning against smoking or eating or drinking. For example, if the alarm device 3 is a vibration generating device, the alarm output control information is information for causing the vibration generating device to generate vibrations warning against smoking or eating or drinking. The warning necessity determining unit 14 may change the type of warning sound, the content of the warning message, or the type of vibration depending on whether the driver is smoking or eating or drinking.

[0027] In addition, the alarm necessity determination unit 14 determines that an alarm is unnecessary when the target action determination unit 11 outputs an action determination result via the determination control unit 13 indicating that the driver is not smoking or eating or drinking, or when the determination control unit 13 outputs information indicating that the action determination result has been rejected (hereinafter referred to as "rejection information").

[0028] Returning to the description of the determination control unit 13. When the mask wearing determination unit 12 determines that the driver is wearing a mask, the determination control unit 13 controls the warning necessity determination unit 14 to determine whether or not an alarm is necessary, as described above, so that the warning necessity determination unit 14 does not need to output an alarm.

[0029] An example of a method in which the determination control unit 13 controls the warning necessity determination unit 14 to determine that it is not necessary to output a warning will be described. The determination control unit 13 first determines whether the target action determination unit 11 has determined that the driver is smoking or eating or drinking and whether the mask wearing determination unit 12 has determined that the driver is wearing a mask. The determination control unit 13 can determine whether the driver has been determined to be smoking or eating or drinking from the action determination result output from the target action determination unit 11. Furthermore, the determination control unit 13 can determine whether the driver has been determined to be wearing a mask from the mask determination result output from the mask wearing determination unit 12.

[0030] If the target action determination unit 11 determines that the driver is smoking or eating and drinking and the mask wearing determination unit 12 determines that the driver is wearing a mask, the determination control unit 13 rejects the action determination result output from the target action determination unit 11. That is, the determination control unit 13 prevents the movement determination result output from the target movement determination unit 11, more specifically, the movement determination result that the driver is smoking or eating or drinking, from being output to the warning necessity determination unit 14. At this time, the determination control unit 13 outputs the rejection information to the warning necessity determination unit 14 instead of outputting the movement determination result output from the target movement determination unit 11. The action determination result output by the target action determination unit 11, that the driver is smoking or eating or drinking, is no longer output to the alarm necessity determination unit 14, and instead rejection information is output, so that the alarm necessity determination unit 14 determines, as a result of performing an alarm necessity determination, that an alarm is not necessary.

[0031] Note that the above example is merely one example, and for example, the judgment control unit 13 may rewrite the action judgment result output from the target action judgment unit 11, which indicates that the driver is smoking or eating or drinking, into an action judgment result indicating that the driver is not smoking or eating or drinking, and output it to the alarm necessity judgment unit 14. In this case as well, the behavior determination result indicating that the driver is smoking or eating or drinking is not output, so the warning necessity determining unit 14 determines that a warning is not necessary.

[0032] If the target action determination unit 11 does not determine that the driver is smoking or eating or drinking, or if the mask wearing determination unit 12 determines that the driver is not wearing a mask, the determination control unit 13 outputs the action determination result output from the target action determination unit 11 as is to the alarm necessity determination unit 14.

[0033] When the mask wearing determination unit 12 determines that the subject is wearing a mask, in other words, when mask presence information is output from the mask wearing determination unit 12, the wearing notification control unit 15 outputs information to the alarm device 3 to notify that the subject is wearing a mask (hereinafter referred to as "wearing notification information"). Here, the wearing notification control unit 15 outputs wearing notification information to the warning device 3 when the mask wearing determination unit 12 determines that the driver is wearing a mask.

[0034] For example, if the warning device 3 is an audio output device, the wearing notification information is information for causing the audio output device to output a notification sound or a message notifying that the driver is wearing a mask. For example, if the warning device 3 is a display device, the wearing notification information is information for causing the display device to display a message or an icon notifying that the driver is wearing a mask. For example, if the warning device 3 is a vibration generating device, the wearing notification information is information for causing the vibration generating device to generate vibration notifying that the driver is wearing a mask. For example, as described above, the type of warning sound or vibration that warns the driver that he is smoking or eating or drinking and the type of notification sound or vibration that notifies the driver that he is wearing a mask may be different types of sound or vibration.

[0035] In addition, in the first embodiment, the wearing notification control unit 15 outputs the wearing notification information to the warning device 3, but this is merely an example. The wearing notification control unit 15 may output the wearing notification information to, for example, an output device (not shown) other than the warning device 3.

[0036] The operation of the determination device 1 according to the first embodiment will be described. FIG. 2 is a flowchart for explaining the operation of the determination device 1 according to the first embodiment. For example, when the power of the vehicle is turned on and the power of the determination device 1 is turned on, the determination device 1 starts the operation shown in the flowchart of Figure 2, and repeats the operation shown in the flowchart of Figure 2 until the power of the determination device 1 is turned off.

[0037] The target action determination unit 11 acquires an image of the face of the vehicle driver from the camera 2, and performs action determination based on the acquired image to determine whether the driver is smoking or eating or drinking (step ST1). The target action determination unit 11 outputs the action determination result to the determination control unit 13.

[0038] The mask wearing determination unit 12 acquires an image of the face of the vehicle driver from the camera 2, and performs a mask wearing determination to determine whether the driver is wearing a mask based on the acquired image (step ST2). The mask wearing determiner 12 outputs the mask wearing determination result to the determination controller 13. Furthermore, when the mask wearing determination unit 12 determines that the driver is wearing a mask as a result of the mask wearing determination, it outputs mask presence information to the wearing notification control unit 15.

[0039] The determination control unit 13 outputs the action determination result output from the target action determination unit 11 in step ST1 to the warning necessity determination unit 14. At this time, if the mask wearing determination unit 12 determines in step ST2 that the driver is wearing a mask, the determination control unit 13 performs processing to control the warning necessity determination unit 14 so that output of a warning is not necessary (step ST3).

[0040] The alarm necessity determining unit 14 executes an alarm necessity determination to determine whether an alarm needs to be output (step ST4). The alarm necessity determination unit 14 determines that an alarm needs to be output when, in the alarm necessity determination, the target behavior determination unit 11 outputs a behavior determination result indicating that the driver is smoking or eating or drinking via the determination control unit 13. When determining that an alarm needs to be output, the alarm necessity determination unit 14 outputs alarm output control information to the alarm device 3. The alarm necessity determination unit 14 determines that an alarm is unnecessary when the target action determination unit 11 outputs an action determination result indicating that the driver is not smoking or eating or drinking via the determination control unit 13, or when rejection information is output from the determination control unit 13.

[0041] If the mask wearing determination unit 12 determines in step ST2 that the driver is wearing a mask, in other words, if the mask wearing determination unit 12 outputs mask presence information, the wearing notification control unit 15 outputs wearing notification information to the warning device 3 (step ST4).

[0042] 2, the processing of step ST2 is performed after the processing of step ST1, but this is merely an example. The processing of step ST1 and the processing of step ST2 may be performed in parallel, or the processing of step ST1 may be performed after the processing of step ST2. It is sufficient that the processing of step ST1 and the processing of step ST2 are performed before the processing of step ST3 is performed. 2, the process of step ST5 is performed after the process of step ST4, but this is merely an example. The process of step ST5 may be performed in parallel with the processes of steps ST3 to ST4, or may be performed before the process of step ST3. The process of step ST5 may be performed at an appropriate timing after the process of step ST2.

[0043] FIG. 3 is a flowchart for explaining the detailed operation of step ST3 in FIG. The determination control unit 13 determines whether the target operation determination unit 11 has determined in step ST1 that the driver is smoking or eating or drinking (step ST301). If the target operation determination unit 11 determines in step ST301 that the driver is smoking or eating or drinking (if "YES" in step ST301), the determination control unit 13 determines whether the mask wearing determination unit 12 has determined in step ST2 that the driver is wearing a mask (step ST302).

[0044] In step ST302, if the mask wearing determination unit 12 determines that the driver is wearing a mask (if "YES" in step ST302), the determination control unit 13 rejects the action determination result output from the target action determination unit 11. At this time, the determination control unit 13 outputs rejection information to the warning necessity determination unit 14 instead of outputting the movement determination result output from the target movement determination unit 11. The determination control unit 13 may rewrite the movement determination result output from the target movement determination unit 11, that the driver is smoking or eating or drinking, into a movement determination result that the driver is not smoking or eating or drinking, and output this to the warning necessity determination unit 14. Then, the determination control unit 13 ends the operation shown in the flowchart of FIG. 3.

[0045] On the other hand, if the target operation determination unit 11 determines in step ST301 that the driver is not smoking or eating or drinking (if "NO" in step ST301), or if the mask wearing determination unit 12 determines in step ST302 that the driver is not wearing a mask (if "NO" in step ST302), the determination control unit 13 outputs the operation determination result output from the target operation determination unit 11 as is to the warning necessity determination unit 14, and terminates the operation shown in the flowchart of Figure 3.

[0046] In this way, the determination device 1 performs target behavior determination to determine whether the driver is smoking or eating or drinking, and mask wearing determination to determine whether the driver is wearing a mask, based on the captured image. The determination device 1 performs alarm necessity determination to determine whether an alarm needs to be output, and if it is determined that the driver is eating, drinking, or smoking, it determines that an alarm needs to be output. However, if it is determined that the driver is wearing a mask, the determination device 1 does not determine that an alarm needs to be output. In detail, if the determination device 1 determines that the driver is smoking or eating and drinking and also determines that the driver is wearing a mask, it rejects the behavior determination result that the driver is smoking or eating and drinking and determines that an alarm is unnecessary. This allows the determination device 1 to prevent a false alarm due to an obvious erroneous determination that the driver is smoking or eating or drinking when the driver is wearing a mask.

[0047] As described above, it is technically difficult to eliminate erroneous determinations when determining whether or not a subject is performing a target action from a captured image. For example, when determining whether the driver is smoking or eating or drinking based on a captured image, depending on the relative positions of the driver and camera 2 (for example, if camera 2 is installed on the center console), or the direction of the driver's face, interior objects or the scenery outside the vehicle may be reflected next to the driver's face in the captured image. Because whether the driver is smoking or eating or drinking is determined based on a two-dimensional captured image, there is a possibility that the reflected interior objects or scenery or other background objects may be mistakenly determined to be cigarettes even if they are actually far away. Furthermore, for example, the machine learning model (first machine learning model) may not have learned about cases where the background objects or the way the light hits the object are unusual and could not be anticipated. Furthermore, because cigarettes are small objects, the driver's hands are also a factor in determining whether a person is smoking. Therefore, for example, if a driver brings their hand close to their mouth without holding a cigarette, they may be mistakenly determined to be smoking. For example, if a driver touches their mouth over a mask, they may also be mistakenly determined to be smoking. As mentioned above, when background objects are reflected in the captured image or when the machine learning model has not yet been trained, adverse conditions overlap, increasing the likelihood of a false positive. Such misidentification of a background object in the captured image as a cigarette, insufficient learning in the machine learning model, or the driver touching their mouth without holding a cigarette may result in a false determination that the driver is smoking or eating or drinking when they are not, which may result in a false alarm indicating that the driver is smoking or eating or drinking when they are not. In particular, if the driver is wearing a mask, it is unlikely that the driver is smoking or eating or drinking. If a false alarm like the one described above is issued in such a situation where it is assumed that the driver is clearly not smoking or eating or drinking, it may be an annoyance to the driver.

[0048] In contrast, as described above, when the determination device 1 according to embodiment 1 determines that the driver is smoking or eating and drinking and also determines that the driver is wearing a mask, it rejects the action determination result that the driver is smoking or eating and drinking and determines that an alarm is unnecessary. This allows the determination device 1 to prevent a false alarm due to an obvious erroneous determination that the driver is smoking or eating or drinking when the driver is wearing a mask, in other words, in a situation where it is assumed that the driver is clearly not smoking or eating or drinking.

[0049] Furthermore, when the determination device 1 determines that the driver is wearing a mask, it outputs wearing notification information to notify the driver that the mask is being worn. This allows the determination device 1 to notify the driver that it has been determined that the driver is wearing a mask. There are various types, colors, patterns, and shapes of masks that drivers can wear. There are technical limitations to accurately determining that all of these are masks, and the determination device 1 may not be able to determine that the driver is wearing a mask, even if the mask has a different shape or pattern from a typical mask, for example. In such a case, if the determination device 1 erroneously determines that the driver is smoking or eating or drinking, as described above, a false alarm may be issued. On the other hand, at this time, the determination device 1 does not notify the user that the mask is being worn. Even if a false alarm is issued, the driver will not be notified that they are wearing a mask, and will be able to understand that the false alarm was issued because it was not determined that they were wearing a mask.

[0050] In the above-described embodiment 1, in the determination device 1, when the target action determination unit 11 determines that the driver is smoking or eating and drinking and the mask wearing determination unit 12 determines that the driver is wearing a mask, the determination control unit 13 rejects the action determination result output from the target action determination unit 11, thereby controlling the alarm necessity determination unit 14 to determine that it is not necessary to output an alarm. However, this is merely an example, and the determination control unit 13 may use other methods to control the alarm necessity determination unit 14 to determine that it is not necessary to output an alarm. Hereinafter, in the first embodiment, other methods will be described in which the determination control unit 13 controls the alarm necessity determination unit 14 to determine that it is not necessary to output an alarm.

[0051] For example, when the mask wearing determination unit 12 determines that the driver is wearing a mask, the determination control unit 13 may cause the target action determination unit 11 to stop executing the target action determination, thereby controlling the alarm necessity determination unit 14 not to determine that an alarm needs to be output. The configuration example of the determination device 1 in this case is the same as the configuration example of the determination device 1 shown in FIG. 1, and therefore is not shown in the drawings. In this case, the specific operations of the judgment control unit 13, the target operation judgment unit 11, and the alarm necessity judgment unit 14 differ from the specific operations of the judgment control unit 13, the target operation judgment unit 11, and the alarm necessity judgment unit 14 described in the above embodiment 1.

[0052] When the mask wearing determination unit 12 determines that the driver is wearing a mask, in other words, when the mask wearing determination unit 12 outputs a mask determination result that the driver is wearing a mask, the determination control unit 13 outputs an instruction to the target behavior determination unit 11 to stop the execution of the target behavior determination (hereinafter referred to as a "target behavior determination stop instruction"). Also, at this time, the determination control unit 13 outputs information to the warning necessity determination unit 14 that the target behavior determination has been stopped (hereinafter referred to as "cancellation information"). When the mask wearing determination unit 12 determines that the driver is not wearing a mask, the determination control unit 13 outputs an instruction to the target action determination unit 11 to execute a target action determination (hereinafter referred to as a "target action determination execution instruction"). After causing the target action determination unit 11 to execute a target action determination, the determination control unit 13 outputs the action determination result output from the target action determination unit 11 to the warning necessity determination unit 14. In the example configuration of the determination device 1 shown in FIG. 1, the arrow from the determination control unit 13 to the target behavior determination unit 11 is omitted.

[0053] The target action determination unit 11 does not perform the target action determination when an instruction to stop the target action determination is output from the determination control unit 13. Note that since the target action determination unit 11 does not perform the target action determination, the target action determination unit 11 does not output an action determination result. When an instruction to execute target action determination is output from the determination control unit 13, the target action determination unit 11 executes target action determination. The details of the target behavior determination performed by the target behavior determining unit 11 have already been explained, so a duplicated explanation will be omitted.

[0054] The warning necessity determining unit 14 executes a warning necessity determination to determine whether a warning needs to be output. The alarm necessity determination unit 14 determines that an alarm needs to be issued when, in the alarm necessity determination, the target action determination unit 11 outputs an action determination result indicating that the driver is smoking or eating or drinking via the determination control unit 13. The alarm necessity determining unit 14 outputs alarm output control information to the alarm device 3 when it determines that an alarm needs to be output. The alarm necessity determination unit 14 determines that an alarm is unnecessary when the target action determination unit 11 outputs an action determination result via the determination control unit 13 indicating that the driver is not smoking or eating or drinking, or when cancellation information is output from the determination control unit 13.

[0055] Even in this way, the judgment control unit 13 can control the alarm necessity judgment unit 14 so that it does not determine that an alarm needs to be output when the mask wearing judgment unit 12 judges that the driver is wearing a mask.

[0056] FIG. 4 is a flowchart for explaining the operation of the determination device 1 in the first embodiment, in which the determination device 1 stops the execution of the target action determination when it determines that the driver is wearing a mask, thereby controlling the determination so that it is not determined that an alarm needs to be output. The timing at which the operation of the determination device 1 shown in the flowchart of FIG. 4 is executed is the same as the timing at which the operation of the determination device 1 shown in the flowchart of FIG. 1 is executed, and therefore a duplicated description will be omitted. Furthermore, the specific contents of the processes in steps ST11 and ST15 in FIG. 4 are the same as the specific contents of the processes in steps ST2 and ST5 in FIG. 2, respectively, which have already been explained, and therefore will not be explained again.

[0057] If the mask wearing determination unit 12 determines in step ST11 that the driver is wearing a mask, the determination control unit 13 performs processing to control the warning necessity determination unit 14 so that output of a warning is not necessary (step ST12).

[0058] When an instruction to execute target operation determination is output from the determination control unit 13 in step ST12, the target operation determination unit 11 executes target operation determination (step ST13). The target operation determination unit 11 outputs the operation determination result to the determination control unit 13. The determination control unit 13 outputs the operation determination result output from the target operation determination unit 11 to the warning necessity determination unit 14. If the target action determination unit 11 receives a target action determination stop instruction from the determination control unit 13 in step ST12, the target action determination unit 11 does not execute the target action determination.

[0059] The alarm necessity determining unit 14 executes an alarm necessity determination to determine whether an alarm needs to be output (step ST14). The alarm necessity determination unit 14 determines that an alarm needs to be output when, in the alarm necessity determination, the target behavior determination unit 11 outputs a behavior determination result indicating that the driver is smoking or eating or drinking via the determination control unit 13. When determining that an alarm needs to be output, the alarm necessity determination unit 14 outputs alarm output control information to the alarm device 3. The alarm necessity determination unit 14 determines that an alarm is unnecessary when the target action determination unit 11 outputs an action determination result indicating that the driver is not smoking or eating or drinking via the determination control unit 13, or when cancellation information is output from the determination control unit 13.

[0060] 4, the process of step ST15 is performed after the process of step ST14, but this is merely an example. The process of step ST15 may be performed in parallel with the processes of steps ST12 to ST14, or may be performed before the process of step ST12. The process of step ST15 may be performed at an appropriate timing after the process of step ST11.

[0061] FIG. 5 is a flowchart for explaining the detailed operation of step ST12 in FIG. The determination control unit 13 determines whether the mask wearing determination unit 12 has determined in step ST11 that the driver is wearing a mask (step ST311). If the mask wearing determination unit 12 determines in step ST311 that the driver is wearing a mask (if "YES" in step ST311), the determination control unit 13 outputs a target behavior determination stop instruction to the target behavior determination unit 11 (step ST312). At this time, the determination control unit 13 also outputs stop information to the warning necessity determination unit 14. Then, the determination control unit 13 ends the operation shown in the flowchart of FIG. 5.

[0062] In step ST311, if the mask wearing determination unit 12 determines that the driver is not wearing a mask (in the case of "NO" in step ST311), the determination control unit 13 outputs an instruction to execute target behavior determination to the target behavior determination unit 11 (step ST313). Then, the determination control unit 13 ends the operation shown in the flowchart of FIG.

[0063] In this way, when it is determined that the driver is wearing a mask, the determination device 1 may stop the execution of the target behavior determination and thereby determine that a warning is not necessary. This allows the determination device 1 to prevent a false alarm due to an obvious erroneous determination that the driver is smoking or eating when the driver is wearing a mask. That is, the determination device 1 can prevent a false alarm due to an obvious erroneous determination that the driver is smoking or eating in a situation where it is assumed that the driver is clearly not smoking or eating.

[0064] Furthermore, by adopting a method in which the determination device 1 stops the execution of the target action determination and determines that an alarm is unnecessary when it determines that the driver is wearing a mask, it is possible to avoid executing unnecessary target action determinations compared to when a method is adopted in which the action determination result from the target action determination is rejected and a determination that an alarm is unnecessary. However, the determination device 1 can execute various processes (e.g., target action determination process and mask wearing determination process) in parallel without having to make a determination as to whether or not execution is necessary, by adopting a method of rejecting the action determination result from the target action determination and determining that an alarm is unnecessary, rather than adopting a method of stopping the execution of the target action determination and determining that an alarm is unnecessary if it is determined that the driver is wearing a mask, and can perform the processes smoothly without delaying the overall processing flow.

[0065] 6A and 6B are diagrams illustrating an example of a hardware configuration of the determination device 1 according to the first embodiment. In the first embodiment, the functions of the target behavior determination unit 11, the mask wearing determination unit 12, the determination control unit 13, the warning necessity determination unit 14, and the wearing notification control unit 15 are realized by the processing circuit 1001. That is, the determination device 1 is provided with the processing circuit 1001 that determines whether or not the target person is performing the target behavior and whether or not the person is wearing a mask, based on the captured image acquired from the camera 2, and that, if it is determined that the person is wearing a mask, performs control so as not to issue a warning even if it is determined that the person is performing the target behavior. The processing circuit 1001 may be dedicated hardware as shown in FIG. 6A, or may be a processor 1004 that executes a program stored in memory as shown in FIG. 6B.

[0066] When the processing circuit 1001 is dedicated hardware, the processing circuit 1001 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0067] When the processing circuit is the processor 1004, the functions of the target behavior determination unit 11, the mask wearing determination unit 12, the determination control unit 13, the warning necessity determination unit 14, and the wearing notification control unit 15 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 1005. The processor 1004 reads and executes the program stored in the memory 1005 to execute the functions of the target behavior determination unit 11, the mask wearing determination unit 12, the determination control unit 13, the warning necessity determination unit 14, and the wearing notification control unit 15. That is, the determination device 1 includes the memory 1005 for storing a program that, when executed by the processor 1004, results in the execution of steps ST1 to ST5 in FIG. 2 or steps ST11 to ST15 in FIG. 4 described above. It can also be said that the program stored in memory 1005 causes the computer to execute the processing procedures or methods of target behavior determination unit 11, mask wearing determination unit 12, determination control unit 13, warning necessity determination unit 14, and wearing notification control unit 15. Here, memory 1005 corresponds to, for example, non-volatile or volatile semiconductor memory such as RAM, ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), or magnetic disk, flexible disk, optical disk, compact disk, mini disk, DVD (Digital Versatile Disc), etc.

[0068] It is also possible to realize some of the functions of the target action determination unit 11, the mask wearing determination unit 12, the determination control unit 13, the warning necessity determination unit 14, and the wearing notification control unit 15 with dedicated hardware and some with software or firmware. For example, the functions of the warning necessity determination unit 14 and the wearing notification control unit 15 can be realized by the processing circuit 1001 as dedicated hardware, and the functions of the target action determination unit 11, the mask wearing determination unit 12, and the determination control unit 13 can be realized by the processor 1004 reading and executing a program stored in the memory 1005. The determination device 1 also includes an input interface device 1002 and an output interface device 1003 that perform wired or wireless communication with devices such as the camera 2 or the alarm device 3.

[0069] In the above-described first embodiment, the determination device 1 is provided with the attachment notification control unit 15, but this is merely an example, and the determination device 1 does not necessarily have to include the attachment notification control unit 15. If the determination device 1 does not include the attachment notification control unit 15, the process of step ST5 is omitted from the operation of the determination device 1 described using the flowchart in Fig. 2. Also, the process of step ST15 is omitted from the operation of the determination device 1 described using the flowchart in Fig. 4.

[0070] Furthermore, in the above-described first embodiment, the function of outputting the alarm output control information, which was assumed to be possessed by the alarm necessity determination unit 14, may be possessed by another device connected to the determination device 1, outside the determination device 1. In this case, when the alarm necessity determination unit 14 determines that an alarm needs to be output, it outputs a determination result indicating that an alarm needs to be output to the other device.

[0071] In addition, in the above-described first embodiment, the determination device 1 is an in-vehicle device mounted on a vehicle, and the target behavior determination unit 11, the mask wearing determination unit 12, the determination control unit 13, the warning necessity determination unit 14, and the wearing notification control unit 15 are provided in the in-vehicle device. Without being limited to this, for example, some of the target action determination unit 11, mask wearing determination unit 12, determination control unit 13, alarm necessity determination unit 14, and wearing notification control unit 15 may be mounted on the vehicle's on-board device, and the others may be provided on a server connected to the on-board device via a network, and a determination system may be formed by the on-board device and the server. Also, for example, the target behavior determining unit 11, the mask wearing determining unit 12, the determination control unit 13, the warning necessity determining unit 14, and the wearing notification control unit 15 may all be provided in the server.

[0072] Furthermore, in the first embodiment, the subject is the driver of the vehicle, but this is merely an example, and the subject may be a passenger other than the driver of the vehicle. Furthermore, the subject is not limited to a vehicle occupant, but may be, for example, a occupant of a moving body other than a vehicle, or a person present in a room or the like. Furthermore, when the faces of multiple people are captured in the image captured by the camera 2, the determination device 1 determines whether each of the multiple subjects is performing a target action and whether they are wearing a mask, treating the multiple people as targets, and can determine that an alarm needs to be output when the target action is being performed, while controlling so as not to determine that an alarm needs to be output when a mask is being worn.

[0073] As described above, according to the first embodiment, the determination device 1 is configured to include: a target movement determination unit 11 that performs a target movement determination to determine whether a subject, whose face is to be determined as to whether or not the subject is performing a target movement accompanied by mouth movement, is performing a target movement determination to determine whether or not the subject is performing a target movement based on a captured image of the face of the subject; a mask wearing determination unit 12 that performs a mask wearing determination to determine whether or not the subject is wearing a mask based on the captured image; an alarm necessity determination unit 14 that determines that an alarm needs to be output when the target movement determination unit 11 outputs a movement determination result indicating that the subject is performing the target movement; and a determination control unit 13 that controls the alarm necessity determination unit 14 not to determine that an alarm needs to be output when the mask wearing determination unit 12 determines that the subject is wearing a mask. Therefore, when the subject, whose face is to be determined as to whether or not the subject is performing a target movement accompanied by mouth movement, is wearing a mask, the determination device 1 can prevent a false alarm from being issued due to an obvious erroneous determination that the subject is performing the target movement.

[0074] In detail, in the judgment device 1, if the target action judgment unit 11 judges that the subject is performing the target action and the mask wearing judgment unit 12 judges that the subject is wearing a mask, the judgment control unit 13 rejects the action judgment result output from the target action judgment unit 11, and the alarm necessity judgment unit 14 judges that an alarm is unnecessary if the judgment control unit 13 rejects the action judgment result. This allows the determination device 1 to prevent a false alarm due to an obvious erroneous determination that the subject is performing the target action involving mouth movement when the subject is wearing a mask and is being determined to be performing the target action.

[0075] Furthermore, in the determination device 1, when the mask wearing determination unit 12 determines that the subject is wearing a mask, the determination control unit 13 may cause the target action determination unit 11 to stop executing the target action determination, and the alarm necessity determination unit 14 may determine that an alarm is unnecessary when the determination control unit 13 causes the target action determination unit 11 to stop executing the target action determination. This also enables the determination device 1 to prevent a false alarm due to an obvious erroneous determination that the subject is performing a target action involving mouth movement when the subject is wearing a mask and is being determined to be performing the target action.

[0076] In addition, in the present disclosure, any of the components of the embodiments may be modified or omitted. [Industrial Applicability]

[0077] The determination device of the present disclosure can prevent false alarms due to an obvious erroneous determination that the subject is performing a target action involving mouth movement when the subject is wearing a mask and is being determined to be performing a target action. [Explanation of symbols]

[0078] 1 Determination device, 2 Camera, 3 Alarm device, 11 Target operation determination unit, 12 Mask wearing determination unit, 13 Determination control unit, 14 Alarm necessity determination unit, 15 Wearing notification control unit, 1001 Processing circuit, 1002 Input interface device, 1003 Output interface device, 1004 Processor, 1005 Memory.

Claims

1. a target action determination unit that performs target action determination to determine whether a target person is performing a target action involving mouth movement based on a captured image of the face of the target person; a mask wearing determination unit that performs a mask wearing determination to determine whether the subject is wearing a mask based on the captured image; an alarm necessity determination unit that determines that an alarm needs to be output when the target action determination unit outputs a motion determination result indicating that the subject is performing the target action; a determination control unit that controls the alarm necessity determination unit not to determine that output of the alarm is necessary when the mask wearing determination unit determines that the subject is wearing the mask, the determination control unit, when the target action determination unit determines that the subject is performing the target action and the mask wearing determination unit determines that the subject is wearing the mask, rejects the action determination result output from the target action determination unit; The warning necessity determination unit determines that the warning is unnecessary when the determination control unit rejects the operation determination result. A determination device characterized by:

2. a target action determination unit that performs target action determination to determine whether a target person is performing a target action involving mouth movement based on a captured image of the face of the target person; a mask wearing determination unit that performs a mask wearing determination to determine whether the subject is wearing a mask based on the captured image; an alarm necessity determination unit that determines that an alarm needs to be output when the target action determination unit outputs a motion determination result indicating that the subject is performing the target action; a determination control unit that controls the alarm necessity determination unit not to determine that output of the alarm is necessary when the mask wearing determination unit determines that the subject is wearing the mask, the determination control unit causes the target behavior determination unit to stop executing the target behavior determination when the mask wearing determination unit determines that the subject is wearing the mask; The warning necessity determination unit determines that the warning is unnecessary when the determination control unit causes the target operation determination unit to stop executing the target operation determination. A determination device characterized by:

3. a mask wearing notification control unit that outputs, when the mask wearing determination unit determines that the subject is wearing the mask, wearing notification information for notifying the subject that the mask is being worn; 3. The determination device according to claim 1, further comprising:

4. When determining that the output of the alarm is necessary, the alarm necessity determination unit outputs alarm output control information for outputting the alarm.