Determination device and determination method

Abstract

The present invention is provided with: a target motion determination unit (11) that, on the basis of a captured image obtained by capturing the face of a target person who is a target for determining whether or not a target motion involving mouth movement has been performed, executes target motion determination for determining whether or not the target person is performing the target motion; a mask wearing determination unit (12) that, on the basis of the captured image, executes mask wearing determination for determining whether or not the target person is wearing a mask; an alarm necessity determination unit (14) for determining that an output of an alarm is necessary when a motion determination result indicating that the target motion is being performed by the target person is outputted from the target motion determination unit (11); and a determination control unit (13) for performing control so that the alarm necessity determination unit (14) does not determine that the output of the alarm is necessary when the mask wearing determination unit (12) has determined that the target person is wearing a mask.

Description

Determination device and determination method

[0001] The present disclosure relates to a determination device and a determination method for determining whether a person is performing an action involving mouth movement that is subject to an alarm.

[0002] Conventionally, there is known a technology that determines whether a person is performing an action involving mouth movement that is subject to an alarm (hereinafter referred to as a "target action"), such as smoking or eating in an inappropriate place, based on an image of the person's face, and issues an alarm if the person is performing the target action. For example, Patent Literature 1 discloses a technology that outputs alarm information when a distracting action that may distract the driver, such as smoking, drinking, eating, talking on the phone, playing entertainment, or applying makeup, is detected based on an image of the driver of a vehicle. However, there is a case where the person (hereinafter referred to as the "target person") whose performance is to be determined is wearing a mask.

[0003] Special Publication No. 2021-516829

[0004] When determining whether a subject is performing a target action from a captured image, it is technically difficult to eliminate false positives. For example, if a subject is determined to be performing a target action by holding a specific object near their mouth, the size of the object may lead to a false positive determination that the subject is performing the target action even when they are not. Furthermore, for example, depending on the subject's facial orientation, the subject may be falsely determined to be performing the target action even when they are not. If a false positive determination is made that the subject is performing the target action even when they are not, a false alarm is issued based on the false positive determination. Conventional technology has a problem in that if a subject is wearing a mask and is clearly unable to perform the target action, a false alarm, which is technically unavoidable as described above, may be issued, causing inconvenience to the subject. Patent Document 1 discloses a method of determining whether a predetermined distraction action has occurred by determining a motion target frame based on characteristics of a driver image, classifying and detecting predetermined distraction actions based on the motion target frame, and determining whether a predetermined distraction action has occurred. The motion target frame includes local facial regions and motion-related objects, and the motion-related objects include a mask. However, the technology disclosed in Patent Document 1 does not specifically describe how the mask is used to determine whether a predetermined distracting behavior has occurred, and the technology disclosed in Patent Document 1 still does not solve the above problem.

[0005] The present disclosure has been made to solve the above-mentioned problems, and aims to provide a determination device that prevents false alarms due to an obvious erroneous determination that a subject is performing a target action involving mouth movement when the subject is wearing a mask and is to be determined as to whether or not the subject is performing a target action involving mouth movement.

[0006] The determination device according to the present disclosure includes a target movement determination unit that performs a target movement determination to determine whether or not a subject is performing a target movement based on an image of the face of the subject who is to be determined as to whether or not the subject is performing a target movement involving mouth movement; a mask wearing determination unit that performs a mask wearing determination to determine whether or not the subject is wearing a mask based on the image; an alarm necessity determination unit that determines that an alarm needs to be output when the target movement determination unit outputs a movement determination result indicating that the subject is performing the target movement; and a determination control unit that controls the alarm necessity determination unit not to determine that an alarm needs to be output when the mask wearing determination unit determines that the subject is wearing a mask.

[0007] According to the present disclosure, when a subject who is to be judged as to whether or not he or she is performing a target action involving mouth movement is wearing a mask, it is possible to prevent a false alarm due to an obvious erroneous judgment that the subject is performing the target action.

[0008] 6A and 6B are diagrams illustrating an example of the configuration of a determination device according to embodiment 1. FIG. 6B is a flowchart illustrating the operation of the determination device according to embodiment 1. FIG. 6C is a flowchart illustrating the detailed operation of step ST3 in FIG. 2. FIG. 6D is a flowchart illustrating the operation of the determination device in embodiment 1, in which the determination device stops execution of target behavior determination when it determines that the driver is wearing a mask, thereby controlling the device so that it does not determine that output of an alarm is necessary. FIG. 6E is a flowchart illustrating the detailed operation of step ST12 in FIG. 6A. FIGS. 6A and 6B are diagrams illustrating an example of the hardware configuration of a determination device according to embodiment 1.

[0009] Embodiments of the present disclosure will be described in detail below with reference to the drawings. Embodiment 1. FIG. 1 is a diagram illustrating an example configuration of a determination device 1 according to embodiment 1. The determination device 1 according to embodiment 1 determines whether a person (hereinafter referred to as a "subject") is performing a motion involving mouth movement that is subject to an alert (hereinafter referred to as a "target motion") or not. In embodiment 1, the target motion is, for example, a motion involving mouth movement in an inappropriate location, and is an action that requires an alert to be output when the motion is performed. Specifically, in embodiment 1, the target motion is assumed to be smoking or eating or drinking in a vehicle. In the following embodiment 1, as an example, the determination device 1 is mounted on a vehicle (not shown) and determines whether the driver of the vehicle is performing the target motion, specifically, smoking or eating or drinking. That is, in the following embodiment 1, the subject is the driver of the vehicle. The driver of the vehicle may be wearing a mask. Note that in embodiment 1, a "mask" refers to a protective device made of nonwoven fabric, gauze, or the like that covers the nose or mouth.

[0010] The determination device 1 is connected to a camera 2 and a warning device 3 mounted on a vehicle. For example, the determination device 1, the camera 2, and the warning device 3 may constitute a so-called DMS (Driver Monitoring System). The determination device 1 determines whether or not the driver is performing a target action based on an image of the driver's face captured by the camera 2 (hereinafter referred to as the "captured image"). If the determination device 1 determines that the driver is performing the target action, it outputs information to the warning device 3 to cause the warning device 3 to output a warning.

[0011] The camera 2 is installed, for example, in the center of the vehicle's instrument panel, meter panel, steering column, A-pillar, or around the center console. The camera 2 is installed so as to be able to capture at least the face of the occupant. The camera 2 is a visible light camera or an infrared camera. Note that in FIG. 1 , only one camera 2 is installed in the vehicle, but this is merely an example. A plurality of cameras 2 may be installed in the vehicle. The camera 2 outputs the captured image to the determination device 1.

[0012] The warning device 3 is, for example, an audio output device installed in a vehicle. The warning device 3 outputs, for example, an audible or audio warning. The warning device 3 is installed, for example, in the center of the vehicle's instrument panel, a meter panel, a steering column, an A-pillar, or near the center console. The warning device 3 may be mounted on, for example, a navigation device (not shown). Note that the warning device 3 is not limited to an audio output device, and may be, for example, a display device that displays a warning message or a warning icon, or a vibration generating device that generates vibrations. Furthermore, in FIG. 1 , only one warning device 3 is connected to the determination device 1, but this is merely an example. Multiple warning devices 3 may be connected to the determination device 1.

[0013] The following describes details of the determination device 1 according to embodiment 1. The determination device 1 includes a target behavior determination unit 11, a mask wearing determination unit 12, a determination control unit 13, a warning necessity determination unit 14, and a wearing notification control unit 15.

[0014] The target action determination unit 11 acquires an image of the face of the vehicle driver from the camera 2, and performs action determination based on the acquired image to determine whether the driver is smoking or eating or drinking.

[0015] The target action determination unit 11 determines whether the driver is smoking or eating or drinking, for example, based on an image acquired from the camera 2 and a trained model (hereinafter referred to as a "machine learning model"). In the following description, the machine learning model used by the target action determination unit 11 when determining whether the driver is smoking or eating or drinking is also referred to as a "first machine learning model." The first machine learning model has been trained to receive an image of a person's face as input and output information indicating whether the person is performing a target action (hereinafter referred to as "target action information"). The target action information includes, for example, information indicating whether the person is smoking and information indicating whether the person is eating or drinking. Note that this is merely an example, and the target action information may also include, for example, information indicating whether an action other than smoking or eating or drinking is being performed. The target action information only needs to include information indicating at least whether the person is smoking and whether the person is eating or drinking. In other words, the first machine learning model only needs to be a model that has been trained to receive an image of a person's face as input and output target action information including information indicating whether the person is smoking or eating or drinking.

[0016] When inputting an image into the first machine learning model, the target action determination unit 11 determines that the driver is smoking if target action information indicating smoking is obtained. Furthermore, when inputting an image into the first machine learning model, the target action determination unit 11 determines that the driver is eating or drinking if target action information indicating eating or drinking is obtained. On the other hand, when inputting an image into the first machine learning model, the target action determination unit 11 determines that the driver is not smoking if target action information indicating not smoking is obtained. Furthermore, when inputting an image into the first machine learning model, the target action determination unit 11 determines that the driver is not eating or drinking.

[0017] The above-described method is merely an example, and the target action determination unit 11 may determine whether the driver is smoking or eating or drinking using other methods based on the captured image. For example, the target action determination unit 11 may acquire a heat source image from a thermal source camera (not shown) mounted on the vehicle, and determine whether the driver is smoking based on whether a heat source is detected around the driver's mouth based on the heat source image and the captured image acquired from camera 2. The thermal source camera generates a heat source image based on the amount of infrared rays emitted from an object, a so-called thermography. The target action determination unit 11 may perform a known image recognition process on the captured image and the heat source image to identify which area on the heat source image corresponds to the area around the driver's mouth. Note that the installation position and angle of view of camera 2 and the installation position and angle of view of the heat source camera are known in advance, so the target action determination unit 11 can associate areas on the captured image with areas on the heat source image. Furthermore, for example, the target action determination unit 11 may determine whether the driver is eating or drinking by performing pattern matching between the captured image obtained from the camera 2 and an image of a beverage container such as a plastic bottle or tableware that has been prepared in advance.

[0018] When the target action determination unit 11 performs a target action determination as to whether or not the driver is smoking or eating or drinking, it outputs the result (hereinafter referred to as the "action determination result") to the determination control unit 13. The action determination result includes information indicating whether or not the driver is smoking and information indicating whether or not the driver is eating or drinking.

[0019] The mask wearing determination unit 12 acquires an image of the face of the driver of the vehicle from the camera 2, and performs a mask wearing determination to determine whether the driver is wearing a mask based on the acquired image.

[0020] The mask wearing determiner 12 determines whether the driver is wearing a mask, for example, based on a captured image acquired from the camera 2 and a machine learning model. In the following description, the machine learning model used by the mask wearing determiner 12 to determine whether the driver is wearing a mask is also referred to as a "second machine learning model." The second machine learning model has been trained to receive an image of a person's face as input and output information indicating whether the person is wearing a mask (hereinafter referred to as "mask wearing information"). The mask wearing information includes, for example, information indicating whether the person is wearing a mask. Note that this is merely an example, and the mask wearing information may include, for example, information indicating the color, pattern, or type of mask (nonwoven fabric, gauze, etc.) in addition to the information indicating whether the person is wearing a mask. The mask wearing information is sufficient as long as it includes at least information indicating whether the person is wearing a mask. In other words, the second machine learning model is sufficient as long as it receives an image of a person's face as input and outputs mask wearing information including information indicating whether the person is wearing a mask.

[0021] When the captured image is input to the second machine learning model and mask wearing information indicating that a mask is being worn is obtained, the mask wearing determination unit 12 determines that the driver is wearing a mask. On the other hand, when the captured image is input to the second machine learning model and mask wearing information indicating that a mask is not being worn is obtained, the mask wearing determination unit 12 determines that the driver is not wearing a mask.

[0022] The above-described method is merely an example, and the mask wearing determiner 12 may determine whether the driver is wearing a mask using other methods based on the captured image. For example, the mask wearing determiner 12 may determine whether the driver is wearing a mask based on whether or not a nose feature point can be extracted based on the captured image. If the nose feature point cannot be extracted, the mask wearing determiner 12 determines that the driver is wearing a mask. The mask wearing determiner 12 may extract feature points that represent parts of the driver's face (specifically, the nose) in the captured image using a known technique such as a method using Haar-Like features or pattern matching.

[0023] When the mask wearing determiner 12 determines whether the driver is wearing a mask or not, it outputs the result (hereinafter referred to as the "mask wearing determination result") to the determination controller 13. The mask wearing determination result includes information indicating whether the driver is wearing a mask or not.

[0024] Furthermore, if the mask wearing determination unit 12 determines that the driver is wearing a mask as a result of performing the mask wearing determination, it outputs information indicating that the driver is wearing a mask (hereinafter referred to as "mask present information") to the wearing notification control unit 15.

[0025] When the mask wearing determination unit 12 determines that the driver is wearing a mask, the determination control unit 13 controls the warning necessity determination unit 14 so that output of a warning is not required.

[0026] Here, the alarm necessity determination unit 14 will be described. The alarm necessity determination unit 14 executes an alarm necessity determination to determine whether an alarm needs to be output. When the target behavior determination unit 11 outputs a behavior determination result indicating that the driver is smoking or eating or drinking in the alarm necessity determination, the alarm necessity determination unit 14 determines that an alarm needs to be output. The alarm necessity determination unit 14 acquires the behavior determination result output from the target behavior determination unit 11 via the determination control unit 13. When the alarm necessity determination unit 14 determines that an alarm needs to be output, it outputs information for causing the alarm device 3 to output an alarm (hereinafter referred to as "alarm output control information"). For example, if the alarm device 3 is an audio output device, the alarm output control information is information for causing the audio output device to output an alarm sound or a warning message warning against smoking or drinking. For example, if the alarm device 3 is a display device, the alarm output control information is information for causing the display device to display a warning message warning against smoking or eating or drinking. For example, if the warning device 3 is a vibration generator, the warning output control information is information for causing the vibration generator to generate vibrations to warn the driver of smoking or eating or drinking. The warning necessity determination unit 14 may change the type of warning sound, the content of the warning message, or the type of vibration depending on whether the driver is smoking or eating or drinking.

[0027] In addition, the warning necessity determination unit 14 determines that a warning is unnecessary when the target behavior determination unit 11 outputs a behavior determination result via the determination control unit 13 indicating that the driver is not smoking or eating or drinking, or when the determination control unit 13 outputs information indicating that the behavior determination result has been rejected (hereinafter referred to as "rejection information").

[0028] Returning to the description of the determination control unit 13, when the mask wearing determination unit 12 determines that the driver is wearing a mask, the determination control unit 13 controls the warning necessity determination unit 14 to determine that it is not necessary to output a warning, as described above, in the warning necessity determination performed by the warning necessity determination unit 14.

[0029] An example of a method in which the determination control unit 13 controls the warning necessity determination unit 14 to determine that output of a warning is not necessary will be described. The determination control unit 13 first determines whether the target behavior determination unit 11 has determined that the driver is smoking or eating or drinking, and whether the mask wearing determination unit 12 has determined that the driver is wearing a mask. The determination control unit 13 can determine whether the driver has been determined to be smoking or eating or drinking from the behavior determination result output from the target behavior determination unit 11. Furthermore, the determination control unit 13 can determine whether the driver has been determined to be wearing a mask from the mask determination result output from the mask wearing determination unit 12.

[0030] When the target action determination unit 11 determines that the driver is smoking or eating and the mask wearing determination unit 12 determines that the driver is wearing a mask, the determination control unit 13 rejects the action determination result output from the target action determination unit 11. That is, the determination control unit 13 prevents the action determination result output from the target action determination unit 11, more specifically, the action determination result that the driver is smoking or eating and drinking, from being output to the alarm necessity determination unit 14. At this time, the determination control unit 13 outputs the rejection information to the alarm necessity determination unit 14 instead of outputting the action determination result output from the target action determination unit 11. The action determination result that the driver is smoking or eating and drinking, output by the target action determination unit 11, is no longer output to the alarm necessity determination unit 14, and rejection information is output instead, so that the alarm necessity determination unit 14 determines, as a result of performing the alarm necessity determination, that an alarm is unnecessary.

[0031] Note that the above example is merely one example, and for example, the determination control unit 13 may rewrite the action determination result output from the target action determination unit 11, which indicates that the driver is smoking or eating or drinking, into an action determination result indicating that the driver is not smoking or eating or drinking, and output it to the alarm necessity determination unit 14. In this case as well, since the action determination result indicating that the driver is smoking or eating or drinking is no longer output, the alarm necessity determination unit 14 will determine that an alarm is not necessary.

[0032] If the target action determination unit 11 does not determine that the driver is smoking or eating or drinking, or if the mask wearing determination unit 12 determines that the driver is not wearing a mask, the determination control unit 13 outputs the action determination result output from the target action determination unit 11 as is to the alarm necessity determination unit 14.

[0033] When the mask wearing determiner 12 determines that the subject is wearing a mask, in other words, when mask presence information is output from the mask wearing determiner 12, the wearing notification control unit 15 outputs information for notifying that a mask is being worn (hereinafter referred to as "wearing notification information") to the warning device 3. Here, when the mask wearing determiner 12 determines that the driver is wearing a mask, the wearing notification control unit 15 outputs the wearing notification information to the warning device 3.

[0034] For example, if the warning device 3 is an audio output device, the wearing notification information is information for causing the audio output device to output a notification sound or a message notifying the driver that the driver is wearing a mask. For example, if the warning device 3 is a display device, the wearing notification information is information for causing the display device to display a message or icon notifying the driver that the driver is wearing a mask. For example, if the warning device 3 is a vibration generator, the wearing notification information is information for causing the vibration generator to generate a vibration notifying the driver that the driver is wearing a mask. Note that, for example, the type of warning sound or vibration that warns the driver that the driver is smoking or eating, as described above, and the type of notification sound or vibration that notifies the driver that the driver is wearing a mask may be different types of sound or vibration.

[0035] In addition, in the first embodiment, the wearing notification control unit 15 outputs the wearing notification information to the warning device 3, but this is merely an example. The wearing notification control unit 15 may output the wearing notification information to, for example, an output device (not shown) other than the warning device 3.

[0036] The operation of the determination device 1 according to embodiment 1 will now be described. Fig. 2 is a flowchart for explaining the operation of the determination device 1 according to embodiment 1. For example, when the power of the vehicle is turned on and the power of the determination device 1 is turned on, the determination device 1 starts the operation shown in the flowchart of Fig. 2, and repeats the operation shown in the flowchart of Fig. 2 until the power of the determination device 1 is turned off.

[0037] The target action determination unit 11 acquires an image of the face of the vehicle driver from the camera 2, and performs action determination based on the acquired image to determine whether the driver is smoking or eating or drinking (step ST1). The target action determination unit 11 outputs the action determination result to the determination control unit 13.

[0038] The mask wearing determiner 12 acquires an image of the face of the vehicle driver from the camera 2, and executes a mask wearing determination to determine whether the driver is wearing a mask based on the acquired image (step ST2). The mask wearing determiner 12 outputs the mask determination result to the determination control unit 13. Furthermore, if the mask wearing determiner 12 determines that the driver is wearing a mask as a result of executing the mask wearing determination, the mask wearing determiner 12 outputs mask presence information to the wearing notification control unit 15.

[0039] The determination control unit 13 outputs the action determination result output from the target action determination unit 11 in step ST1 to the warning necessity determination unit 14. At this time, if the mask wearing determination unit 12 determines in step ST2 that the driver is wearing a mask, the determination control unit 13 performs processing to control the warning necessity determination unit 14 so that output of a warning is not necessary (step ST3).

[0040] The alarm necessity determination unit 14 executes an alarm necessity determination to determine whether an alarm needs to be output (step ST4). When the target behavior determination unit 11 outputs a behavior determination result indicating that the driver is smoking or eating or drinking via the determination control unit 13 in the alarm necessity determination, the alarm necessity determination unit 14 determines that an alarm needs to be output. When the target behavior determination unit 11 outputs a behavior determination result indicating that the driver is not smoking or eating or drinking via the determination control unit 13, the alarm necessity determination unit 14 determines that an alarm is not needed.

[0041] If the mask wearing determination unit 12 determines in step ST2 that the driver is wearing a mask, in other words, if the mask wearing determination unit 12 outputs mask presence information, the wearing notification control unit 15 outputs wearing notification information to the warning device 3 (step ST4).

[0042] In the flowchart of FIG. 2, the processing of step ST2 is performed after the processing of step ST1, but this is merely an example. The processing of step ST1 and the processing of step ST2 may be performed in parallel, or the processing of step ST1 may be performed after the processing of step ST2. It is sufficient that the processing of step ST1 and the processing of step ST2 are performed before the processing of step ST3 is performed. Furthermore, in the flowchart of FIG. 2, the processing of step ST5 is performed after the processing of step ST4, but this is merely an example. The processing of step ST5 may be performed in parallel with the processing of steps ST3 and ST4, or may be performed before the processing of step ST3. It is sufficient that the processing of step ST5 is performed at an appropriate timing after the processing of step ST2.

[0043] Fig. 3 is a flowchart for explaining the detailed operation of step ST3 in Fig. 2. The determination control unit 13 determines whether the target operation determination unit 11 has determined in step ST1 that the driver is smoking or eating or drinking (step ST301). If the target operation determination unit 11 has determined in step ST301 that the driver is smoking or eating or drinking ("YES" in step ST301), the determination control unit 13 determines whether the mask wearing determination unit 12 has determined in step ST2 that the driver is wearing a mask (step ST302).

[0044] If, in step ST302, the mask wearing determination unit 12 determines that the driver is wearing a mask ("YES" in step ST302), the determination control unit 13 rejects the movement determination result output from the target movement determination unit 11. At this time, the determination control unit 13 outputs rejection information to the warning necessity determination unit 14, instead of outputting the movement determination result output from the target movement determination unit 11. The determination control unit 13 may rewrite the movement determination result output from the target movement determination unit 11, which indicates that the driver is smoking or eating, into a movement determination result indicating that the driver is not smoking or eating, and output this to the warning necessity determination unit 14. Then, the determination control unit 13 ends the operation shown in the flowchart of FIG. 3.

[0045] On the other hand, if the target operation determination unit 11 determines in step ST301 that the driver is not smoking or eating or drinking (if "NO" in step ST301), or if the mask wearing determination unit 12 determines in step ST302 that the driver is not wearing a mask (if "NO" in step ST302), the determination control unit 13 outputs the operation determination result output from the target operation determination unit 11 as is to the alarm necessity determination unit 14, and terminates the operation shown in the flowchart of Figure 3.

[0046] In this way, the determination device 1 performs a target action determination to determine whether the driver is smoking or eating or drinking, and a mask wearing determination to determine whether the driver is wearing a mask, based on the captured image. The determination device 1 performs an alarm necessity determination to determine whether an alarm needs to be output. If the determination device 1 determines that the driver is eating, drinking, or smoking, it determines that an alarm needs to be output. However, if the determination device 1 determines that the driver is wearing a mask, it does not determine that an alarm needs to be output. In particular, if the determination device 1 determines that the driver is smoking or eating and drinking and also determines that the driver is wearing a mask, it rejects the action determination result that the driver is smoking or eating and determines that an alarm is unnecessary. This allows the determination device 1 to prevent false alarms due to an obvious erroneous determination that the driver is smoking or eating when the driver is wearing a mask.

[0047] As described above, it is technically difficult to eliminate erroneous determinations when determining whether a subject is performing a target action from a captured image. For example, when determining whether a driver is smoking or eating or drinking based on a captured image, depending on the relative positions of the driver and camera 2, such as when camera 2 is installed on the center console, or the driver's facial orientation, interior objects or the scenery outside the vehicle may be reflected next to the driver's face in the captured image. Because whether a driver is smoking or eating or drinking is determined based on a two-dimensional captured image, reflected background objects such as interior objects or scenery may be erroneously determined to be cigarettes even if they are actually far away. Furthermore, for example, the machine learning model (first machine learning model) may not have learned about unexpected cases where the background objects or lighting conditions are unusual. Furthermore, because cigarettes are small objects, the driver's hands are also part of the determination factor when determining smoking behavior. Therefore, for example, if a driver brings their hand close to their mouth without holding a cigarette, they may be erroneously determined to be smoking. For example, if a driver touches their mouth over a mask, they may be erroneously determined to be smoking. As described above, when a background object is reflected in the captured image or when the machine learning model is undertrained, adverse conditions overlap, increasing the likelihood of a false positive. Such a false positive may occur due to a misidentification of a background object reflected in the captured image as a cigarette, an undertrained machine learning model, or the driver touching their mouth without holding a cigarette, resulting in a false positive determination that the driver is smoking or eating or drinking even when they are not. As a result, a false alarm may be issued indicating that the driver is smoking or eating or drinking even when they are not. In particular, if the driver is wearing a mask, it is unlikely that the driver is smoking or eating or drinking. If a false alarm such as the one described above is issued in a situation where it is clearly assumed that smoking or eating or drinking is not occurring, it may be annoyed for the driver.

[0048] In contrast, as described above, when the determination device 1 according to the first embodiment determines that the driver is smoking or eating and that the driver is wearing a mask, it rejects the action determination result that the driver is smoking or eating and determines that an alarm is unnecessary. This allows the determination device 1 to prevent a false alarm due to an obvious erroneous determination that the driver is smoking or eating when the driver is wearing a mask, in other words, in a situation where it is assumed that the driver is clearly not smoking or eating.

[0049] Furthermore, if the determination device 1 determines that the driver is wearing a mask, it outputs mask-wearing notification information to notify the driver that the mask is being worn. This allows the determination device 1 to notify the driver that it has determined that the driver is wearing a mask. Drivers can wear masks of various types, colors, patterns, and shapes. There are technical limitations to accurately determining that all of these are masks. For example, even if the driver is wearing a mask, the determination device 1 may not be able to determine that the driver is wearing a mask because the mask has a different shape or pattern than a typical mask. In such a case, if the determination device 1 erroneously determines that the driver is smoking or eating or drinking, as described above, a false alarm may be issued. However, in this case, the determination device 1 does not notify the driver that the mask is being worn. Even if a false alarm is issued, the lack of notification that the driver is wearing a mask allows the driver to understand that the false alarm was issued because the driver was not determined to be wearing a mask.

[0050] In the above-described first embodiment, in the determination device 1, when the target behavior determination unit 11 determines that the driver is smoking or eating and the mask wearing determination unit 12 determines that the driver is wearing a mask, the determination control unit 13 rejects the behavior determination result output from the target behavior determination unit 11, thereby controlling the alarm necessity determination unit 14 to determine that output of an alarm is unnecessary. However, this is merely an example, and the determination control unit 13 may control the alarm necessity determination unit 14 to determine that output of an alarm is unnecessary in other ways. Below, other methods in which the determination control unit 13 controls the alarm necessity determination unit 14 to determine that output of an alarm is unnecessary in the first embodiment will be described.

[0051] For example, when the mask wearing determination unit 12 determines that the driver is wearing a mask, the determination control unit 13 may cause the target behavior determination unit 11 to stop executing the target behavior determination, thereby controlling the warning necessity determination unit 14 not to determine that output of a warning is necessary. An example configuration of the determination device 1 in this case is similar to the example configuration of the determination device 1 shown in Fig. 1 , and therefore is not shown in the figure. In this case, the specific operations of the determination control unit 13, the target behavior determination unit 11, and the warning necessity determination unit 14 differ from the specific operations of the determination control unit 13, the target behavior determination unit 11, and the warning necessity determination unit 14 described in the first embodiment above.

[0052] When the mask wearing determination unit 12 determines that the driver is wearing a mask, in other words, when the mask wearing determination unit 12 outputs a mask determination result indicating that the driver is wearing a mask, the determination control unit 13 outputs an instruction to the target behavior determination unit 11 to stop executing the target behavior determination (hereinafter referred to as a "target behavior determination stop instruction"). At this time, the determination control unit 13 also outputs information to the warning necessity determination unit 14 that the target behavior determination has been stopped (hereinafter referred to as "cancellation information"). When the mask wearing determination unit 12 determines that the driver is not wearing a mask, the determination control unit 13 outputs an instruction to the target behavior determination unit 11 to execute the target behavior determination (hereinafter referred to as a "target behavior determination execution instruction"). After causing the target behavior determination unit 11 to execute the target behavior determination, the determination control unit 13 outputs the behavior determination result output from the target behavior determination unit 11 to the warning necessity determination unit 14. In the example configuration of the determination device 1 shown in FIG. 1, the arrow from the determination control unit 13 to the target operation determination unit 11 is omitted.

[0053] The target action determination unit 11 does not perform the target action determination when an instruction to stop the target action determination is output from the determination control unit 13. Note that since the target action determination unit 11 does not perform the target action determination, no action determination result is output from the target action determination unit 11. The target action determination unit 11 performs the target action determination when an instruction to perform the target action determination is output from the determination control unit 13. Details of the target action determination performed by the target action determination unit 11 have already been explained, so repeated explanation will be omitted.

[0054] The alarm necessity determination unit 14 executes an alarm necessity determination to determine whether an alarm needs to be output. When the target behavior determination unit 11 outputs a behavior determination result indicating that the driver is smoking or eating or drinking via the determination control unit 13 in the alarm necessity determination, the alarm necessity determination unit 14 determines that an alarm needs to be output. When the alarm necessity determination unit 14 determines that an alarm needs to be output, it outputs alarm output control information to the alarm device 3. When the target behavior determination unit 11 outputs a behavior determination result indicating that the driver is not smoking or eating or drinking via the determination control unit 13, or when cancellation information is output from the determination control unit 13, the alarm necessity determination unit 14 determines that an alarm is not necessary.

[0055] Even in this way, the judgment control unit 13 can control the alarm necessity judgment unit 14 so that it does not determine that an alarm needs to be output when the mask wearing judgment unit 12 judges that the driver is wearing a mask.

[0056] 4 is a flowchart for explaining the operation of the determination device 1 in the first embodiment, in which the determination device 1 stops execution of the target behavior determination when it determines that the driver is wearing a mask, thereby controlling so as not to determine that output of an alarm is necessary. The timing at which the operation of the determination device 1 shown in the flowchart of FIG. 4 is executed is the same as the timing at which the operation of the determination device 1 shown in the flowchart of FIG. 1 is executed, and therefore a duplicated explanation will be omitted. Furthermore, the specific contents of the processes of steps ST11 and ST15 in FIG. 4 are the same as the specific contents of the processes of steps ST2 and ST5 in FIG. 2, respectively, which have already been explained, and therefore a duplicated explanation will be omitted.

[0057] If the mask wearing determination unit 12 determines in step ST11 that the driver is wearing a mask, the determination control unit 13 performs processing to control the warning necessity determination unit 14 so that output of a warning is not necessary (step ST12).

[0058] When an instruction to execute target operation determination is output from the determination control unit 13 in step ST12, the target operation determination unit 11 executes target operation determination (step ST13). The target operation determination unit 11 outputs the operation determination result to the determination control unit 13. The determination control unit 13 outputs the operation determination result output from the target operation determination unit 11 to the warning necessity determination unit 14. When an instruction to cancel target operation determination is output from the determination control unit 13 in step ST12, the target operation determination unit 11 does not execute target operation determination.

[0059] The alarm necessity determination unit 14 executes an alarm necessity determination to determine whether an alarm needs to be output (step ST14). When the target behavior determination unit 11 outputs a behavior determination result indicating that the driver is smoking or eating or drinking via the determination control unit 13 in the alarm necessity determination, the alarm necessity determination unit 14 determines that an alarm needs to be output. When the target behavior determination unit 11 outputs a behavior determination result indicating that the driver is not smoking or eating or drinking via the determination control unit 13, or when cancellation information is output from the determination control unit 13, the alarm necessity determination unit 14 determines that an alarm is not needed.

[0060] 4, the process of step ST15 is performed after the process of step ST14, but this is merely an example. The process of step ST15 may be performed in parallel with the processes of steps ST12 to ST14, or may be performed before the process of step ST12. The process of step ST15 may be performed at an appropriate timing after the process of step ST11.

[0061] Fig. 5 is a flowchart for explaining the detailed operation of step ST12 in Fig. 4. The determination control unit 13 determines whether the mask wearing determination unit 12 has determined in step ST11 that the driver is wearing a mask (step ST311). If the mask wearing determination unit 12 has determined in step ST311 that the driver is wearing a mask ("YES" in step ST311), the determination control unit 13 outputs a target operation determination stop instruction to the target operation determination unit 11 (step ST312). At this time, the determination control unit 13 also outputs stop information to the warning necessity determination unit 14. Then, the determination control unit 13 ends the operation shown in the flowchart in Fig. 5.

[0062] If the mask wearing determination unit 12 determines in step ST311 that the driver is not wearing a mask (NO in step ST311), the determination control unit 13 outputs an instruction to execute target behavior determination to the target behavior determination unit 11 (step ST313). Then, the determination control unit 13 ends the operation shown in the flowchart of FIG. 5.

[0063] In this way, when the determination device 1 determines that the driver is wearing a mask, it may determine that an alarm is unnecessary by halting the execution of the target behavior determination. This allows the determination device 1 to prevent a false alarm due to an obvious erroneous determination that the driver is smoking or eating when the driver is wearing a mask. In other words, the determination device 1 can prevent a false alarm due to an obvious erroneous determination that the driver is smoking or eating in a situation where it is assumed that the driver is clearly not smoking or eating.

[0064] Furthermore, by adopting a method of halting execution of the target action determination and determining that an alarm is unnecessary when it is determined that the driver is wearing a mask, the determination device 1 can avoid executing unnecessary target action determinations compared to adopting a method of rejecting the action determination result of the target action determination and determining that an alarm is unnecessary. However, by adopting a method of rejecting the action determination result of the target action determination and determining that an alarm is unnecessary when it is determined that the driver is wearing a mask, the determination device 1 can execute various processes (e.g., the target action determination process and the mask wearing determination process) in parallel without inserting a determination as to whether or not execution is necessary, and can perform the process smoothly without delaying the overall processing flow, compared to adopting a method of canceling the action determination result of the target action determination and determining that an alarm is unnecessary when it is determined that the driver is wearing a mask.

[0065] 6A and 6B are diagrams illustrating an example of the hardware configuration of the determination device 1 according to the first embodiment. In the first embodiment, the functions of the target behavior determination unit 11, the mask wearing determination unit 12, the determination control unit 13, the warning necessity determination unit 14, and the mask wearing notification control unit 15 are realized by a processing circuit 1001. That is, the determination device 1 includes the processing circuit 1001 for determining whether the subject is performing the target behavior and whether the subject is wearing a mask based on the captured image acquired from the camera 2, and for controlling the device 1 so that no warning is issued even if it is determined that the subject is performing the target behavior, if it is determined that the subject is wearing a mask. The processing circuit 1001 may be dedicated hardware as shown in FIG. 6A, or a processor 1004 that executes a program stored in memory as shown in FIG. 6B.

[0066] If the processing circuit 1001 is dedicated hardware, the processing circuit 1001 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0067] When the processing circuit is the processor 1004, the functions of the target behavior determination unit 11, mask wearing determination unit 12, determination control unit 13, warning necessity determination unit 14, and wearing notification control unit 15 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 1005. The processor 1004 reads and executes the program stored in the memory 1005 to perform the functions of the target behavior determination unit 11, mask wearing determination unit 12, determination control unit 13, warning necessity determination unit 14, and wearing notification control unit 15. In other words, the determination device 1 includes the memory 1005 for storing a program that, when executed by the processor 1004, results in the execution of steps ST1 to ST5 in FIG. 2 or steps ST11 to ST15 in FIG. 4 described above. In addition, the program stored in memory 1005 can also be said to cause the computer to execute the processing procedures or methods of the target action determination unit 11, the mask wearing determination unit 12, the determination control unit 13, the alarm necessity determination unit 14, and the wearing notification control unit 15. Here, the memory 1005 may be, for example, a non-volatile or volatile semiconductor memory such as a RAM, a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically Erasable Programmable Read-Only Memory), or a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, or a DVD (Digital Versatile Disc).

[0068] It is also possible to realize some of the functions of the target behavior determination unit 11, mask wearing determination unit 12, determination control unit 13, alarm necessity determination unit 14, and mask wearing notification control unit 15 with dedicated hardware and some with software or firmware. For example, the functions of the alarm necessity determination unit 14 and mask wearing notification control unit 15 can be realized by a processing circuit 1001 as dedicated hardware, and the functions of the target behavior determination unit 11, mask wearing determination unit 12, and determination control unit 13 can be realized by a processor 1004 reading and executing a program stored in a memory 1005. The determination device 1 also includes an input interface device 1002 and an output interface device 1003 that perform wired or wireless communication with devices such as the camera 2 or the alarm device 3.

[0069] In the above-described first embodiment, the determination device 1 is provided with the attachment notification control unit 15, but this is merely an example, and the determination device 1 is not required to include the attachment notification control unit 15. If the determination device 1 is configured not to include the attachment notification control unit 15, the processing of step ST5 is omitted from the operation of the determination device 1 described using the flowchart in Fig. 2. Furthermore, the processing of step ST15 is omitted from the operation of the determination device 1 described using the flowchart in Fig. 4.

[0070] Furthermore, in the above-described first embodiment, the function of outputting the alarm output control information, which is assumed to be possessed by the alarm necessity determination unit 14, may be possessed by another device connected to the determination device 1, outside the determination device 1. In this case, when the alarm necessity determination unit 14 determines that an alarm needs to be output, it outputs a determination result indicating that an alarm needs to be output to the other device.

[0071] In the first embodiment described above, the determination device 1 is an in-vehicle device mounted on a vehicle, and the target action determination unit 11, mask wearing determination unit 12, determination control unit 13, alarm necessity determination unit 14, and wearing notification control unit 15 are provided in the in-vehicle device. However, this is not limiting. For example, some of the target action determination unit 11, mask wearing determination unit 12, determination control unit 13, alarm necessity determination unit 14, and wearing notification control unit 15 may be mounted in the in-vehicle device of the vehicle, and the others may be provided in a server connected to the in-vehicle device via a network, thereby constituting a determination system with the in-vehicle device and the server. Furthermore, for example, the target action determination unit 11, mask wearing determination unit 12, determination control unit 13, alarm necessity determination unit 14, and wearing notification control unit 15 may all be provided in the server.

[0072] Furthermore, in the above-described first embodiment, the subject is the driver of the vehicle, but this is merely an example. The subject may be a vehicle occupant other than the driver of the vehicle. Furthermore, the subject is not limited to a vehicle occupant. For example, the subject may be an occupant of a moving body other than a vehicle, or a person present in a room, etc. Furthermore, when the faces of multiple people are captured in the captured image captured by the camera 2, the determination device 1 determines whether each of the multiple subjects is performing a target action and whether they are wearing a mask, with the multiple people as subjects, and can perform control so as not to determine that an alarm needs to be output if the subject is wearing a mask, while determining that an alarm needs to be output if the subject is performing a target action.

[0073] As described above, according to the first embodiment, the determination device 1 is configured to include a target movement determination unit 11 that performs a target movement determination to determine whether the subject is performing a target movement based on an captured image of the face of the subject, the target movement determination unit determining whether the subject is performing a target movement involving mouth movement; a mask wearing determination unit 12 that performs a mask wearing determination to determine whether the subject is wearing a mask based on the captured image; an alarm necessity determination unit 14 that determines that an alarm needs to be output when the target movement determination unit 11 outputs a movement determination result indicating that the subject is performing the target movement; and a determination control unit 13 that controls the alarm necessity determination unit 14 not to determine that an alarm needs to be output when the mask wearing determination unit 12 determines that the subject is wearing a mask. Therefore, when the subject, the target movement determination unit for determining whether the subject is performing a target movement involving mouth movement, is wearing a mask, the determination device 1 can prevent a false alarm due to an obvious erroneous determination that the subject is performing the target movement.

[0074] In detail, in the determination device 1, when the target action determination unit 11 determines that the subject is performing the target action and the mask wearing determination unit 12 determines that the subject is wearing a mask, the determination control unit 13 rejects the action determination result output from the target action determination unit 11, and the alarm necessity determination unit 14 determines that an alarm is unnecessary if the determination control unit 13 rejects the action determination result. In this way, when the subject, whose presence or absence of a target action involving mouth movement is to be determined, is wearing a mask, the determination device 1 can prevent a false alarm from being issued due to an obvious erroneous determination that the subject is performing the target action.

[0075] Furthermore, in the determination device 1, the determination control unit 13 may cause the target action determination unit 11 to stop executing the target action determination when the mask wearing determination unit 12 determines that the subject is wearing a mask, and the alarm necessity determination unit 14 may determine that an alarm is unnecessary when the determination control unit 13 causes the target action determination unit 11 to stop executing the target action determination. This also allows the determination device 1 to prevent a false alarm due to an obvious erroneous determination that the subject is performing the target action when the subject, whose presence or absence of a target action involving mouth movement is to be determined, is wearing a mask.

[0076] In addition, in the present disclosure, any of the components of the embodiments may be modified or omitted.

[0077] The determination device of the present disclosure can prevent false alarms due to an obvious erroneous determination that the subject is performing a target action involving mouth movement when the subject is wearing a mask and is being determined to be performing a target action.

[0078] REFERENCE SIGNS LIST 1 Determination device, 2 Camera, 3 Alarm device, 11 Target operation determination unit, 12 Mask wearing determination unit, 13 Determination control unit, 14 Alarm necessity determination unit, 15 Wearing notification control unit, 1001 Processing circuit, 1002 Input interface device, 1003 Output interface device, 1004 Processor, 1005 Memory.

Claims

1. a target action determination unit that performs a target action determination to determine whether a target person is performing a target action involving mouth movement based on a captured image of a face of the target person, the target person being determined to be performing the target action; a mask wearing determination unit that performs a mask wearing determination to determine whether the subject is wearing a mask based on the captured image; an alarm necessity determination unit that determines that an alarm needs to be output when the target action determination unit outputs a motion determination result indicating that the target person is performing the target motion; and a determination control unit that controls the warning necessity determination unit not to determine that output of the warning is necessary when the mask wearing determination unit determines that the subject is wearing the mask, when the target action determination unit determines that the subject is performing the target action and the mask wearing determination unit determines that the subject is wearing the mask, the determination control unit rejects the action determination result output from the target action determination unit; The warning necessity determination unit determines that the warning is unnecessary when the determination control unit rejects the operation determination result. A determination device characterized by:

2. a target action determination unit that performs a target action determination to determine whether a target person is performing a target action involving mouth movement based on a captured image of a face of the target person, the target person being determined to be performing the target action; a mask wearing determination unit that performs a mask wearing determination to determine whether the subject is wearing a mask based on the captured image; an alarm necessity determination unit that determines that an alarm needs to be output when the target action determination unit outputs a motion determination result indicating that the target person is performing the target motion; and a determination control unit that controls the warning necessity determination unit not to determine that output of the warning is necessary when the mask wearing determination unit determines that the subject is wearing the mask, when the mask wearing determination unit determines that the subject is wearing the mask, the determination control unit causes the target behavior determination unit to stop executing the target behavior determination; The warning necessity determination unit determines that the warning is unnecessary when the determination control unit causes the target operation determination unit to stop executing the target operation determination. A determination device characterized by:

3. When the mask wearing determination unit determines that the subject is wearing the mask, a wearing notification control unit that outputs wearing notification information for notifying that the subject is wearing the mask The determination device according to claim 1 or claim 2, comprising the above.

4. When the alarm necessity determination unit determines that the output of the alarm is necessary, the alarm necessity determination unit outputs alarm output control information for causing the alarm to be output The determination device according to claim 1 or claim 2, characterized by the above.

5. The target action is a smoking action or a eating / drinking action The determination device according to claim 1 or claim 2, characterized by the above.

6. The subject is a driver of a vehicle The determination device according to claim 1 or claim 2, characterized by the above.

7. The target action determination unit inputs the captured image into a machine learning model that takes the captured image as an input and outputs information indicating whether a person is performing the target action, and determines whether the subject is performing the target action The determination device according to claim 1 or claim 2, characterized by the above.

8. The mask wearing determination unit inputs the captured image into a machine learning model that takes the captured image as an input and outputs information indicating whether a person is wearing the mask, and determines whether the subject is wearing the mask The determination device according to claim 1 or claim 2, characterized by the above.

9. A step of executing a target action determination for determining whether the subject is performing the target action based on a captured image in which the face of the target subject for determining whether the target action involving mouth movement is being performed is captured; A step of executing a mask wearing determination for determining whether the subject is wearing a mask based on the captured image by the mask wearing determination unit; A step in which the alarm necessity determination unit determines that the output of the alarm is necessary when an action determination result indicating that the subject is performing the target action is output from the target action determination unit; The determination control unit includes a step of controlling so that the alarm necessity determination unit does not determine that the output of the alarm is necessary when the mask wearing determination unit determines that the subject is wearing the mask; When the target action determination unit determines that the subject is performing the target action and the mask wearing determination unit determines that the subject is wearing the mask, the determination control unit rejects the action determination result output from the target action determination unit When the operation determination result is rejected by the determination control unit, the alarm necessity determination unit determines that the alarm is unnecessary. A determination method characterized by this.

10. A step of executing a target motion determination for determining whether or not the target person is performing a target motion that involves mouth movement based on a captured image of the face of the target person to be determined; A step of executing a mask wearing determination for determining whether or not the target person is wearing a mask based on the captured image by the mask wearing determination unit; A step in which the alarm necessity determination unit determines that it is necessary to output an alarm when an operation determination result indicating that the target person is performing the target motion is output from the target motion determination unit; A determination control unit includes a step of controlling so that when the mask wearing determination unit determines that the target person is wearing the mask, the alarm necessity determination unit does not determine that it is necessary to output the alarm; When the mask wearing determination unit determines that the target person is wearing the mask, the determination control unit causes the target motion determination unit to abort the execution of the target motion determination; When the determination control unit causes the target motion determination unit to abort the execution of the target motion determination, the alarm necessity determination unit determines that the alarm is unnecessary. A determination method characterized by this.