Nose inhalation determination device, nose inhalation determination method, and nose inhalation determination program

Abstract

To provide a nose inhalation determination device capable of reducing a psychological burden on a determination object person.SOLUTION: The nose inhalation determination device comprises: a camera image acquisition unit which acquires a camera image obtained by taking an image of the face of the determination object person; an image analysis unit which applies image analysis processing on a camera image and determines whether or not the determination object person projected in the camera image is doing a motion of inhalation from the nose; and a determination result output unit which outputs a result of the determination by the image analysis unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a nasal inhalation determination device, a nasal inhalation determination method, and a nasal inhalation determination program. [Background technology]

[0002] Nasal breathing detection sensors that detect breathing through the nose have been developed. For example, a conventional nasal breathing detection sensor is composed of a pair of tabs on the left and right, which are attached to the skin between the nose and mouth. When breath from the nose hits the sensor, a piezoelectric element attached to the tab bends, and the bending generates a voltage that can detect breathing through the nose (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2006-212271 Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional nasal breathing detection sensors are attached to the skin between the nose and mouth, which can be uncomfortable for some people and can be a psychological burden.

[0005] The present invention has been made in consideration of the above points, and an object of the present invention is to provide a nasal inhalation determination device, a nasal inhalation determination method, and a nasal inhalation determination program that can reduce the psychological burden on the person being tested. [Means for solving the problem]

[0006] The nasal inhalation determination device according to the first aspect of the present invention comprises: a camera image acquisition unit that acquires a camera image capturing a face of the person to be determined; an image analysis unit that performs image analysis processing on the camera image and determines whether the subject of determination shown in the camera image is inhaling air through the nose; a determination result output unit that outputs according to the determination result of the image analysis unit; Equipped with.

[0007] According to this aspect, an image analysis process is performed on a camera image of the face of the person to be determined, and it is determined whether the person to be determined in the camera image is inhaling through their nose. This makes it possible to determine whether the person is inhaling through their nose based on the camera image of the person to be determined, thereby reducing the psychological burden on the person to be determined compared to conventional nasal breathing detection sensors (which are used by being attached to the skin between the nose and mouth).

[0008] A nasal inhalation determination device according to a second aspect of the present invention is the nasal inhalation determination device according to the first aspect, The image analysis unit identifies a target area corresponding to the nostrils of the person being judged from the camera image, and determines whether the person being judged is inhaling air through their nose based on a change in the shape of the target area.

[0009] A nasal inhalation determination device according to a third aspect of the present invention is the nasal inhalation determination device according to the second aspect, The image analysis unit determining whether the area of ​​the determination target region is smaller than a first reference value, and determining that the person being determined is inhaling through their nose if the area of ​​the determination target region is smaller than the first reference value; and / or It is determined whether the ratio of the horizontal length to the vertical length of the area to be determined is smaller than a second reference value, and if it is smaller than the second reference value, it is determined that the person to be determined is inhaling through their nose.

[0010] A nasal inhalation determination device according to a fourth aspect of the present invention is the nasal inhalation determination device according to the second aspect, The image analysis unit extracts a first feature point corresponding to the skin around the nose of the person being judged from the camera image, and determines whether the person being judged is inhaling air through their nose, taking into account the movement of the first feature point.

[0011] According to this aspect, in addition to the change in shape of the area to be judged corresponding to the nostrils of the person to be judged, the movement of the first feature point corresponding to the skin around the nose of the person to be judged is also taken into account, thereby making it possible to comprehensively judge the state of nasal inhalation, thereby improving the accuracy of the judgment.

[0012] A nasal inhalation determination device according to a fifth aspect of the present invention is the nasal inhalation determination device according to the second aspect, The image analysis unit extracts second feature points corresponding to the jawline and / or throat of the person being judged from the camera image, and determines whether the person being judged is inhaling air through their nose, taking into account the movement of the second feature points.

[0013] According to this aspect, in addition to the change in shape of the area to be judged corresponding to the nostrils of the person being judged, the movement of the second feature point corresponding to the jawline and / or throat of the person being judged is also taken into account to comprehensively judge the state of nasal inhalation, thereby improving the accuracy of the judgment.

[0014] A nasal inhalation determination device according to a sixth aspect of the present invention is the nasal inhalation determination device according to the second aspect, The image analysis unit uses a trained model that has machine-learned the relationship between past camera footage of the subject's face and whether or not the subject was inhaling through their nose, and inputs new camera footage of the face of the person being evaluated to estimate whether or not the person being evaluated is inhaling through their nose, and taking this estimation result into consideration, determines whether or not the person being evaluated is inhaling through their nose.

[0015] According to this aspect, the accuracy of the judgment can be improved by comprehensively judging the nasal inhalation state by taking into account not only the change in shape of the judgment target area corresponding to the nostrils of the person being judged, but also the estimation results from a trained machine learning model.

[0016] A seventh aspect of the present invention is a nasal inhalation determination device according to any one of the first to sixth aspects, When the judgment result output unit determines that the person being judged is inhaling through their nose, it outputs a trigger signal to a performance device that gives a predetermined performance effect to the person being judged, to cause the performance effect to be produced.

[0017] According to this aspect, a predetermined dramatic effect is given to the subject of evaluation in accordance with the timing of the inhalation action through the nose, thereby emphasizing or altering the breathing and olfactory senses of the subject of evaluation.

[0018] A nasal inhalation determination method according to an eighth aspect of the present invention is a computer-implemented method, comprising: A step of acquiring a camera image of the face of the person to be determined; a step of performing an image analysis process on the camera image to determine whether the subject of determination shown in the camera image is inhaling air through the nose; a step of outputting the result of the determination; Includes.

[0019] A nasal inhalation determination program according to a ninth aspect of the present invention includes: On the computer, A step of acquiring a camera image of the face of the person to be determined; a step of performing an image analysis process on the camera image to determine whether the subject of determination shown in the camera image is inhaling air through the nose; outputting an output according to the determination result of the image analysis unit; Execute the following. [Effects of the Invention]

[0020] According to the present invention, when determining whether or not the subject is inhaling through the nose, the psychological burden on the subject can be reduced. [Brief explanation of the drawings]

[0021] [Figure 1] FIG. 1 is a block diagram showing a schematic configuration of a system including a nasal inhalation determination device according to one embodiment. [Figure 2] FIG. 2 is a diagram showing an example of specifying a determination target region. [Figure 3] FIG. 3 is a diagram showing an example of a change in the shape of the determination target region. [Figure 4] FIG. 4 is a diagram showing an example of the movements of the first feature points and the second feature points. [Figure 5] FIG. 5 is a flowchart showing an example of the operation of the nasal inhalation determination device. DETAILED DESCRIPTION OF THE INVENTION

[0022] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description and the drawings used in the following description, parts that can be configured identically will be designated by the same reference numerals, and duplicated descriptions will be omitted.

[0023] 1 is a diagram showing a schematic configuration of a system 10 according to one embodiment. As shown in FIG. 1, the system 10 includes a nasal inhalation determination device 1, a photographing device 100, and a performance device 101.

[0024] Of these, the photographing device 100 is a camera having an image sensor such as a CCD or CMOS, and is arranged so as to be able to photograph the face of the person to be judged. The photographing device 100 may be arranged so as to be able to photograph the face of the person to be judged from the front.

[0025] The performance device 101 is a device that has the function of providing various performance effects to the person being assessed, and is, for example, an aroma diffuser, an air cooler, an air warmer, bone conduction earphones, a speaker, a display, or the like.

[0026] Next, we will explain the nasal inhalation determination device 1. The nasal inhalation determination device 1 may be composed of a single computer, or may be composed of multiple computers connected to each other so that they can communicate with each other via a network.

[0027] 1, the nasal inhalation determination device 1 has a camera image acquisition unit 2, an image analysis unit 3, and a determination result output unit 4. These units 2 to 4 may be realized by a processor in the nasal inhalation determination device 1 executing a predetermined program, or may be implemented in hardware.

[0028] Of these, the camera image acquisition unit 2 acquires camera images captured by the image capturing device 100 (camera images capturing the face of the person to be determined).

[0029] The image analysis unit 3 performs image analysis processing on the camera image acquired by the camera image acquisition unit 2, and determines whether the person being evaluated in the camera image is inhaling air through their nose.

[0030] For example, referring to Fig. 2, the image analysis unit 3 first identifies determination target areas 51 and 52 corresponding to the nostrils (external nostrils) of the person being determined from the camera image. The image analysis unit 3 may identify the outer diameters (contours) of the determination target areas 51 and 52 corresponding to the nostrils of the person being determined from the edges of contrast in the camera image, etc. When multiple candidates for the determination target area are identified from the edges of contrast in the camera image, the image analysis unit 3 may use known face recognition technology to extract feature points corresponding to the eyes, nose, and mouth of the person being determined from the camera image, and determine the determination target areas 51 and 52 corresponding to the nostrils of the person being determined based on the positional relationship between these feature points.

[0031] Then, based on the change in shape of the identified determination target regions 51 and 52, the image analysis unit 3 determines whether the person to be determined is inhaling air through their nose.

[0032] 3 is a diagram showing an example of a change in the shape of the determination target regions 51 and 52. As shown in FIG. 3, when the person to be determined is not inhaling through their nose, the shapes of the determination target regions 51 and 52 expand (their areas become larger and they extend horizontally), and when the person to be determined is inhaling through their nose, the shapes of the determination target regions 51 and 52 narrow (their areas become smaller and they extend vertically). Here, "vertical" refers to the direction perpendicular to a reference line A connecting the centers of the pair of determination target regions 51 and 52, and "horizontal" refers to the direction parallel to the reference line A.

[0033] Therefore, as an example, the image analysis unit 3 may calculate the area of ​​the identified determination target regions 51, 52, and determine whether the area of ​​the determination target regions 51, 52 in the current frame image has become smaller than a first reference value (for example, 95% of the area of ​​the determination target regions 51, 52 in the frame image a predetermined number of frames before). If the area of ​​the determination target regions 51, 52 has become smaller than the first reference value, the image analysis unit 3 may determine that the person being determined is inhaling air through their nose. On the other hand, if the area of ​​the determination target regions 51, 52 has not become smaller than the first reference value, the image analysis unit 3 may determine that the person being determined is not inhaling air through their nose.

[0034] As another example, referring to FIG. 3, the image analysis unit 3 may calculate the vertical length L1 and horizontal length L2 of the identified determination target regions 51 and 52, and determine whether the ratio (L2 / L1) of the horizontal length L2 to the vertical length L1 of the determination target regions 51 and 52 in the current frame image is smaller than a second reference value (e.g., 80% of the ratio (L2 / L1) in the frame image a predetermined number of frames before). If the ratio (L2 / L1) of the horizontal length L2 to the vertical length L1 of the determination target regions 51 and 52 is smaller than the second reference value, the image analysis unit 3 may determine that the person being determined is inhaling through their nose. On the other hand, if the ratio (L2 / L1) of the horizontal length L2 to the vertical length L1 of the determination target regions 51 and 52 is not smaller than the second reference value, the image analysis unit 3 may determine that the person being determined is not inhaling through their nose.

[0035] As another example, the image analysis unit 3 may determine that the person being judged is inhaling through their nose if the area of ​​the judgment target areas 51, 52 becomes smaller than a first reference value and the ratio (L2 / L1) of the horizontal length L2 to the vertical length L1 of the judgment target areas 51, 52 becomes smaller than a second reference value, and if this is not the case, may determine that the person being judged is not inhaling through their nose.

[0036] 4 is a diagram showing an example of the movement of first feature points 53 corresponding to the skin around the nose and second feature points 54 corresponding to the jawline and / or throat. As shown in FIG. 4, when the person to be determined is inhaling through their nose, first feature points 53 corresponding to the skin around the nose move to converge at the center (midline) between the left and right. Also, second feature points 54 corresponding to the jawline and / or throat move up and down.

[0037] Therefore, as a first modified example, the image analysis unit 3 may extract a first feature point 53 corresponding to the skin around the nose of the person being determined from the camera image. A known technique may be used to extract the feature point. The image analysis unit 3 may then determine whether the person being determined is inhaling air through their nose, taking into account the movement of the extracted first feature point 53. Specifically, for example, the image extraction unit 3 calculates the amount of movement D1 (the amount of movement in the direction of converging toward the center) of the position of the first feature point 53 in the current frame image from the position of the first feature point 53 in the frame image a predetermined number of frames before, and determines whether the amount of movement D1 is greater than a predetermined threshold (for example, a change rate of 3% based on the distance to a neighboring feature point when stationary). Then, when it is determined that the person to be determined is inhaling through their nose based on the change in shape of the determination target regions 51, 52 (for example, when the area of ​​the determination target regions 51, 52 becomes smaller than a first reference value) and when the movement amount D1 of the first feature point 53 is greater than a predetermined threshold, the image analysis unit 3 may comprehensively determine that the person to be determined is inhaling through their nose, or otherwise may comprehensively determine that the person to be determined is not inhaling through their nose. The accuracy of the determination can be improved by comprehensively determining the state of nasal inhalation by taking into account not only the change in shape of the determination target regions corresponding to the nostrils of the person to be determined but also the movement of the first feature point corresponding to the skin around the nose of the person to be determined.

[0038] As a second modified example, the image analysis unit 3 may extract second feature points 54 corresponding to the jawline and / or throat of the person being determined from the camera image. The image analysis unit 3 may then determine whether the person being determined is inhaling air through their nose, taking into account the movement of the extracted second feature points 54. Specifically, for example, the image extraction unit 3 calculates the amount of movement D2 in the vertical direction of the position of the second feature point 54 in the current frame image from the position of the second feature point 54 in the frame image a predetermined number of frames before, and determines whether the amount of movement D2 is greater than a predetermined threshold (for example, a change rate of 3% based on the distance to a nearby feature point when stationary). Then, when it is determined that the person to be determined is inhaling through their nose based on the change in shape of the determination target regions 51, 52 (for example, when the area of ​​the determination target regions 51, 52 becomes smaller than the first reference value) and when the movement amount D2 of the second feature point 54 is greater than a predetermined threshold, the image analysis unit 3 may comprehensively determine that the person to be determined is inhaling through their nose, or otherwise may comprehensively determine that the person to be determined is not inhaling through their nose. The accuracy of the determination can be improved by comprehensively determining the state of nasal inhalation by taking into account not only the change in shape of the determination target regions corresponding to the nostrils of the person to be determined but also the movement of the second feature point corresponding to the jawline and / or throat of the person to be determined.

[0039] As a third modified example, the image analysis unit 3 may use a trained model (e.g., a neural network model) that has learned the relationship between past camera images of the subject's face and whether the subject was inhaling through his / her nose as input of new camera images of the subject's face (camera images acquired by the camera image acquisition unit 2) to estimate whether the subject is inhaling through his / her nose. The image analysis unit 3 may then determine whether the subject is inhaling through his / her nose by taking into account the estimation result of the trained model. Specifically, for example, if the image extraction unit 3 determines that the subject is inhaling through his / her nose based on a change in the shape of the determination target areas 51 and 52 (e.g., if the area of ​​the determination target areas 51 and 52 becomes smaller than a first reference value) and if the trained model also estimates that the subject is inhaling through his / her nose, the image extraction unit 3 may comprehensively determine that the subject is inhaling through his / her nose; otherwise, the image extraction unit 3 may comprehensively determine that the subject is not inhaling through his / her nose. In addition to the change in shape of the area to be judged corresponding to the nostrils of the person being judged, the accuracy of the judgment can be improved by taking into account the estimation results from a trained machine learning model to comprehensively judge the nasal inhalation state.

[0040] The determination result output unit 4 outputs the result of the determination by the image analysis unit 3.

[0041] As an example, when it is determined that the person to be determined is inhaling through their nose, the determination result output unit 4 may output a trigger signal to the performance device 101 to perform a predetermined performance effect. This allows the performance device 101 to generate a performance (for example, spraying fragrance from an aroma diffuser, cool air from an air cooler, warm air from an air heater, vibration from bone conduction earphones, sound from a speaker, video images from a display or projector, etc.) in accordance with the timing when it is determined that the person to be determined is inhaling through their nose, and these performance effects can emphasize or alter the breathing or olfactory sensations of the person to be determined.

[0042] Next, an example of the operation of the nasal inhalation determination device 1 will be described with reference to Fig. 5. Fig. 5 is a flowchart showing an example of the operation of the nasal inhalation determination device 1.

[0043] As shown in FIG. 5, first, the camera image acquisition unit 2 acquires a camera image of the face of the person to be determined from the image capture device 100 (step 10).

[0044] Then, the image analysis unit 3 performs image analysis processing on the camera image acquired by the camera image acquisition unit 2 to determine whether the person being evaluated in the camera image is inhaling air through their nose.

[0045] In the example shown in FIG. 5, the image analysis unit 3 first identifies the determination target areas 51, 52 corresponding to the nostrils (external nostrils) of the person to be determined from the camera image acquired by the camera image acquisition unit 2 (step S11).

[0046] Next, the image analysis unit 3 determines whether the person to be determined is inhaling through their nose based on the change in shape of the identified determination target regions 51, 52 (step S12). For example, the image analysis unit 3 may determine whether the area of ​​the determination target regions 51, 52 has become smaller than a first reference value, and if it has become smaller than the first reference value, determine that the person to be determined is inhaling through their nose, or may determine whether the ratio (L2 / L1) of the horizontal length L2 to the vertical length L1 of the determination target regions 51, 52 has become smaller than a second reference value, and if it has become smaller than the second reference value, determine that the person to be determined is inhaling through their nose.

[0047] In step S12, the image analysis unit 3 may extract a first feature point 53 corresponding to the skin around the nose of the person being judged from the camera image acquired by the camera image acquisition unit 2, and make a comprehensive judgment as to whether or not the person being judged is inhaling air through their nose, taking into account the movement of the first feature point 53, or may extract a second feature point 54 corresponding to the jawline and / or throat of the person being judged from the camera image acquired by the camera image acquisition unit 2, and make a comprehensive judgment as to whether or not the person being judged is inhaling air through their nose, taking into account the movement of the second feature point 54.

[0048] In step S12, the image analysis unit 3 uses a trained model that has been machine-learned to determine the relationship between past camera images of the subject's face and whether the subject was inhaling through their nose, and uses the new camera image acquired by the camera image acquisition unit 2 as input to estimate whether the person being evaluated is inhaling through their nose, and may also take into account the estimation results from the trained model to make a comprehensive determination as to whether the person being evaluated is inhaling through their nose.

[0049] Then, the determination result output unit 4 outputs the determination result of the image analysis unit 3 (step S13). For example, when it is determined that the person to be determined is inhaling through the nose, the determination result output unit 4 may output a trigger signal to the performance device 101, thereby causing the performance device 101 to generate a predetermined performance.

[0050] According to the present embodiment, an image analysis process is performed on the camera image of the face of the person to be determined, and it is determined whether the person to be determined in the camera image is inhaling through their nose. This makes it possible to determine whether the person is inhaling through their nose based on the camera image of the person to be determined, thereby reducing the psychological burden on the person to be determined compared to conventional nasal breathing detection sensors (which are used by being attached to the skin between the nose and mouth).

[0051] The above-described description of the embodiment and the disclosure of the drawings are merely examples for explaining the invention described in the claims, and the invention described in the claims is not limited by the above-described description of the embodiment or the disclosure of the drawings. The components of the above-described embodiment can be combined in any manner without departing from the spirit of the invention.

[0052] Furthermore, at least a part of the nasal inhalation determination device 1 in this embodiment may be configured by a computer, but the program for realizing at least a part of the nasal inhalation determination device 1 in a computer and a computer-readable recording medium on which the program is non-transitory are also protected by this case. [Explanation of symbols]

[0053] 1. Nasal inhalation detection device 2 Camera image acquisition unit 3. Image analysis section 4. Judgment result output section 51, 52 Judgment area 53 First feature point 54 Second feature point 100 Imaging device 101 Production Devices

Claims

1. a camera image acquisition unit that acquires a camera image capturing a face of the person to be determined; an image analysis unit that performs image analysis processing on the camera image and determines whether the subject of determination shown in the camera image is inhaling air through their nose; a determination result output unit that outputs according to the determination result of the image analysis unit; A nasal inhalation determination device comprising:

2. the image analysis unit identifies a determination target area corresponding to the nostrils of the person to be determined from the camera image, and determines whether the person to be determined is inhaling air through their nose based on a change in shape of the determination target area. The nasal inhalation determination device according to claim 1 .

3. The image analysis unit determining whether the area of ​​the determination target region is smaller than a first reference value, and determining that the person being determined is inhaling through their nose if the area of ​​the determination target region is smaller than the first reference value; and / or determining whether the ratio of the horizontal length to the vertical length of the determination target area is smaller than a second reference value, and if the ratio is smaller than the second reference value, determining that the person being determined is inhaling through their nose; The nasal inhalation determination device according to claim 2.

4. the image analysis unit extracts a first feature point corresponding to the skin around the nose of the person to be determined from the camera image, and determines whether the person to be determined is inhaling air through the nose, taking into account the movement of the first feature point. The nasal inhalation determination device according to claim 2.

5. the image analysis unit extracts second feature points corresponding to the jawline and / or throat of the person being determined from the camera image, and determines whether the person being determined is inhaling air through their nose, taking into account the movement of the second feature points. The nasal inhalation determination device according to claim 2.

6. The image analysis unit uses a trained model that has been machine-learned to understand the relationship between past camera images of the subject's face and whether the subject has been inhaling through their nose, and inputs new camera images of the subject's face to estimate whether the subject has been inhaling through their nose, and takes the estimation result into consideration to determine whether the subject has been inhaling through their nose. The nasal inhalation determination device according to claim 2.

7. the determination result output unit, when it is determined that the person to be determined is inhaling through the nose, outputs a trigger signal to a performance device that provides a predetermined performance effect to the person to be determined, for the performance effect to be provided. The nasal inhalation determination device according to any one of claims 1 to 6.

8. A computer-implemented method for determining nasal inspiration, comprising: A step of acquiring a camera image of the face of the person to be determined; a step of performing an image analysis process on the camera image to determine whether the subject of determination shown in the camera image is inhaling air through the nose; a step of outputting the result of the determination; A method for determining nasal inhalation, comprising:

9. On the computer, A step of acquiring a camera image of the face of the person to be determined; a step of performing an image analysis process on the camera image to determine whether the subject of determination shown in the camera image is inhaling air through the nose; a step of outputting the result of the determination; A nasal inhalation determination program that executes the following.

Images